diff --git a/docs/troubleshooting/troubleshooting.md b/docs/troubleshooting/troubleshooting.md index 882815c22..48b69bfc0 100644 --- a/docs/troubleshooting/troubleshooting.md +++ b/docs/troubleshooting/troubleshooting.md @@ -1,14 +1,55 @@ --- title: Troubleshooting +sidebar_label: Troubleshooting Guide --- -## GPU Memory Limit Not Enforced {#gpu-memory-limit-not-enforced} +This guide covers common issues you might encounter when installing, scheduling, and running workloads with HAMi. It is broken down into three main sections: Installation, Scheduling, and Runtime. -If a container exceeds its `nvidia.com/gpumem` limit, check the following causes: +## Diagnostic Flowchart + +Use this flowchart to quickly identify where your issue might be occurring: + +```mermaid +%% title: HAMi Troubleshooting Decision Tree +flowchart TD + A[Pod not running as expected] --> B{What is the Pod status?} + B -->|Pending| C[Scheduling Issues] + B -->|CrashLoopBackOff / Error| D[Runtime Issues] + B -->|Running but no GPU| E[Installation / Device Plugin Issues] + + C --> F{Are GPUs available?} + F -->|No| G[Check node capacity and HAMi limits] + F -->|Yes| H[Check nodeSelector / Scheduler logs] + + D --> I{Is it an OOM error?} + I -->|Yes| J[Check nvidia.com/gpumem limits] + I -->|No| K[Check libvgpu.so preload] + + E --> L[Check hami-device-plugin logs] +``` + +--- + +## 1. Installation Issues + +Installation issues usually manifest as the `hami-device-plugin` pods crashing or failing to register GPUs to the Kubernetes nodes. + +### Device Plugin Fails to Start + +- Since v2.3.10, HAMi has changed the `device-plugin` environment variable name from `NodeName` to `NODE_NAME`. If you are using an image version earlier than v2.3.10, the `device-plugin` may fail to start. + + To resolve this issue, you have two options: + - Manually edit the DaemonSet using `kubectl edit daemonset` and update the environment variable from `NodeName` to `NODE_NAME`. + - Upgrade the `device-plugin` image to the latest version using Helm: + + ```bash + helm upgrade hami hami/hami -n kube-system + ``` + +### Container Runtime Configuration + +If the `hami-device-plugin` is running but your nodes don't show `nvidia.com/gpu` resources, verify your containerd configuration: -- **`CUDA_DISABLE_CONTROL=true` is set** - disables HAMi-core enforcement entirely. Remove it from production workloads. -- **Docker-in-Docker (DinD)** - inner containers do not inherit the `/etc/ld.so.preload` hostPath mount. HAMi enforcement does not apply inside DinD. -- **Direct driver API usage** - workloads calling NVML or the CUDA Driver API directly bypass `libvgpu.so`. - **`nvidia-container-runtime` not set as default** - verify with: ```bash @@ -17,19 +58,52 @@ If a container exceeds its `nvidia.com/gpumem` limit, check the following causes The output must show `nvidia`. If not, follow the [Prerequisites](../installation/online-installation) guide. +--- + +## 2. Scheduling Issues + +Scheduling issues occur when your Pods remain in the `Pending` state. + +### Pods Stuck in Pending + +- **NodeName unsupported:** Tasks with the `nodeName` field cannot be scheduled at the moment; please use `nodeSelector` instead. +- **A100 MIG restrictions:** Currently, A100 MIG can be supported in only "none" and "mixed" modes. +- **Check Scheduler Logs:** Use the following command to view why the HAMi scheduler rejected a pod: + + ```bash + kubectl logs -n kube-system -l component=hami-scheduler + ``` + +--- + +## 3. Runtime Issues + +Runtime issues occur after the Pod is scheduled, but it fails to execute correctly or doesn't respect isolation limits. + +### GPU Memory Limit Not Enforced {#gpu-memory-limit-not-enforced} + +If a container exceeds its `nvidia.com/gpumem` limit, check the following causes: + +- **`CUDA_DISABLE_CONTROL=true` is set** - disables HAMi-core enforcement entirely. Remove it from production workloads. +- **Docker-in-Docker (DinD)** - inner containers do not inherit the `/etc/ld.so.preload` hostPath mount. HAMi enforcement does not apply inside DinD. +- **Direct driver API usage** - workloads calling NVML or the CUDA Driver API directly bypass `libvgpu.so`. - If you don’t explicitly request vGPUs when using the device plugin with NVIDIA images, all GPUs on the host may be exposed to your container. -- Currently, A100 MIG can be supported in only "none" and "mixed" modes. -- Tasks with the "nodeName" field cannot be scheduled at the moment; please use "nodeSelector" instead. + +### Unsupported Workloads + - Only computing tasks are currently supported; video codec processing is not supported. -- Since v2.3.10, HAMi has changed the `device-plugin` environment variable name from `NodeName` to `NODE_NAME`. If you are using an image version earlier than v2.3.10, the `device-plugin` may fail to start. - To resolve this issue, you have two options: +## Diagnostic Cheat Sheet - - Manually edit the DaemonSet using `kubectl edit daemonset` and update the environment variable from `NodeName` to `NODE_NAME`. - - Upgrade the `device-plugin` image to the latest version using Helm: +Here are some helpful commands for diagnosing HAMi clusters: - ```bash - helm upgrade hami hami/hami -n kube-system - ``` +```bash +# Check if HAMi resources are allocatable on a node +kubectl get node -o yaml | grep nvidia.com + +# Check the logs of the HAMi device plugin +kubectl logs -n kube-system -l app.kubernetes.io/name=hami - This will apply the fix automatically. +# Verify if the libvgpu library is mounted inside a running pod +kubectl exec -it -- ls -l /usr/local/vgpu/libvgpu.so +``` diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/bagualu-intelligent-computing-stack.webp b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/bagualu-intelligent-computing-stack.webp index 50428b435..dfb636952 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/bagualu-intelligent-computing-stack.webp and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/bagualu-intelligent-computing-stack.webp differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/bagualu-performance-delivery.webp b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/bagualu-performance-delivery.webp index a43fa3453..a5ca19309 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/bagualu-performance-delivery.webp and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/bagualu-performance-delivery.webp differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/beike-hami-partnership.webp b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/beike-hami-partnership.webp index 53b2b8c28..0b74d0059 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/beike-hami-partnership.webp and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/beike-hami-partnership.webp differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/beike-vgpu-inference-cluster-practice.webp b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/beike-vgpu-inference-cluster-practice.webp index 2a24d74c8..f92677140 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/beike-vgpu-inference-cluster-practice.webp and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/beike-vgpu-inference-cluster-practice.webp differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/dcu-practice-implementation.webp b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/dcu-practice-implementation.webp index 13c9d3300..31e36abd4 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/dcu-practice-implementation.webp and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/dcu-practice-implementation.webp differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/dcu-virtualization-basics.webp b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/dcu-virtualization-basics.webp index a689a261b..009e724f1 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/dcu-virtualization-basics.webp and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/dcu-virtualization-basics.webp differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/device-plugin-architecture.webp b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/device-plugin-architecture.webp index 85122e9f9..bdee49ca8 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/device-plugin-architecture.webp and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/device-plugin-architecture.webp differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/device-plugin-new-features.webp b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/device-plugin-new-features.webp index 523c61692..6d6b30568 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/device-plugin-new-features.webp and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/device-plugin-new-features.webp differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/dra-driver-practice.webp b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/dra-driver-practice.webp index 00ab35ed7..686d728c0 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/dra-driver-practice.webp and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/dra-driver-practice.webp differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/hami-community.webp b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/hami-community.webp index 8d81c59f4..89505356c 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/hami-community.webp and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/hami-community.webp differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/hami-core-dra-architecture.webp b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/hami-core-dra-architecture.webp index b0cbdbdbf..1f2249d67 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/hami-core-dra-architecture.webp and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/hami-core-dra-architecture.webp differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/hami-meetup-beijing-banner.webp b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/hami-meetup-beijing-banner.webp index c0acda3f9..18b462617 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/hami-meetup-beijing-banner.webp and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/hami-meetup-beijing-banner.webp differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/hami-new-features-capability-matrix.webp b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/hami-new-features-capability-matrix.webp index f72b0f35d..50c612f6f 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/hami-new-features-capability-matrix.webp and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/hami-new-features-capability-matrix.webp differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/hami-v2.7.0-domestic-compute.webp b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/hami-v2.7.0-domestic-compute.webp index 35908b1ec..19d56a868 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/hami-v2.7.0-domestic-compute.webp and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/hami-v2.7.0-domestic-compute.webp differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/kunlunxin-xpu-adapter.webp b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/kunlunxin-xpu-adapter.webp index 824b7d1e1..36010cf54 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/kunlunxin-xpu-adapter.webp and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/kunlunxin-xpu-adapter.webp differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/meetup-networking-session.webp b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/meetup-networking-session.webp index 6f1937a8a..6b8340dab 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/meetup-networking-session.webp and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-beijing-2025/meetup-networking-session.webp differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/daocloud-drun-practice.png b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/daocloud-drun-practice.png index 2685ea960..c3c5aa9cb 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/daocloud-drun-practice.png and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/daocloud-drun-practice.png differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/hami-version-evolution.png b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/hami-version-evolution.png index 6a4254383..90f007cd9 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/hami-version-evolution.png and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/hami-version-evolution.png differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/keith-opening-keynote.png b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/keith-opening-keynote.png index a834eec7e..e937abc71 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/keith-opening-keynote.png and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/keith-opening-keynote.png differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/meetup-banner.png b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/meetup-banner.png index a5c4db4d5..864131b7d 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/meetup-banner.png and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/meetup-banner.png differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/metax-sgpu-hami.png b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/metax-sgpu-hami.png index 7e35fc486..c944e3679 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/metax-sgpu-hami.png and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/metax-sgpu-hami.png differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/networking-session.png b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/networking-session.png index e8de65cb3..7252039e1 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/networking-session.png and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/networking-session.png differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/performance-optimization.png b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/performance-optimization.png index 8bead6ad2..d47849d14 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/performance-optimization.png and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/performance-optimization.png differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/transwarp-technology.png b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/transwarp-technology.png index 9e449c2e1..f969b84b3 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/transwarp-technology.png and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shanghai-2025/transwarp-technology.png differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shenzhen-2026/meetup-shenzhen-group-photo.webp b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shenzhen-2026/meetup-shenzhen-group-photo.webp index 7c944a136..75d263d6d 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shenzhen-2026/meetup-shenzhen-group-photo.webp and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shenzhen-2026/meetup-shenzhen-group-photo.webp differ diff --git a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shenzhen-2026/meetup-shenzhen-video-qr.png b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shenzhen-2026/meetup-shenzhen-video-qr.png index 0a3c837c3..9db82fd6e 100644 Binary files a/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shenzhen-2026/meetup-shenzhen-video-qr.png and b/i18n/zh/docusaurus-plugin-content-blog/hami-meetup-shenzhen-2026/meetup-shenzhen-video-qr.png differ diff --git a/sidebars-tutorials.js b/sidebars-tutorials.js index e3881dc71..be7fa7d1c 100644 --- a/sidebars-tutorials.js +++ b/sidebars-tutorials.js @@ -41,6 +41,11 @@ module.exports = { id: "labs/hami-vllm", customProps: { level: "Intermediate", duration: "about 45 minutes" }, }, + { + type: "doc", + id: "labs/hami-sglang", + customProps: { level: "Intermediate", duration: "about 45 minutes" }, + }, { type: "doc", id: "labs/hami-isolation-k3s", diff --git a/static/favicons/android-chrome-192x192.png b/static/favicons/android-chrome-192x192.png index 1d0b9a540..29b0fb385 100644 Binary files a/static/favicons/android-chrome-192x192.png and b/static/favicons/android-chrome-192x192.png differ diff --git a/static/favicons/apple-touch-icon.png b/static/favicons/apple-touch-icon.png index f2f418bb1..cc55818dd 100644 Binary files a/static/favicons/apple-touch-icon.png and b/static/favicons/apple-touch-icon.png differ diff --git a/static/favicons/favicon-16x16.png b/static/favicons/favicon-16x16.png index 7337bf8d7..2995fc870 100644 Binary files a/static/favicons/favicon-16x16.png and b/static/favicons/favicon-16x16.png differ diff --git a/static/favicons/favicon-32x32.png b/static/favicons/favicon-32x32.png index b21c512bb..ca270056a 100644 Binary files a/static/favicons/favicon-32x32.png and b/static/favicons/favicon-32x32.png differ diff --git a/static/favicons/mstile-150x150.png b/static/favicons/mstile-150x150.png index dca2136e1..3af8dfba4 100644 Binary files a/static/favicons/mstile-150x150.png and b/static/favicons/mstile-150x150.png differ diff --git a/static/img/adopters/ashermed.png b/static/img/adopters/ashermed.png index cff1b7a70..3c0add592 100644 Binary files a/static/img/adopters/ashermed.png and b/static/img/adopters/ashermed.png differ diff --git a/static/img/adopters/baidu-cloud.png b/static/img/adopters/baidu-cloud.png index 4ddc67dba..b3c02d64a 100644 Binary files a/static/img/adopters/baidu-cloud.png and b/static/img/adopters/baidu-cloud.png differ diff --git a/static/img/adopters/baihai.jpg b/static/img/adopters/baihai.jpg index 6064f97f6..39febef42 100644 Binary files a/static/img/adopters/baihai.jpg and b/static/img/adopters/baihai.jpg differ diff --git a/static/img/adopters/boardware.jpg b/static/img/adopters/boardware.jpg index 0adb0b2bd..69836b809 100644 Binary files a/static/img/adopters/boardware.jpg and b/static/img/adopters/boardware.jpg differ diff --git a/static/img/adopters/bonc.png b/static/img/adopters/bonc.png index 5022d349d..20ad5cde4 100644 Binary files a/static/img/adopters/bonc.png and b/static/img/adopters/bonc.png differ diff --git a/static/img/adopters/cetc-zh.png b/static/img/adopters/cetc-zh.png index cbb8fed7e..dc71c8a4d 100644 Binary files a/static/img/adopters/cetc-zh.png and b/static/img/adopters/cetc-zh.png differ diff --git a/static/img/adopters/cetc.png b/static/img/adopters/cetc.png index dce2b1105..b6a2a275a 100644 Binary files a/static/img/adopters/cetc.png and b/static/img/adopters/cetc.png differ diff --git a/static/img/adopters/china-merchants-bank.png b/static/img/adopters/china-merchants-bank.png index 28886fbcd..9a63f563e 100644 Binary files a/static/img/adopters/china-merchants-bank.png and b/static/img/adopters/china-merchants-bank.png differ diff --git a/static/img/adopters/china-mobile.png b/static/img/adopters/china-mobile.png index 528eec82c..3b2dc2589 100644 Binary files a/static/img/adopters/china-mobile.png and b/static/img/adopters/china-mobile.png differ diff --git a/static/img/adopters/china-unicom.png b/static/img/adopters/china-unicom.png index 71cdf4793..80f24402d 100644 Binary files a/static/img/adopters/china-unicom.png and b/static/img/adopters/china-unicom.png differ diff --git a/static/img/adopters/china-university-of-mining-and-technology.jpg b/static/img/adopters/china-university-of-mining-and-technology.jpg index 8c9b84ac1..6ee6aba65 100644 Binary files a/static/img/adopters/china-university-of-mining-and-technology.jpg and b/static/img/adopters/china-university-of-mining-and-technology.jpg differ diff --git a/static/img/adopters/coocaa.png b/static/img/adopters/coocaa.png index e05232369..7a454c359 100644 Binary files a/static/img/adopters/coocaa.png and b/static/img/adopters/coocaa.png differ diff --git a/static/img/adopters/cyintell.png b/static/img/adopters/cyintell.png index 7d60d0ee6..2a461ecb9 100644 Binary files a/static/img/adopters/cyintell.png and b/static/img/adopters/cyintell.png differ diff --git a/static/img/adopters/deeproute.png b/static/img/adopters/deeproute.png index 97d4b2930..65894902d 100644 Binary files a/static/img/adopters/deeproute.png and b/static/img/adopters/deeproute.png differ diff --git a/static/img/adopters/dialo.png b/static/img/adopters/dialo.png index 1d01ea26e..9781b6428 100644 Binary files a/static/img/adopters/dialo.png and b/static/img/adopters/dialo.png differ diff --git a/static/img/adopters/donghua-university.png b/static/img/adopters/donghua-university.png index 339bcb65b..809a030aa 100644 Binary files a/static/img/adopters/donghua-university.png and b/static/img/adopters/donghua-university.png differ diff --git a/static/img/adopters/ecloud.jpg b/static/img/adopters/ecloud.jpg index b97fcabe8..733bcab56 100644 Binary files a/static/img/adopters/ecloud.jpg and b/static/img/adopters/ecloud.jpg differ diff --git a/static/img/adopters/empathy.png b/static/img/adopters/empathy.png index 50b5b796a..4b0fd5f94 100644 Binary files a/static/img/adopters/empathy.png and b/static/img/adopters/empathy.png differ diff --git a/static/img/adopters/ghostcloud.png b/static/img/adopters/ghostcloud.png index 5926de18f..87fb164f5 100644 Binary files a/static/img/adopters/ghostcloud.png and b/static/img/adopters/ghostcloud.png differ diff --git a/static/img/adopters/gsafety.jpg b/static/img/adopters/gsafety.jpg index e84d20c58..5b9d9ac98 100644 Binary files a/static/img/adopters/gsafety.jpg and b/static/img/adopters/gsafety.jpg differ diff --git a/static/img/adopters/guangdong-university-of-technology.jpg b/static/img/adopters/guangdong-university-of-technology.jpg index ceb5dba18..03fe614d1 100644 Binary files a/static/img/adopters/guangdong-university-of-technology.jpg and b/static/img/adopters/guangdong-university-of-technology.jpg differ diff --git a/static/img/adopters/guangdong-university-of-technology.png b/static/img/adopters/guangdong-university-of-technology.png index 54e8a9b9d..208d45cc5 100644 Binary files a/static/img/adopters/guangdong-university-of-technology.png and b/static/img/adopters/guangdong-university-of-technology.png differ diff --git a/static/img/adopters/guangzhou-pingao.jpg b/static/img/adopters/guangzhou-pingao.jpg index 77a592cdb..bc8c6bcec 100644 Binary files a/static/img/adopters/guangzhou-pingao.jpg and b/static/img/adopters/guangzhou-pingao.jpg differ diff --git a/static/img/adopters/h3c.png b/static/img/adopters/h3c.png index 06f5a0903..acfe50124 100644 Binary files a/static/img/adopters/h3c.png and b/static/img/adopters/h3c.png differ diff --git a/static/img/adopters/hangzhou-lianhui.png b/static/img/adopters/hangzhou-lianhui.png index 00d5a60cb..d553ae856 100644 Binary files a/static/img/adopters/hangzhou-lianhui.png and b/static/img/adopters/hangzhou-lianhui.png differ diff --git a/static/img/adopters/haofan.jpg b/static/img/adopters/haofan.jpg index 4e92eb780..a6f934295 100644 Binary files a/static/img/adopters/haofan.jpg and b/static/img/adopters/haofan.jpg differ diff --git a/static/img/adopters/harbin-institute-of-technology.png b/static/img/adopters/harbin-institute-of-technology.png index 6c7ff920f..c1960fc24 100644 Binary files a/static/img/adopters/harbin-institute-of-technology.png and b/static/img/adopters/harbin-institute-of-technology.png differ diff --git a/static/img/adopters/i-tudou.png b/static/img/adopters/i-tudou.png index cfdaefbbe..21ad15278 100644 Binary files a/static/img/adopters/i-tudou.png and b/static/img/adopters/i-tudou.png differ diff --git a/static/img/adopters/infervision.png b/static/img/adopters/infervision.png index 320ecb75a..518eca4c6 100644 Binary files a/static/img/adopters/infervision.png and b/static/img/adopters/infervision.png differ diff --git a/static/img/adopters/institute-of-information-engineering-cas.png b/static/img/adopters/institute-of-information-engineering-cas.png index 7461fdc15..f08b70c55 100644 Binary files a/static/img/adopters/institute-of-information-engineering-cas.png and b/static/img/adopters/institute-of-information-engineering-cas.png differ diff --git a/static/img/adopters/kylinsoft.png b/static/img/adopters/kylinsoft.png index 415ec82d9..8ca7fc062 100644 Binary files a/static/img/adopters/kylinsoft.png and b/static/img/adopters/kylinsoft.png differ diff --git a/static/img/adopters/linkedin.png b/static/img/adopters/linkedin.png index 4e4f8b134..6c92b895e 100644 Binary files a/static/img/adopters/linkedin.png and b/static/img/adopters/linkedin.png differ diff --git a/static/img/adopters/linklogis.jpg b/static/img/adopters/linklogis.jpg index 7c65f84fd..53f0d1e06 100644 Binary files a/static/img/adopters/linklogis.jpg and b/static/img/adopters/linklogis.jpg differ diff --git a/static/img/adopters/miaoyun.png b/static/img/adopters/miaoyun.png index 3c804c33d..08a382f12 100644 Binary files a/static/img/adopters/miaoyun.png and b/static/img/adopters/miaoyun.png differ diff --git a/static/img/adopters/msxf.png b/static/img/adopters/msxf.png index ef194d98d..33aaa684f 100644 Binary files a/static/img/adopters/msxf.png and b/static/img/adopters/msxf.png differ diff --git a/static/img/adopters/nankai-university.png b/static/img/adopters/nankai-university.png index 768006491..589579c52 100644 Binary files a/static/img/adopters/nankai-university.png and b/static/img/adopters/nankai-university.png differ diff --git a/static/img/adopters/northsoft.png b/static/img/adopters/northsoft.png index 42a0fc086..6eccf66cd 100644 Binary files a/static/img/adopters/northsoft.png and b/static/img/adopters/northsoft.png differ diff --git a/static/img/adopters/pingan-bank.png b/static/img/adopters/pingan-bank.png index 953bb5c56..5af0f9be3 100644 Binary files a/static/img/adopters/pingan-bank.png and b/static/img/adopters/pingan-bank.png differ diff --git a/static/img/adopters/pingan-securities.png b/static/img/adopters/pingan-securities.png index 1df89a0ed..a00d6064e 100644 Binary files a/static/img/adopters/pingan-securities.png and b/static/img/adopters/pingan-securities.png differ diff --git a/static/img/adopters/ppio.png b/static/img/adopters/ppio.png index e98bbc93a..399abc6b9 100644 Binary files a/static/img/adopters/ppio.png and b/static/img/adopters/ppio.png differ diff --git a/static/img/adopters/ragehealth.png b/static/img/adopters/ragehealth.png index 84b2d1ba0..43065a5c8 100644 Binary files a/static/img/adopters/ragehealth.png and b/static/img/adopters/ragehealth.png differ diff --git a/static/img/adopters/sangfor.jpg b/static/img/adopters/sangfor.jpg index 9f2fccca2..3dd74b225 100644 Binary files a/static/img/adopters/sangfor.jpg and b/static/img/adopters/sangfor.jpg differ diff --git a/static/img/adopters/sap.png b/static/img/adopters/sap.png index 647e917f7..59015a45c 100644 Binary files a/static/img/adopters/sap.png and b/static/img/adopters/sap.png differ diff --git a/static/img/adopters/si-tech.png b/static/img/adopters/si-tech.png index f23179582..e625d5308 100644 Binary files a/static/img/adopters/si-tech.png and b/static/img/adopters/si-tech.png differ diff --git a/static/img/adopters/sinochem.png b/static/img/adopters/sinochem.png index 8fea9ca72..cded2bcec 100644 Binary files a/static/img/adopters/sinochem.png and b/static/img/adopters/sinochem.png differ diff --git a/static/img/adopters/snow.png b/static/img/adopters/snow.png index 35b561891..99326496b 100644 Binary files a/static/img/adopters/snow.png and b/static/img/adopters/snow.png differ diff --git a/static/img/adopters/southeast-university.png b/static/img/adopters/southeast-university.png index 459a7375f..266b035c4 100644 Binary files a/static/img/adopters/southeast-university.png and b/static/img/adopters/southeast-university.png differ diff --git a/static/img/adopters/szzt.png b/static/img/adopters/szzt.png index 5541a289f..bfbd7e971 100644 Binary files a/static/img/adopters/szzt.png and b/static/img/adopters/szzt.png differ diff --git a/static/img/adopters/tongcheng-travel.png b/static/img/adopters/tongcheng-travel.png index 4476c9ca9..61c1f2f38 100644 Binary files a/static/img/adopters/tongcheng-travel.png and b/static/img/adopters/tongcheng-travel.png differ diff --git a/static/img/adopters/tum.png b/static/img/adopters/tum.png index 1fd6c0c8a..4e059436b 100644 Binary files a/static/img/adopters/tum.png and b/static/img/adopters/tum.png differ diff --git a/static/img/adopters/ucloud.png b/static/img/adopters/ucloud.png index c439d0ac2..0374f7caa 100644 Binary files a/static/img/adopters/ucloud.png and b/static/img/adopters/ucloud.png differ diff --git a/static/img/adopters/unicdata.png b/static/img/adopters/unicdata.png index 226c6c0ee..6f8326755 100644 Binary files a/static/img/adopters/unicdata.png and b/static/img/adopters/unicdata.png differ diff --git a/static/img/adopters/viettel.png b/static/img/adopters/viettel.png index d1139ce7f..aa7168486 100644 Binary files a/static/img/adopters/viettel.png and b/static/img/adopters/viettel.png differ diff --git a/static/img/adopters/weibo.png b/static/img/adopters/weibo.png index 1384972e6..509d94e86 100644 Binary files a/static/img/adopters/weibo.png and b/static/img/adopters/weibo.png differ diff --git a/static/img/adopters/weidian.png b/static/img/adopters/weidian.png index 3cfc40172..99c728e89 100644 Binary files a/static/img/adopters/weidian.png and b/static/img/adopters/weidian.png differ diff --git a/static/img/adopters/woqu.png b/static/img/adopters/woqu.png index 51e50959c..6bde99e9e 100644 Binary files a/static/img/adopters/woqu.png and b/static/img/adopters/woqu.png differ diff --git a/static/img/adopters/xuanyuan-network.png b/static/img/adopters/xuanyuan-network.png index e19d7b328..7629a9514 100644 Binary files a/static/img/adopters/xuanyuan-network.png and b/static/img/adopters/xuanyuan-network.png differ diff --git a/static/img/adopters/zstack.png b/static/img/adopters/zstack.png index 8d376029e..7c96b2f7a 100644 Binary files a/static/img/adopters/zstack.png and b/static/img/adopters/zstack.png differ diff --git a/static/img/architect.jpg b/static/img/architect.jpg index 0dfcd8063..573888560 100644 Binary files a/static/img/architect.jpg and b/static/img/architect.jpg differ diff --git a/static/img/blog-hami-at-kubecon-eu-2026-booth.png b/static/img/blog-hami-at-kubecon-eu-2026-booth.png index bfa7c0814..ab5a11192 100644 Binary files a/static/img/blog-hami-at-kubecon-eu-2026-booth.png and b/static/img/blog-hami-at-kubecon-eu-2026-booth.png differ diff --git a/static/img/blog-hami-at-kubecon-eu-2026-kubecon.png b/static/img/blog-hami-at-kubecon-eu-2026-kubecon.png index 77274f4f3..bafaee420 100644 Binary files a/static/img/blog-hami-at-kubecon-eu-2026-kubecon.png and b/static/img/blog-hami-at-kubecon-eu-2026-kubecon.png differ diff --git a/static/img/blog/flowchart.jpeg b/static/img/blog/flowchart.jpeg index 1cbe0a590..34fd31bc2 100644 Binary files a/static/img/blog/flowchart.jpeg and b/static/img/blog/flowchart.jpeg differ diff --git a/static/img/blog/hami-pod-scheduler-preparation.png b/static/img/blog/hami-pod-scheduler-preparation.png index 307779b64..35a4bf8c6 100644 Binary files a/static/img/blog/hami-pod-scheduler-preparation.png and b/static/img/blog/hami-pod-scheduler-preparation.png differ diff --git a/static/img/case-studies/ke-holdings.png b/static/img/case-studies/ke-holdings.png index 163159b9d..22dddb2dc 100644 Binary files a/static/img/case-studies/ke-holdings.png and b/static/img/case-studies/ke-holdings.png differ diff --git a/static/img/case-studies/snow-corp.webp b/static/img/case-studies/snow-corp.webp index 4214264cc..e758b57b2 100644 Binary files a/static/img/case-studies/snow-corp.webp and b/static/img/case-studies/snow-corp.webp differ diff --git a/static/img/community/wechat-assistant-qr.jpg b/static/img/community/wechat-assistant-qr.jpg index 65e77d75b..a6ff8dfbc 100644 Binary files a/static/img/community/wechat-assistant-qr.jpg and b/static/img/community/wechat-assistant-qr.jpg differ diff --git a/static/img/community/wechat-official-account-qr.jpg b/static/img/community/wechat-official-account-qr.jpg index 2487afca5..00787ad07 100644 Binary files a/static/img/community/wechat-official-account-qr.jpg and b/static/img/community/wechat-official-account-qr.jpg differ diff --git a/static/img/community/wechat-video-channel-qr.jpg b/static/img/community/wechat-video-channel-qr.jpg index 8b53e95cd..8fbd46375 100644 Binary files a/static/img/community/wechat-video-channel-qr.jpg and b/static/img/community/wechat-video-channel-qr.jpg differ diff --git a/static/img/contributors/alauda.jpg b/static/img/contributors/alauda.jpg index 3a1ea05d2..7dca7808e 100644 Binary files a/static/img/contributors/alauda.jpg and b/static/img/contributors/alauda.jpg differ diff --git a/static/img/contributors/bit.png b/static/img/contributors/bit.png index 5b2c426af..99fe1846d 100644 Binary files a/static/img/contributors/bit.png and b/static/img/contributors/bit.png differ diff --git a/static/img/contributors/caih.png b/static/img/contributors/caih.png index ed88cfb89..24f79e067 100644 Binary files a/static/img/contributors/caih.png and b/static/img/contributors/caih.png differ diff --git a/static/img/contributors/cncr.png b/static/img/contributors/cncr.png index 7facf9f1f..143b88d4b 100644 Binary files a/static/img/contributors/cncr.png and b/static/img/contributors/cncr.png differ diff --git a/static/img/contributors/glovo.png b/static/img/contributors/glovo.png index 2a2b4de86..c0fdf3d76 100644 Binary files a/static/img/contributors/glovo.png and b/static/img/contributors/glovo.png differ diff --git a/static/img/contributors/hygon.png b/static/img/contributors/hygon.png index 74b432881..108322645 100644 Binary files a/static/img/contributors/hygon.png and b/static/img/contributors/hygon.png differ diff --git a/static/img/contributors/iluvatar.png b/static/img/contributors/iluvatar.png index 5fbc8327b..03e9c7df3 100644 Binary files a/static/img/contributors/iluvatar.png and b/static/img/contributors/iluvatar.png differ diff --git a/static/img/contributors/iqiy.png b/static/img/contributors/iqiy.png index 107f6b0e7..1ab17bcf6 100644 Binary files a/static/img/contributors/iqiy.png and b/static/img/contributors/iqiy.png differ diff --git a/static/img/contributors/jd-zh.png b/static/img/contributors/jd-zh.png index 64d5e9a76..fccb2b389 100644 Binary files a/static/img/contributors/jd-zh.png and b/static/img/contributors/jd-zh.png differ diff --git a/static/img/contributors/kunlunxin.jpg b/static/img/contributors/kunlunxin.jpg index 62f1c744f..d308da39d 100644 Binary files a/static/img/contributors/kunlunxin.jpg and b/static/img/contributors/kunlunxin.jpg differ diff --git a/static/img/contributors/memverge.png b/static/img/contributors/memverge.png index 40b14de89..0d526593e 100644 Binary files a/static/img/contributors/memverge.png and b/static/img/contributors/memverge.png differ diff --git a/static/img/contributors/metax.png b/static/img/contributors/metax.png index 45d580e2e..6eba87044 100644 Binary files a/static/img/contributors/metax.png and b/static/img/contributors/metax.png differ diff --git a/static/img/contributors/mthread.png b/static/img/contributors/mthread.png index 7d3c5d9bb..d8f8f783f 100644 Binary files a/static/img/contributors/mthread.png and b/static/img/contributors/mthread.png differ diff --git a/static/img/contributors/phancy.png b/static/img/contributors/phancy.png index b004ae263..a3d517e64 100644 Binary files a/static/img/contributors/phancy.png and b/static/img/contributors/phancy.png differ diff --git a/static/img/contributors/qxzg-ai.png b/static/img/contributors/qxzg-ai.png index 09986f5d9..48fd91148 100644 Binary files a/static/img/contributors/qxzg-ai.png and b/static/img/contributors/qxzg-ai.png differ diff --git a/static/img/vllm-meetup-shanghai-2026-recap/li-mengxuan-speaking.webp b/static/img/vllm-meetup-shanghai-2026-recap/li-mengxuan-speaking.webp index 7a780067c..b32a97726 100644 Binary files a/static/img/vllm-meetup-shanghai-2026-recap/li-mengxuan-speaking.webp and b/static/img/vllm-meetup-shanghai-2026-recap/li-mengxuan-speaking.webp differ diff --git a/static/img/vllm-meetup-shanghai-2026-recap/phase0-code.webp b/static/img/vllm-meetup-shanghai-2026-recap/phase0-code.webp index 943d77310..f3dc18cbc 100644 Binary files a/static/img/vllm-meetup-shanghai-2026-recap/phase0-code.webp and b/static/img/vllm-meetup-shanghai-2026-recap/phase0-code.webp differ diff --git a/static/img/vllm-meetup-shanghai-2026-recap/phase0-overview.webp b/static/img/vllm-meetup-shanghai-2026-recap/phase0-overview.webp index 88022fa88..9e36a24b6 100644 Binary files a/static/img/vllm-meetup-shanghai-2026-recap/phase0-overview.webp and b/static/img/vllm-meetup-shanghai-2026-recap/phase0-overview.webp differ diff --git a/static/img/vllm-meetup-shanghai-2026-recap/phase1-k8s.webp b/static/img/vllm-meetup-shanghai-2026-recap/phase1-k8s.webp index 52681195d..bc1b1bae8 100644 Binary files a/static/img/vllm-meetup-shanghai-2026-recap/phase1-k8s.webp and b/static/img/vllm-meetup-shanghai-2026-recap/phase1-k8s.webp differ diff --git a/static/img/vllm-meetup-shanghai-2026-recap/phase1-overview.webp b/static/img/vllm-meetup-shanghai-2026-recap/phase1-overview.webp index a69dee6d5..6f7d20f30 100644 Binary files a/static/img/vllm-meetup-shanghai-2026-recap/phase1-overview.webp and b/static/img/vllm-meetup-shanghai-2026-recap/phase1-overview.webp differ diff --git a/static/img/vllm-meetup-shanghai-2026-recap/phase2-llmd-detail.webp b/static/img/vllm-meetup-shanghai-2026-recap/phase2-llmd-detail.webp index 35fbc44bf..bff28f237 100644 Binary files a/static/img/vllm-meetup-shanghai-2026-recap/phase2-llmd-detail.webp and b/static/img/vllm-meetup-shanghai-2026-recap/phase2-llmd-detail.webp differ diff --git a/static/img/vllm-meetup-shanghai-2026-recap/phase2-llmd.webp b/static/img/vllm-meetup-shanghai-2026-recap/phase2-llmd.webp index f5fb6cdd6..a91727b17 100644 Binary files a/static/img/vllm-meetup-shanghai-2026-recap/phase2-llmd.webp and b/static/img/vllm-meetup-shanghai-2026-recap/phase2-llmd.webp differ diff --git a/static/img/vllm-meetup-shanghai-2026-recap/phase2-mooncake-code.webp b/static/img/vllm-meetup-shanghai-2026-recap/phase2-mooncake-code.webp index 8248a26cc..9de5cb5b7 100644 Binary files a/static/img/vllm-meetup-shanghai-2026-recap/phase2-mooncake-code.webp and b/static/img/vllm-meetup-shanghai-2026-recap/phase2-mooncake-code.webp differ diff --git a/static/img/vllm-meetup-shanghai-2026-recap/phase2-mooncake.webp b/static/img/vllm-meetup-shanghai-2026-recap/phase2-mooncake.webp index b91ac08af..c2a1f20a4 100644 Binary files a/static/img/vllm-meetup-shanghai-2026-recap/phase2-mooncake.webp and b/static/img/vllm-meetup-shanghai-2026-recap/phase2-mooncake.webp differ diff --git a/static/img/vllm-meetup-shanghai-2026-recap/phase2-nccl.webp b/static/img/vllm-meetup-shanghai-2026-recap/phase2-nccl.webp index 0a86d912f..4794a7c39 100644 Binary files a/static/img/vllm-meetup-shanghai-2026-recap/phase2-nccl.webp and b/static/img/vllm-meetup-shanghai-2026-recap/phase2-nccl.webp differ diff --git a/static/img/vllm-meetup-shanghai-2026-recap/phase2-pd.webp b/static/img/vllm-meetup-shanghai-2026-recap/phase2-pd.webp index 7f56f8457..fc0b755c9 100644 Binary files a/static/img/vllm-meetup-shanghai-2026-recap/phase2-pd.webp and b/static/img/vllm-meetup-shanghai-2026-recap/phase2-pd.webp differ diff --git a/static/img/vllm-meetup-shanghai-2026-recap/phase3-architecture.webp b/static/img/vllm-meetup-shanghai-2026-recap/phase3-architecture.webp index 7881f3a81..dbb023e14 100644 Binary files a/static/img/vllm-meetup-shanghai-2026-recap/phase3-architecture.webp and b/static/img/vllm-meetup-shanghai-2026-recap/phase3-architecture.webp differ diff --git a/static/img/vllm-meetup-shanghai-2026-recap/phase3-overview.webp b/static/img/vllm-meetup-shanghai-2026-recap/phase3-overview.webp index 417d00ca8..97a84e0d8 100644 Binary files a/static/img/vllm-meetup-shanghai-2026-recap/phase3-overview.webp and b/static/img/vllm-meetup-shanghai-2026-recap/phase3-overview.webp differ diff --git a/static/img/vllm-meetup-shanghai-2026-recap/phase3-result.webp b/static/img/vllm-meetup-shanghai-2026-recap/phase3-result.webp index 4771af23a..00fbc7369 100644 Binary files a/static/img/vllm-meetup-shanghai-2026-recap/phase3-result.webp and b/static/img/vllm-meetup-shanghai-2026-recap/phase3-result.webp differ diff --git a/static/img/vllm-meetup-shanghai-2026-recap/title.webp b/static/img/vllm-meetup-shanghai-2026-recap/title.webp index 4a4775762..f7ea94b59 100644 Binary files a/static/img/vllm-meetup-shanghai-2026-recap/title.webp and b/static/img/vllm-meetup-shanghai-2026-recap/title.webp differ diff --git a/static/img/vllm-meetup-shanghai-2026-recap/vllm-meetup-group-photo.webp b/static/img/vllm-meetup-shanghai-2026-recap/vllm-meetup-group-photo.webp index d3f48ac12..6514b9b37 100644 Binary files a/static/img/vllm-meetup-shanghai-2026-recap/vllm-meetup-group-photo.webp and b/static/img/vllm-meetup-shanghai-2026-recap/vllm-meetup-group-photo.webp differ diff --git a/tutorials/labs/hami-sglang.md b/tutorials/labs/hami-sglang.md new file mode 100644 index 000000000..d88f29e50 --- /dev/null +++ b/tutorials/labs/hami-sglang.md @@ -0,0 +1,186 @@ +--- +title: "Lab 7: Run SGLang on HAMi GPU Shares" +description: "Install HAMi on a GPU cluster and schedule SGLang inference services with GPU partitioning." +sidebar_label: "Lab 7: SGLang Inference" +lab: + level: Intermediate + duration: about 45 minutes + environment: Kubernetes cluster with NVIDIA GPUs +tags: + - sglang + - inference +toc_max_heading_level: 2 +--- + +This lab demonstrates how to deploy [SGLang](https://sgl-project.github.io/), a high-performance LLM serving framework optimized for RadixAttention, on a Kubernetes cluster using HAMi for GPU memory and compute isolation. Upon completion, you will have an OpenAI-compatible model service running SGLang on a partitioned GPU. + +## Learning Objectives + +- Understand the benefits of SGLang for high-throughput inference +- Run SGLang using HAMi's `nvidia.com/gpu`, `nvidia.com/gpumem`, and `nvidia.com/gpucores` resources +- Test the SGLang OpenAI-compatible API via port forwarding + +## Lab Overview + +```mermaid +%% title: HAMi + SGLang Lab Flowchart +flowchart LR + Step1["Step 1
Check GPU Cluster"] --> Step2["Step 2
Install HAMi"] + Step2 --> Step3["Step 3
Deploy SGLang"] + Step3 --> Step4["Step 4
Test Inference"] + Step4 --> Step5["Step 5
Cleanup"] +``` + +## Deployment Architecture + +```mermaid +%% title: HAMi + SGLang Deployment Architecture +flowchart TB + Client["Client
curl / OpenAI SDK"] --> SVC["SGLang Service
port 30000"] + SVC --> P1["SGLang Pod 1
Meta-Llama-3-8B-Instruct
1 GPU slot / 24 GiB"] + SVC --> P2["SGLang Pod 2
Meta-Llama-3-8B-Instruct
1 GPU slot / 24 GiB"] + + subgraph K8S["Kubernetes GPU Cluster"] + HAMI_S["hami-scheduler"] + HAMI_D["hami-device-plugin"] + P1 + P2 + N1["GPU Node 1
NVIDIA A100"] + end + + HAMI_S --> P1 + HAMI_S --> P2 + HAMI_D --> N1 + P1 --> N1 + P2 --> N1 +``` + +## Prerequisites + +- A working Kubernetes cluster with HAMi installed +- At least 1 NVIDIA GPU node with sufficient memory for the target model +- `kubectl` connected to the cluster +- The cluster can pull SGLang images (`lmsysorg/sglang:latest`) and access Hugging Face models + +## Step 1: Create the SGLang Deployment + +We will create a Kubernetes Deployment for SGLang, requesting a specific partition of the GPU using HAMi's extended resources. + +Save the following YAML as `sglang-deployment.yaml`: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: sglang-llama3 + labels: + app: sglang +spec: + replicas: 1 + selector: + matchLabels: + app: sglang + template: + metadata: + labels: + app: sglang + spec: + containers: + - name: sglang + image: lmsysorg/sglang:latest + command: + - python3 + - "-m" + - "sglang.launch_server" + - "--model-path" + - "meta-llama/Meta-Llama-3-8B-Instruct" + - "--host" + - "0.0.0.0" + - "--port" + - "30000" + ports: + - containerPort: 30000 + resources: + limits: + nvidia.com/gpu: 1 + nvidia.com/gpumem: 24000 + nvidia.com/gpucores: 50 + env: + - name: HUGGING_FACE_HUB_TOKEN + value: "YOUR_HF_TOKEN" # Replace with your Hugging Face token +``` + +**Key Resource Constraints:** + +- `nvidia.com/gpu: 1`: Requests 1 virtual GPU slot. +- `nvidia.com/gpumem: 24000`: Allocates exactly 24GB of GPU memory to the SGLang worker. +- `nvidia.com/gpucores: 50`: Allocates 50% of the physical GPU's compute capacity. + +Apply the deployment: + +```bash +kubectl apply -f sglang-deployment.yaml +``` + +## Step 2: Create the Service + +Expose the SGLang deployment internally using a ClusterIP service. + +Save as `sglang-service.yaml`: + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: sglang-service +spec: + selector: + app: sglang + ports: + - protocol: TCP + port: 30000 + targetPort: 30000 +``` + +Apply the service: + +```bash +kubectl apply -f sglang-service.yaml +``` + +## Step 3: Verify Inference + +Wait for the SGLang pod to become `Running`. Since it needs to download the Llama 3 weights, this might take several minutes depending on your network speed. + +```bash +kubectl get pods -l app=sglang -w +``` + +Once running, port-forward the service to your local machine: + +```bash +kubectl port-forward svc/sglang-service 30000:30000 +``` + +Open a new terminal and test the OpenAI-compatible endpoint: + +```bash +curl http://localhost:30000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "meta-llama/Meta-Llama-3-8B-Instruct", + "messages": [{"role": "user", "content": "Explain GPU virtualization in one sentence."}], + "temperature": 0.7 + }' +``` + +If successful, SGLang will return a generated response powered by your HAMi-partitioned GPU! + +## Step 4: Cleanup + +To remove the lab resources from your cluster, run: + +```bash +kubectl delete -f sglang-service.yaml +kubectl delete -f sglang-deployment.yaml +```