docs(tutorials): add SGLang lab and overhaul troubleshooting

docs/troubleshooting/troubleshooting.md

@@ -1,14 +1,55 @@
 ---
 title: Troubleshooting
+sidebar_label: Troubleshooting Guide
 ---
 
-## GPU Memory Limit Not Enforced {#gpu-memory-limit-not-enforced}
+This guide covers common issues you might encounter when installing, scheduling, and running workloads with HAMi. It is broken down into three main sections: Installation, Scheduling, and Runtime.
 
-If a container exceeds its `nvidia.com/gpumem` limit, check the following causes:
+## Diagnostic Flowchart
+
+Use this flowchart to quickly identify where your issue might be occurring:
+
+```mermaid
+%% title: HAMi Troubleshooting Decision Tree
+flowchart TD
+    A[Pod not running as expected] --> B{What is the Pod status?}
+    B -->|Pending| C[Scheduling Issues]
+    B -->|CrashLoopBackOff / Error| D[Runtime Issues]
+    B -->|Running but no GPU| E[Installation / Device Plugin Issues]
+
+    C --> F{Are GPUs available?}
+    F -->|No| G[Check node capacity and HAMi limits]
+    F -->|Yes| H[Check nodeSelector / Scheduler logs]
+
+    D --> I{Is it an OOM error?}
+    I -->|Yes| J[Check nvidia.com/gpumem limits]
+    I -->|No| K[Check libvgpu.so preload]
+
+    E --> L[Check hami-device-plugin logs]
+```
+
+---
+
+## 1. Installation Issues
+
+Installation issues usually manifest as the `hami-device-plugin` pods crashing or failing to register GPUs to the Kubernetes nodes.
+
+### Device Plugin Fails to Start
+
+- Since v2.3.10, HAMi has changed the `device-plugin` environment variable name from `NodeName` to `NODE_NAME`. If you are using an image version earlier than v2.3.10, the `device-plugin` may fail to start.
+
+  To resolve this issue, you have two options:
+  - Manually edit the DaemonSet using `kubectl edit daemonset` and update the environment variable from `NodeName` to `NODE_NAME`.
+  - Upgrade the `device-plugin` image to the latest version using Helm:
+
+    ```bash
+    helm upgrade hami hami/hami -n kube-system
+    ```
+
+### Container Runtime Configuration
+
+If the `hami-device-plugin` is running but your nodes don't show `nvidia.com/gpu` resources, verify your containerd configuration:
 
-- **`CUDA_DISABLE_CONTROL=true` is set** - disables HAMi-core enforcement entirely. Remove it from production workloads.
-- **Docker-in-Docker (DinD)** - inner containers do not inherit the `/etc/ld.so.preload` hostPath mount. HAMi enforcement does not apply inside DinD.
-- **Direct driver API usage** - workloads calling NVML or the CUDA Driver API directly bypass `libvgpu.so`.
 - **`nvidia-container-runtime` not set as default** - verify with:
 
   ```bash
@@ -17,19 +58,52 @@ If a container exceeds its `nvidia.com/gpumem` limit, check the following causes
 
   The output must show `nvidia`. If not, follow the [Prerequisites](../installation/online-installation) guide.
 
+---
+
+## 2. Scheduling Issues
+
+Scheduling issues occur when your Pods remain in the `Pending` state.
+
+### Pods Stuck in Pending
+
+- **NodeName unsupported:** Tasks with the `nodeName` field cannot be scheduled at the moment; please use `nodeSelector` instead.
+- **A100 MIG restrictions:** Currently, A100 MIG can be supported in only "none" and "mixed" modes.
+- **Check Scheduler Logs:** Use the following command to view why the HAMi scheduler rejected a pod:
+
+  ```bash
+  kubectl logs -n kube-system -l component=hami-scheduler
+  ```
+
+---
+
+## 3. Runtime Issues
+
+Runtime issues occur after the Pod is scheduled, but it fails to execute correctly or doesn't respect isolation limits.
+
+### GPU Memory Limit Not Enforced {#gpu-memory-limit-not-enforced}
+
+If a container exceeds its `nvidia.com/gpumem` limit, check the following causes:
+
+- **`CUDA_DISABLE_CONTROL=true` is set** - disables HAMi-core enforcement entirely. Remove it from production workloads.
+- **Docker-in-Docker (DinD)** - inner containers do not inherit the `/etc/ld.so.preload` hostPath mount. HAMi enforcement does not apply inside DinD.
+- **Direct driver API usage** - workloads calling NVML or the CUDA Driver API directly bypass `libvgpu.so`.
 - If you don’t explicitly request vGPUs when using the device plugin with NVIDIA images, all GPUs on the host may be exposed to your container.
-- Currently, A100 MIG can be supported in only "none" and "mixed" modes.
-- Tasks with the "nodeName" field cannot be scheduled at the moment; please use "nodeSelector" instead.
+
+### Unsupported Workloads
+
 - Only computing tasks are currently supported; video codec processing is not supported.
-- Since v2.3.10, HAMi has changed the `device-plugin` environment variable name from `NodeName` to `NODE_NAME`. If you are using an image version earlier than v2.3.10, the `device-plugin` may fail to start.
 
-  To resolve this issue, you have two options:
+## Diagnostic Cheat Sheet
 
-  - Manually edit the DaemonSet using `kubectl edit daemonset` and update the environment variable from `NodeName` to `NODE_NAME`.
-  - Upgrade the `device-plugin` image to the latest version using Helm:
+Here are some helpful commands for diagnosing HAMi clusters:
 
-    ```bash
-    helm upgrade hami hami/hami -n kube-system
-    ```
+```bash
+# Check if HAMi resources are allocatable on a node
+kubectl get node <node-name> -o yaml | grep nvidia.com
+
+# Check the logs of the HAMi device plugin
+kubectl logs -n kube-system -l app.kubernetes.io/name=hami
 
-    This will apply the fix automatically.
+# Verify if the libvgpu library is mounted inside a running pod
+kubectl exec -it <pod-name> -- ls -l /usr/local/vgpu/libvgpu.so
+```

sidebars-tutorials.js

@@ -41,6 +41,11 @@ module.exports = {
           id: "labs/hami-vllm",
           customProps: { level: "Intermediate", duration: "about 45 minutes" },
         },
+        {
+          type: "doc",
+          id: "labs/hami-sglang",
+          customProps: { level: "Intermediate", duration: "about 45 minutes" },
+        },
         {
           type: "doc",
           id: "labs/hami-isolation-k3s",

tutorials/labs/hami-sglang.md

@@ -0,0 +1,186 @@
+---
+title: "Lab 7: Run SGLang on HAMi GPU Shares"
+description: "Install HAMi on a GPU cluster and schedule SGLang inference services with GPU partitioning."
+sidebar_label: "Lab 7: SGLang Inference"
+lab:
+  level: Intermediate
+  duration: about 45 minutes
+  environment: Kubernetes cluster with NVIDIA GPUs
+tags:
+  - sglang
+  - inference
+toc_max_heading_level: 2
+---
+
+This lab demonstrates how to deploy [SGLang](https://sgl-project.github.io/), a high-performance LLM serving framework optimized for RadixAttention, on a Kubernetes cluster using HAMi for GPU memory and compute isolation. Upon completion, you will have an OpenAI-compatible model service running SGLang on a partitioned GPU.
+
+## Learning Objectives
+
+- Understand the benefits of SGLang for high-throughput inference
+- Run SGLang using HAMi's `nvidia.com/gpu`, `nvidia.com/gpumem`, and `nvidia.com/gpucores` resources
+- Test the SGLang OpenAI-compatible API via port forwarding
+
+## Lab Overview
+
+```mermaid
+%% title: HAMi + SGLang Lab Flowchart
+flowchart LR
+    Step1["Step 1<br/>Check GPU Cluster"] --> Step2["Step 2<br/>Install HAMi"]
+    Step2 --> Step3["Step 3<br/>Deploy SGLang"]
+    Step3 --> Step4["Step 4<br/>Test Inference"]
+    Step4 --> Step5["Step 5<br/>Cleanup"]
+```
+
+## Deployment Architecture
+
+```mermaid
+%% title: HAMi + SGLang Deployment Architecture
+flowchart TB
+    Client["Client<br/>curl / OpenAI SDK"] --> SVC["SGLang Service<br/>port 30000"]
+    SVC --> P1["SGLang Pod 1<br/>Meta-Llama-3-8B-Instruct<br/>1 GPU slot / 24 GiB"]
+    SVC --> P2["SGLang Pod 2<br/>Meta-Llama-3-8B-Instruct<br/>1 GPU slot / 24 GiB"]
+
+    subgraph K8S["Kubernetes GPU Cluster"]
+      HAMI_S["hami-scheduler"]
+      HAMI_D["hami-device-plugin"]
+      P1
+      P2
+      N1["GPU Node 1<br/>NVIDIA A100"]
+    end
+
+    HAMI_S --> P1
+    HAMI_S --> P2
+    HAMI_D --> N1
+    P1 --> N1
+    P2 --> N1
+```
+
+## Prerequisites
+
+- A working Kubernetes cluster with HAMi installed
+- At least 1 NVIDIA GPU node with sufficient memory for the target model
+- `kubectl` connected to the cluster
+- The cluster can pull SGLang images (`lmsysorg/sglang:latest`) and access Hugging Face models
+
+## Step 1: Create the SGLang Deployment
+
+We will create a Kubernetes Deployment for SGLang, requesting a specific partition of the GPU using HAMi's extended resources.
+
+Save the following YAML as `sglang-deployment.yaml`:
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: sglang-llama3
+  labels:
+    app: sglang
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: sglang
+  template:
+    metadata:
+      labels:
+        app: sglang
+    spec:
+      containers:
+        - name: sglang
+          image: lmsysorg/sglang:latest
+          command:
+            - python3
+            - "-m"
+            - "sglang.launch_server"
+            - "--model-path"
+            - "meta-llama/Meta-Llama-3-8B-Instruct"
+            - "--host"
+            - "0.0.0.0"
+            - "--port"
+            - "30000"
+          ports:
+            - containerPort: 30000
+          resources:
+            limits:
+              nvidia.com/gpu: 1
+              nvidia.com/gpumem: 24000
+              nvidia.com/gpucores: 50
+          env:
+            - name: HUGGING_FACE_HUB_TOKEN
+              value: "YOUR_HF_TOKEN" # Replace with your Hugging Face token
+```
+
+**Key Resource Constraints:**
+
+- `nvidia.com/gpu: 1`: Requests 1 virtual GPU slot.
+- `nvidia.com/gpumem: 24000`: Allocates exactly 24GB of GPU memory to the SGLang worker.
+- `nvidia.com/gpucores: 50`: Allocates 50% of the physical GPU's compute capacity.
+
+Apply the deployment:
+
+```bash
+kubectl apply -f sglang-deployment.yaml
+```
+
+## Step 2: Create the Service
+
+Expose the SGLang deployment internally using a ClusterIP service.
+
+Save as `sglang-service.yaml`:
+
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: sglang-service
+spec:
+  selector:
+    app: sglang
+  ports:
+    - protocol: TCP
+      port: 30000
+      targetPort: 30000
+```
+
+Apply the service:
+
+```bash
+kubectl apply -f sglang-service.yaml
+```
+
+## Step 3: Verify Inference
+
+Wait for the SGLang pod to become `Running`. Since it needs to download the Llama 3 weights, this might take several minutes depending on your network speed.
+
+```bash
+kubectl get pods -l app=sglang -w
+```
+
+Once running, port-forward the service to your local machine:
+
+```bash
+kubectl port-forward svc/sglang-service 30000:30000
+```
+
+Open a new terminal and test the OpenAI-compatible endpoint:
+
+```bash
+curl http://localhost:30000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "meta-llama/Meta-Llama-3-8B-Instruct",
+    "messages": [{"role": "user", "content": "Explain GPU virtualization in one sentence."}],
+    "temperature": 0.7
+  }'
+```
+
+If successful, SGLang will return a generated response powered by your HAMi-partitioned GPU!
+
+## Step 4: Cleanup
+
+To remove the lab resources from your cluster, run:
+
+```bash
+kubectl delete -f sglang-service.yaml
+kubectl delete -f sglang-deployment.yaml
+```