diff --git a/tutorials/labs/gpu-partitioning.md b/tutorials/labs/gpu-partitioning.md index 654f3fefb..f76b740ab 100644 --- a/tutorials/labs/gpu-partitioning.md +++ b/tutorials/labs/gpu-partitioning.md @@ -15,6 +15,8 @@ tags: toc_max_heading_level: 2 --- +import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; + This lab continues from [Lab 1](./online-install.md). You have one physical Tesla T4 with 15360 MiB of VRAM. In this lab you will run multiple Pods on that single card, each with its own enforced VRAM and compute limit, and verify that the isolation is real: a Pod that tries to allocate past its slice gets a CUDA OOM while its neighbors keep running. Every command and output in this lab was captured from a live cluster built with Lab 1 (HAMi v2.9.0, GPU Operator v25.3.0, Kubernetes v1.34). @@ -92,11 +94,50 @@ spec: `gpumem-pod-b.yaml` is identical except for the name. Apply both: + + + +Open a seperate terminal. + +Use the same PEM key file and `NODE_PUBLIC_IP` defined in **Step 1.3: SSH Into the Instance** to copy the YAML files to the EC2 instance. + +```bash +scp -i ~//hami-eks.pem \ + tutorials/labs/examples/03-gpu-partitioning/gpumem-pod-a.yaml \ + tutorials/labs/examples/03-gpu-partitioning/gpumem-pod-b.yaml \ + ubuntu@:/home/ubuntu/ +``` + +Go back to your on EC2 terminal. + +```bash +ls /home/ubuntu/ +``` + +Expected output, you should see: + +```plaintext +gpumem-pod-a.yaml +gpumem-pod-b.yaml +``` + +```bash +kubectl apply -f /home/ubuntu/gpumem-pod-a.yaml -f /home/ubuntu/gpumem-pod-b.yaml +kubectl get pods gpumem-pod-a gpumem-pod-b -o wide +``` + + + + ```bash kubectl apply -f tutorials/labs/examples/03-gpu-partitioning/gpumem-pod-a.yaml -f tutorials/labs/examples/03-gpu-partitioning/gpumem-pod-b.yaml + kubectl get pods gpumem-pod-a gpumem-pod-b -o wide ``` + + + ```plaintext NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES gpumem-pod-a 1/1 Running 0 27s 10.244.218.156 hami-workshop @@ -173,10 +214,35 @@ spec: nvidia.com/gpumem: 4000 # limit this Pod to 4000 MiB of VRAM ``` + + + +Open a seperate terminal + +Use the same PEM key file and `NODE_PUBLIC_IP` defined in **Step 1.3: SSH Into the Instance** to copy the YAML files to the EC2 instance. + +```bash +scp -i ~//hami-eks.pem \ + tutorials/labs/examples/03-gpu-partitioning/oom-test-pod.yaml \ + ubuntu@:/home/ubuntu/ +``` + +Go back to your on EC2 terminal + +```bash +kubectl apply -f /home/ubuntu/oom-test-pod.yaml +``` + + + + ```bash kubectl apply -f tutorials/labs/examples/03-gpu-partitioning/oom-test-pod.yaml ``` + + + While the image pulls, watch the HAMi scheduler make its decision: ```bash @@ -191,17 +257,29 @@ kubectl describe pod oom-test-pod | tail -3 > `FilteringSucceed` shows the scheduler scoring nodes (here `hami-workshop:7.21`), and `BindingSucceed` shows it binding the Pod. These events come from hami-scheduler, not the default scheduler. Note it found a fit even though two Pods already occupy the GPU: 8000 of 15360 MiB are reserved, so a third 4000 MiB slice still fits. -Wait for the Pod to complete, then read its logs: +Wait for the Pod to be completed, then read its logs: + +```bash +kubectl get pods oom-test-pod -o wide +``` + +Expected output: + +```plaintext +NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +oom-test-pod 0/1 Completed 0 6m17s 10.244.109.31 ip-172-31-8-251 +``` ```bash kubectl logs oom-test-pod | tail -8 ``` ```plaintext +Allocated 2048 MiB Allocated 2560 MiB Allocated 3072 MiB Allocated 3584 MiB -[HAMI-core ERROR (pid:1 thread=... allocator.c:52)]: Device 0 OOM 4399824896 / 4194304000 +[HAMI-core ERROR (pid:1 thread=128984570163776 allocator.c:52)]: Device 0 OOM 4399824896 / 4194304000 Hit the limit after 3584 MiB: CUDA out of memory ``` @@ -239,10 +317,57 @@ resources: nvidia.com/gpucores: 30 # limit this Pod to 30% of GPU compute ``` + + + +Open a seperate terminal + +Use the same PEM key file and `NODE_PUBLIC_IP` defined in **Step 1.3: SSH Into the Instance** to copy the YAML files to the EC2 instance. + +```bash +scp -i ~//hami-eks.pem \ + tutorials/labs/examples/03-gpu-partitioning/gpucores-pod.yaml \ + ubuntu@:/home/ubuntu/ +``` + +Go back to your on EC2 terminal + +```bash +ls /home/ubuntu/ +``` + +Expected output, you should see: + +```plaintext +gpucores-pod.yaml gpumem-pod-b.yaml +gpumem-pod-a.yaml oom-test-pod.yaml +``` + +```bash +kubectl apply -f /home/ubuntu/gpucores-pod.yaml +``` + + + + ```bash kubectl apply -f tutorials/labs/examples/03-gpu-partitioning/gpucores-pod.yaml ``` + + + +```bash +kubectl get pod gpucores-pod -o wide +``` + +Expected output: + +```plaintext +NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +gpucores-pod 1/1 Running 0 2m 10.244.14.160 ip-172-31-9-124 +``` + Check the environment HAMi injected into the container: ```bash @@ -299,6 +424,14 @@ kubectl exec -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 -c > > Without the `force` env, you would see this same workload run at 100% utilization on an idle card, which is intentional: HAMi gives idle capacity away rather than wasting it. +(Optional) Access the WebUI via port forwarding: + +```bash +kubectl port-forward --address 0.0.0.0 service/my-hami-webui 3000:3000 --namespace=kube-system +``` + +Visit `http://:3000` to open the HAMi WebUI. + ## Step 6: Cleanup ```bash