Skip to content

docs: update GPU partitioning manifests for AWS EC2 #757

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Creativeklvn wants to merge 1 commit into
base: master
Choose a base branch
from
+135 −2
Open

docs: update GPU partitioning manifests for AWS EC2 #757

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 135 additions & 2 deletions tutorials/labs/gpu-partitioning.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ tags:
toc_max_heading_level: 2
---

import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem';

This lab continues from [Lab 1](./online-install.md). You have one physical Tesla T4 with 15360 MiB of VRAM. In this lab you will run multiple Pods on that single card, each with its own enforced VRAM and compute limit, and verify that the isolation is real: a Pod that tries to allocate past its slice gets a CUDA OOM while its neighbors keep running.

Every command and output in this lab was captured from a live cluster built with Lab 1 (HAMi v2.9.0, GPU Operator v25.3.0, Kubernetes v1.34).
Expand Down Expand Up @@ -92,11 +94,50 @@ spec:

`gpumem-pod-b.yaml` is identical except for the name. Apply both:

<Tabs groupId="cloud-provider">
<TabItem value="aws" label="AWS">

Open a seperate terminal.

Use the same PEM key file and `NODE_PUBLIC_IP` defined in **Step 1.3: SSH Into the Instance** to copy the YAML files to the EC2 instance.

```bash
scp -i ~/<your-pem-key-file-path>/hami-eks.pem \
tutorials/labs/examples/03-gpu-partitioning/gpumem-pod-a.yaml \
tutorials/labs/examples/03-gpu-partitioning/gpumem-pod-b.yaml \
ubuntu@<your-vm-public-ip>:/home/ubuntu/
```
Comment thread
Creativeklvn marked this conversation as resolved.

Go back to your on EC2 terminal.
Comment on lines +100 to +111

@coderabbitai coderabbitai Bot Aug 11, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Correct the repeated terminal instructions.

Replace seperate with separate. Replace Go back to your on EC2 terminal with Return to the EC2 terminal.

Also applies to: 219-229, 323-333

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tutorials/labs/gpu-partitioning.md` around lines 100 - 111, Correct the
repeated terminal instructions in all affected sections: change “seperate” to
“separate” and replace “Go back to your on EC2 terminal” with “Return to the EC2
terminal.”


```bash
ls /home/ubuntu/
```

Expected output, you should see:

```plaintext
gpumem-pod-a.yaml
gpumem-pod-b.yaml
```

```bash
kubectl apply -f /home/ubuntu/gpumem-pod-a.yaml -f /home/ubuntu/gpumem-pod-b.yaml
kubectl get pods gpumem-pod-a gpumem-pod-b -o wide
```

</TabItem>
<TabItem value="gcp" label="GCP">

```bash
kubectl apply -f tutorials/labs/examples/03-gpu-partitioning/gpumem-pod-a.yaml -f tutorials/labs/examples/03-gpu-partitioning/gpumem-pod-b.yaml

kubectl get pods gpumem-pod-a gpumem-pod-b -o wide
```

</TabItem>
Comment thread
coderabbitai[bot] marked this conversation as resolved.
</Tabs>

```plaintext
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
gpumem-pod-a 1/1 Running 0 27s 10.244.218.156 hami-workshop <none> <none>
Expand Down Expand Up @@ -173,10 +214,35 @@ spec:
nvidia.com/gpumem: 4000 # limit this Pod to 4000 MiB of VRAM
```

<Tabs groupId="cloud-provider">
<TabItem value="aws" label="AWS">

Open a seperate terminal

Use the same PEM key file and `NODE_PUBLIC_IP` defined in **Step 1.3: SSH Into the Instance** to copy the YAML files to the EC2 instance.

```bash
scp -i ~/<your-pem-key-file-path>/hami-eks.pem \
tutorials/labs/examples/03-gpu-partitioning/oom-test-pod.yaml \
ubuntu@<your-vm-public-ip>:/home/ubuntu/
```

Go back to your on EC2 terminal

```bash
kubectl apply -f /home/ubuntu/oom-test-pod.yaml
```

</TabItem>
<TabItem value="gcp" label="GCP">

```bash
kubectl apply -f tutorials/labs/examples/03-gpu-partitioning/oom-test-pod.yaml
```

</TabItem>
</Tabs>

While the image pulls, watch the HAMi scheduler make its decision:

```bash
Expand All @@ -191,17 +257,29 @@ kubectl describe pod oom-test-pod | tail -3

> `FilteringSucceed` shows the scheduler scoring nodes (here `hami-workshop:7.21`), and `BindingSucceed` shows it binding the Pod. These events come from hami-scheduler, not the default scheduler. Note it found a fit even though two Pods already occupy the GPU: 8000 of 15360 MiB are reserved, so a third 4000 MiB slice still fits.

Wait for the Pod to complete, then read its logs:
Wait for the Pod to be completed, then read its logs:

```bash
kubectl get pods oom-test-pod -o wide
```

Expected output:

```plaintext
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
oom-test-pod 0/1 Completed 0 6m17s 10.244.109.31 ip-172-31-8-251 <none> <none>
```

```bash
kubectl logs oom-test-pod | tail -8
```

```plaintext
Allocated 2048 MiB
Allocated 2560 MiB
Allocated 3072 MiB
Allocated 3584 MiB
[HAMI-core ERROR (pid:1 thread=... allocator.c:52)]: Device 0 OOM 4399824896 / 4194304000
[HAMI-core ERROR (pid:1 thread=128984570163776 allocator.c:52)]: Device 0 OOM 4399824896 / 4194304000
Hit the limit after 3584 MiB:
CUDA out of memory
```
Expand Down Expand Up @@ -239,10 +317,57 @@ resources:
nvidia.com/gpucores: 30 # limit this Pod to 30% of GPU compute
```

<Tabs groupId="cloud-provider">
<TabItem value="aws" label="AWS">

Open a seperate terminal

Use the same PEM key file and `NODE_PUBLIC_IP` defined in **Step 1.3: SSH Into the Instance** to copy the YAML files to the EC2 instance.

```bash
scp -i ~/<your-pem-key-file-path>/hami-eks.pem \
tutorials/labs/examples/03-gpu-partitioning/gpucores-pod.yaml \
ubuntu@<your-vm-public-ip>:/home/ubuntu/
```

Go back to your on EC2 terminal

```bash
ls /home/ubuntu/
```

Expected output, you should see:

```plaintext
gpucores-pod.yaml gpumem-pod-b.yaml
gpumem-pod-a.yaml oom-test-pod.yaml
```

```bash
kubectl apply -f /home/ubuntu/gpucores-pod.yaml
```

</TabItem>
<TabItem value="gcp" label="GCP">

```bash
kubectl apply -f tutorials/labs/examples/03-gpu-partitioning/gpucores-pod.yaml
```

</TabItem>
</Tabs>

```bash
kubectl get pod gpucores-pod -o wide
```

Expected output:

```plaintext
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
gpucores-pod 1/1 Running 0 2m 10.244.14.160 ip-172-31-9-124 <none> <none>
```

Check the environment HAMi injected into the container:

```bash
Expand Down Expand Up @@ -299,6 +424,14 @@ kubectl exec -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 -c
>
> Without the `force` env, you would see this same workload run at 100% utilization on an idle card, which is intentional: HAMi gives idle capacity away rather than wasting it.

(Optional) Access the WebUI via port forwarding:

```bash
kubectl port-forward --address 0.0.0.0 service/my-hami-webui 3000:3000 --namespace=kube-system
```

Visit `http://<your-vm-public-ip-address>:3000` to open the HAMi WebUI.

Comment on lines +427 to +434

@coderabbitai coderabbitai Bot Aug 11, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Do not expose the WebUI on all VM interfaces by default.

--address 0.0.0.0 binds the port forward to every EC2 interface. The public-IP URL then requires an inbound security-group rule and exposes port 3000 to every source allowed by that rule. Prefer an SSH tunnel with localhost binding. If public access is required, restrict the rule to the student's IP and document the authentication and HTTP exposure.

Safer default 
-kubectl port-forward --address 0.0.0.0 service/my-hami-webui 3000:3000 --namespace=kube-system
+kubectl port-forward service/my-hami-webui 3000:3000 --namespace=kube-system

Access the remote port through an SSH -L tunnel.

🧰 Tools
🪛 LanguageTool

[grammar] ~433-~433: Ensure spelling is correct
Context: ...blic-ip-address>:3000` to open the HAMi WebUI. ## Step 6: Cleanup ```bash kubectl delete ...

(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tutorials/labs/gpu-partitioning.md` around lines 427 - 434, Update the WebUI
access instructions to remove the default kubectl port-forward binding to
0.0.0.0 and use localhost binding through an SSH -L tunnel instead. Keep the
WebUI reachable via the remote machine without exposing port 3000 on all VM
interfaces; if public access remains documented, restrict it to the student’s IP
and include authentication and HTTP-exposure requirements.

## Step 6: Cleanup

```bash
Expand Down