Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,21 @@
# CLAUDE.md

Check warning on line 1 in CLAUDE.md

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

CLAUDE.md#L1

Try to keep the Coleman–Liau Index grade (13.67) below 9.

Check warning on line 1 in CLAUDE.md

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

CLAUDE.md#L1

Try to keep the Flesch–Kincaid grade level (8.15) below 8.

Check warning on line 1 in CLAUDE.md

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

CLAUDE.md#L1

Try to keep the Flesch reading ease score (52.67) above 70.

Check warning on line 1 in CLAUDE.md

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

CLAUDE.md#L1

Try to keep the Automated Readability Index (9.31) below 8.

Check warning on line 1 in CLAUDE.md

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

CLAUDE.md#L1

Try to keep the LIX score (41.54) below 35.

Check warning on line 1 in CLAUDE.md

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

CLAUDE.md#L1

'CLAUDE.md' should use sentence-style capitalization.

This file provides guidance to Claude Code when working with this repository.

## Project Overview

Check warning on line 5 in CLAUDE.md

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

CLAUDE.md#L5

'Project Overview' should use sentence-style capitalization.

This is the Runpod documentation site, built with [Mintlify](https://mintlify.com/). The documentation covers Runpod's cloud GPU platform: Serverless endpoints, Pods, Flash SDK, storage, and APIs.

Check warning on line 7 in CLAUDE.md

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

CLAUDE.md#L7

'Serverless' should be in lowercase.

## Quick Reference

Check warning on line 9 in CLAUDE.md

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

CLAUDE.md#L9

'Quick Reference' should use sentence-style capitalization.

| Topic | File |
|-------|------|
| Directory structure, navigation, snippets, tooltips | [.claude/architecture.md](.claude/architecture.md) |
| Writing style, capitalization, terminology | [.claude/style-guide.md](.claude/style-guide.md) |
| Running and writing documentation tests | [.claude/testing.md](.claude/testing.md) |
| Local dev, linting, publishing workflow | [.claude/development.md](.claude/development.md) |
| Directory structure, navigation, snippets, tooltips | `.claude/architecture.md` |
| Writing style, capitalization, terminology | `.claude/style-guide.md` |
| Running and writing documentation tests | `.claude/testing.md` |
| Local dev, linting, publishing workflow | `.claude/development.md` |

## Key Commands

Check warning on line 18 in CLAUDE.md

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

CLAUDE.md#L18

'Key Commands' should use sentence-style capitalization.

```bash
mintlify dev # Start local dev server
Expand All @@ -27,17 +27,17 @@

**Claude should continuously learn and improve these docs.**

If you discover something that would be useful for future sessions, ask me:

Check warning on line 30 in CLAUDE.md

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

CLAUDE.md#L30

Avoid first-person pronouns such as 'me'.
> "I noticed [insight]. Would you like me to add this to `.claude/[appropriate-file].md`?"

Check warning on line 31 in CLAUDE.md

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

CLAUDE.md#L31

Avoid first-person pronouns such as 'me'.

Examples of things worth capturing:
- Patterns that work well (or don't) in this codebase

Check warning on line 34 in CLAUDE.md

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

CLAUDE.md#L34

Use parentheses judiciously.
- Common mistakes to avoid
- Useful commands or workflows discovered during tasks
- Clarifications about how Runpod products work

## Terminology Quick Reference

Check warning on line 39 in CLAUDE.md

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

CLAUDE.md#L39

'Terminology Quick Reference' should use sentence-style capitalization.

**Capitalize:** Runpod, Pods, Serverless, Hub, Instant Clusters, Flash, Secure Cloud, Community Cloud, Public Endpoint

Check warning on line 41 in CLAUDE.md

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

CLAUDE.md#L41

'Runpod' should be in lowercase.

Check warning on line 41 in CLAUDE.md

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

CLAUDE.md#L41

Use 'Google Cloud Platform' or 'GCP' instead of 'Cloud'.

Check warning on line 41 in CLAUDE.md

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

CLAUDE.md#L41

Use 'Google Cloud Platform' or 'GCP' instead of 'Cloud'.

**Lowercase:** endpoint, worker, template, handler, network volume, data center, cluster, fine-tune, repo
3 changes: 2 additions & 1 deletion instant-clusters/ray-vllm.mdx
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
---

Check warning on line 1 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L1

Try to keep the LIX score (35.36) below 35.

Check warning on line 1 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L1

Try to keep the Flesch reading ease score (64.10) above 70.

Check warning on line 1 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L1

Try to keep the Coleman–Liau Index grade (9.58) below 9.
title: "Deploy an Instant Cluster with Ray and vLLM"
sidebarTitle: "Ray + vLLM"
description: "Run distributed inference across multiple nodes using Ray and vLLM on an Instant Cluster."
tag: BETA

Check warning on line 5 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L5

Spell out 'BETA', if it's unfamiliar to the audience.
---

This tutorial shows how to use Instant Clusters with Ray to run distributed inference on large language models. By combining Ray's cluster management with vLLM's tensor and pipeline parallelism, you can serve models that exceed the memory of a single node — for example, a 70B parameter model across multiple 8×H100 pods.

Check warning on line 8 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L8

Don't put a space before or after a dash.

Check warning on line 8 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L8

Put a nonbreaking space between the number and the unit in '70B'.

Ray handles the cluster topology; vLLM uses it to split the model across GPUs both within each node (tensor parallelism) and across nodes (pipeline parallelism).

Check warning on line 10 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L10

Use semicolons judiciously.

Check warning on line 10 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L10

Use parentheses judiciously.

Check warning on line 10 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L10

Use parentheses judiciously.

<Note>
Distributed inference with Ray and vLLM on Instant Clusters is currently in beta. Join our [Discord](https://discord.gg/runpod) to provide feedback and get support.
Expand All @@ -22,23 +22,23 @@

---

## Step 1: Deploy an Instant Cluster

Check warning on line 25 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L25

'Step 1: Deploy an Instant Cluster' should use sentence-style capitalization.

1. Open the [Instant Clusters page](https://console.runpod.io/instant-clusters).
2. Click **Create Cluster**.
3. Name your cluster and configure it. For this walkthrough, set **Pod Count** to **2** and select **8× H100 SXM GPUs** per pod. Use the **Runpod PyTorch** template as your base image.

Check warning on line 29 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L29

Spell out 'SXM', if it's unfamiliar to the audience.

<Note>
Increase `/dev/shm` when configuring your pod. The default (64 MB) is too small for large tensor-parallel workloads. Set it to at least 8 GB. In the pod configuration, add the environment variable `MALLOC_ARENA_MAX=1` and set `--shm-size` to `8g` in your Docker run options.
</Note>

4. Click **Deploy Cluster**. You are redirected to the Instant Clusters page.

Check warning on line 35 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L35

In general, use active voice instead of passive voice ('are redirected').

---

## Step 2: Start the Ray head on pod-0

Check warning on line 39 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L39

'Step 2: Start the Ray head on pod-0' should use sentence-style capitalization.

The first pod (`CLUSTERNAME-pod-0`) runs the Ray head node. All other pods connect to it as workers.

Check warning on line 41 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L41

Use parentheses judiciously.

1. Click your cluster to expand the pod list.
2. Click **CLUSTERNAME-pod-0**, then click **Connect → Web Terminal**.
Expand All @@ -55,7 +55,7 @@
bash ray-vllm-cluster/head.sh
```

The script sets the correct NIC address and starts Ray:

Check warning on line 58 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L58

Spell out 'NIC', if it's unfamiliar to the audience.

```bash
# head.sh (excerpt — see full script in the repo)
Expand All @@ -76,7 +76,7 @@

## Step 3: Join the worker pods to the cluster

Repeat this for each remaining pod in the cluster (`pod-1`, `pod-2`, …).

Check warning on line 79 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L79

Use parentheses judiciously.

1. In the Instant Clusters page, click the next pod and open its **Web Terminal**.
2. Clone the same scripts:
Expand Down Expand Up @@ -111,7 +111,7 @@
--num-gpus=$NUM_TRAINERS
```

`$MASTER_ADDR` is injected automatically by Runpod into all pods in the cluster — it resolves to `pod-0`.

Check warning on line 114 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L114

In general, use active voice instead of passive voice ('is injected').

Check warning on line 114 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L114

Don't put a space before or after a dash.

---

Expand All @@ -137,9 +137,9 @@

---

## Step 5: Launch distributed inference with vLLM

Check warning on line 140 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L140

'Step 5: Launch distributed inference with vLLM' should use sentence-style capitalization.

Run this on `pod-0` only. vLLM uses the Ray cluster that is already running.

Check warning on line 142 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L142

Use 'that's' instead of 'that is'.

```bash
bash ray-vllm-cluster/serve.sh
Expand Down Expand Up @@ -167,7 +167,7 @@

## Step 6: Test the endpoint

Once vLLM reports that it is ready, validate from `pod-0`:

Check warning on line 170 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L170

Use 'it's' instead of 'it is'.

```bash
# Check the server is healthy
Expand All @@ -190,9 +190,9 @@

---

## Step 7: Clean up

Check warning on line 193 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L193

'Step 7: Clean up' should use sentence-style capitalization.

When you are done, return to the [Instant Clusters page](https://console.runpod.io/instant-clusters) and delete your cluster. Leaving it running continues to incur charges.

Check warning on line 195 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L195

In general, use active voice instead of passive voice ('are done').

---

Expand All @@ -213,22 +213,23 @@
## Common issues

**Ray workers don't join**
Confirm `$MASTER_ADDR` resolves from each worker pod. Run `ping $MASTER_ADDR` in a worker terminal. If it fails, the cluster network may still be initializing — wait 30 seconds and try again.

Check warning on line 216 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L216

Don't put a space before or after a dash.

**vLLM OOM during model load**

Check warning on line 218 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L218

Spell out 'OOM', if it's unfamiliar to the audience.
Check that `/dev/shm` is large enough (at least 8 GB for 70B models). Also verify that `--tensor-parallel-size` matches the number of GPUs per node — a mismatch causes uneven shard sizes.

Check warning on line 219 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L219

Use parentheses judiciously.

Check warning on line 219 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L219

Put a nonbreaking space between the number and the unit in '70B'.

Check warning on line 219 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L219

Don't put a space before or after a dash.

**`VLLM_HOST_IP` binding error**
This error occurs when vLLM tries to bind to `0.0.0.0` on a pod with multiple network interfaces. Make sure `VLLM_HOST_IP` is set to the internal IP (`hostname -I | awk '{print $1}'`) before starting the server.

Check warning on line 222 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L222

Use parentheses judiciously.

Check warning on line 222 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L222

In general, use active voice instead of passive voice ('is set').

**Stale Ray cluster after restart**
If you restart a pod, Ray does not automatically rejoin the cluster. Rerun `head.sh` on `pod-0` first, then `worker.sh` on all other pods.

Check warning on line 225 in instant-clusters/ray-vllm.mdx

View check run for this annotation

Mintlify / Mintlify Validation (runpod-b18f5ded) - vale-spellcheck

instant-clusters/ray-vllm.mdx#L225

Use 'doesn't' instead of 'does not'.

---

## Next steps

- Adapt the serve script to load your own model from a [GlobalStore](/storage/globalstore) or [Network Volume](/storage/network-volumes) mount.
{/* TODO: The link target /storage/globalstore does not exist. Restore the link once a GlobalStore docs page is published, or replace "GlobalStore" with the correct product name. */}
- Adapt the serve script to load your own model from a GlobalStore or [Network Volume](/storage/network-volumes) mount.
- Scale up by increasing the pod count and adjusting `--pipeline-parallel-size` accordingly.
- Try [Axolotl on an Instant Cluster](/instant-clusters/axolotl) for distributed fine-tuning.
- Review the [Instant Cluster configuration reference](/instant-clusters/configuration) for full details on environment variables and networking.
Loading