Blog: Distributed Layerwise Offload — scaling toward 200B+ DiT models by david6666666 · Pull Request #302 · vllm-project/vllm-project.github.io

david6666666 · 2026-08-10T02:31:21Z

Summary

Supersedes Blog: Distributed Layerwise Offload — Running 185GB Models on 64GB NPU/GPU #295 while preserving its seven original commits, SHAs, authors, and order.
Addresses the current review feedback on the 200B+ qualification, memory-accounting scope, baseline arithmetic, HBM percentages, code attribution, throughput definitions, and mixed-concurrency comparisons.
Replaces all six ASCII technical diagrams in the post with scalable SVGs generated by fireworks-tech-graph, while retaining the existing double-buffer pipeline GIF.
Adds a separate MiniMax-H3 case study from the provided B300 DLO research note: three evaluated DPxSP routes, the topology-dependent AllGather/rank-local crossover, experiment limitations, and links to the PDF and reproducibility artifacts. Its results remain separate from the Cosmos3 tables.
Clarifies the story boundary between measured 124 GB results and the host-capacity-only 200B extrapolation, separates cgroup-visible from physical RAM, identifies MiniMax-H3 as an experimental topology extension, and closes with deployment guidance.

Review feedback addressed

Qualifies the headline as scaling toward 200B+ and distinguishes measured 124 GB results from extrapolations.
Names both solution families: offloaders and parallelism.
Scopes the host-memory formula to measured Ascend DLO+AllGather runs and separates CUDA PSS from Ascend cgroup accounting.
Corrects the 178 GB baseline breakdown and distinguishes 22% peak-HBM growth from 27% idle-HBM growth.
Attributes meta conversion and mmap loading to distributed_layerwise_backend.py.
Defines generated frames/s, labels the Ascend workload, and makes per-row B300 strategy/concurrency explicit.
Rephrases 3.3x scaling as about 83% of the ideal 4x.
States that double-buffer HBM is independent of total layer count but still scales with the largest block and workload buffers.
Separates Ascend cgroup telemetry from physical capacity planning and qualifies the 400 GB / ~200B estimate as host-memory-only.
Distinguishes the stock Cosmos3 path from the locally patched MiniMax-H3 mixed-topology study and adds a deployment-mode decision guide and conclusion.

MiniMax-H3 B300 addition

Documents the controlled 8x B300 T2VA setup and the selected-route results from 20 measured waves per route across two engine lifecycles.
Keeps exact P50/P95, throughput, measured peak/GPU, and board-energy values in a Markdown table.
Adds one Fireworks-generated topology policy SVG showing the measured latency, balanced, and throughput/energy routes.
Adds the study's original high-resolution multimodal Pareto plot after the FL2VA/Ref2VA ordering result, with n=5 per-route scope and source attribution.
Explains why AllGather helps at DP1xSP8, offers a small gain at DP4xSP2, and reverses at DP8xSP1, where rank-local DLO is preferred.
States the unmeasured DP2xSP4 route, single-node/workload scope, shape-only validation, local subgroup-broadcast fix, and version-mismatch warning.

Validation

fireworks.py check passed XML, marker, collision, geometry, and composition validation for all seven SVGs.
All seven SVG PNG readbacks and the original Pareto PNG were visually inspected for clipping, text overflow, label collisions, and readability.
The Pareto PNG Git blob SHA (0952bc08841fe5c837d02e477d47d0aa5d73cc70) matches the canonical research archive.
git diff --check passed.
All 9 Markdown image paths resolve locally; every new SVG records data-generator="fireworks-tech-graph" and the showcase quality profile.
All 13 Markdown tables have consistent column counts; figure numbering is sequential from 1 through 8.
SEO summary is present and 172 characters (240 maximum).
Jekyll build was not run locally because Ruby/Bundler is unavailable in this Windows environment; Vercel/GitHub checks provide the authoritative remote build validation.

Commit history and co-authorship

The branch starts at the exact #295 head (25ead99c2a84f10415b3d41fc7704304f3e88dc8), adds the review-fix commit, merges the current upstream main without rewriting the original seven commits, adds the MiniMax-H3 study and its original Pareto plot, and finishes with a focused narrative-clarity commit.

The review-fix and upstream-merge commits include:

Co-authored-by: evanchueng <evanchueng@outlook.com>

Both MiniMax-H3 commits include:

Co-authored-by: evanchueng <evanchueng@outlook.com>
Co-authored-by: lishunyang12 <lishunyang12@163.com>

…U/GPU Signed-off-by: evanchueng <evanchueng@outlook.com>

…eview feedback - Title: 'Serving 200B+ DiT Models Efficiently in vLLM-Omni' (per hsliuustc0106) - Add David Zhao as co-author (per hsliuustc0106) - Full rewrite with ASCII diagrams, memory model, validation results - Add Cosmos3-200B synthetic model section with RFC-3 4K results - Clarify 200B = synthetic (100 blocks, 185GB BF16) - All English (per hsliuustc0106) - DP2/DP4 full experiment matrix (per hsliuustc0106) Signed-off-by: evanchueng <evanchueng@outlook.com>

Signed-off-by: evanchueng <evanchueng@outlook.com>

- Add NVIDIA B300 SXM6 validation: T2I 1024x1024, T2V 480p/720p - Correctness verified via byte-identical SHA256 across all strategies - DLO+AG DP4: 1.39x throughput vs HSDP, 30% HBM usage - 720p 10s: within 2.13% of HSDP latency, 47% HBM - Remove synthetic 200B model section - Author: vLLM-Omni Diffusion Team Signed-off-by: evanchueng <evanchueng@outlook.com>

…llm-project#8/vllm-project#9) Replace ASCII art with clean inline SVG showing: - Three-stream timeline (Compute / H2D / AllGather) - Double-buffer slot alternation (Slot 0 / Slot 1) - Event sync arrows (compute waits for AllGather) - Color-coded legend Signed-off-by: evanchueng <evanchueng@outlook.com>

…m-project#8/vllm-project#9) - Generated 35-frame animated GIF (1500x675, 195KB) using matplotlib - Shows progressive reveal of three-stream timeline: Compute (blue) / H2D (orange) / AllGather (green) - Double-buffer slot alternation with event sync arrows - Replaces static inline SVG Signed-off-by: evanchueng <evanchueng@outlook.com>

Co-authored-by: evanchueng <evanchueng@outlook.com> Signed-off-by: David Chen <530634352@qq.com>

Co-authored-by: evanchueng <evanchueng@outlook.com> Co-authored-by: lishunyang12 <lishunyang12@163.com> Signed-off-by: David Chen <530634352@qq.com>

david6666666 · 2026-08-10T08:04:52Z

@lishunyang12 @hsliuustc0106 ptal thx

hsliuustc0106

Thanks for incorporating the earlier feedback. I found five remaining documentation-correctness issues that should be addressed before publishing. DCO and Vercel pass; local asset/link checks and SVG parsing pass, but the Jekyll build was not run locally and Blog CI is currently action-required.

hsliuustc0106 · 2026-08-11T01:07:11Z

+    --dlo-no-use-allgather
+```
+
+The `--dlo-use-allgather` / `--dlo-no-use-allgather` flag controls whether weights are sharded (default: sharded). When disabled, each rank loads full weights independently — useful when AllGather synchronization overhead outweighs the memory savings.


[P2] Narrow the no-AllGather memory claim to pure-DP configurations. The current CLI describes this mode as streaming the standard loader's rank-local tensors, including existing TP-local shards; it does not universally load a full model on every rank. Please state that a full host copy applies to pure DP, while TP/SP can already provide rank-local sharding.

hsliuustc0106 · 2026-08-11T01:07:11Z

+
+Only the primary rank within each DP replica (SP=0, TP=0, CFG=0, PP=0) replies, tagged with `dp_rank` for result matching. The executor collects responses via round-robin polling and sorts by `dp_rank` to match results to requests.
+
+A validation step rejects concurrent requests with different `num_inference_steps` — since AllGather is a collective, mismatched step counts would cause one rank to exit early while others hang.


[P2] Document all DP-wave compatibility constraints here. Matching num_inference_steps is only one requirement: the DLO DP checks also cover shape, CFG, LoRA/extra_args, prompt validity, and pipeline request-batch support/compatibility. Otherwise “each DP rank processes a different request” can be read as allowing arbitrary heterogeneous requests, which can reject the wave or deadlock collectives.

hsliuustc0106 · 2026-08-11T01:07:12Z

+
+### NVIDIA B300 GPU Results
+
+To validate platform-agnosticism, we ran the same DLO stack on NVIDIA B300 SXM6 GPUs. All tests use Cosmos3-Super BF16 (124 GB), 4× NVIDIA B300 (physical GPUs 1,5,6,7), Python 3.12.3, PyTorch 2.11.0+cu130, CUDA 13.0, vLLM 0.25.0.


[P1] Correct and complete the benchmark environment. The linked study's environment.json.txt reports vllm==0.24.0 and vllm_omni==0.26.0rc2.dev11+g6607f4a7f, not vLLM 0.25.0; its runner also sets enforce_eager=True and applies a local subgroup-broadcast patch. Please report those exact versions/flags/patch so readers do not assume these results represent an unmodified release or the default compile path.

hsliuustc0106 · 2026-08-11T01:07:12Z

+| Model | dp_size | cgroup Peak (est.) | Total RAM (est.) | Fits 2 TB? |
+|-------|:-------:|:------------------:|:----------------:|:----------:|
+| 33 GB | 4 | 47 GB | ~80 GB | ✓ |
+| 124 GB | 4 | 172 GB | ~280 GB | ✓ |


[P2] Reconcile this total-RAM estimate. The stated memory model adds cgroup-visible memory to the pinned shard memory. For the 124 GB / DP4 row, that is approximately 172 + 124 = 296 GB, not ~280 GB. Please correct the value or explain the different estimation formula used for this row.

…ment and DP-wave compatibility for #5864 - Quickstart: replace stale commit pin with v0.27.0rc1 requirement; explain that #5864 bypasses supports_request_batch for DLO+DP by running each rank's request through the single-request forward path. - DP multi-concurrency: remove the claim that the pipeline must declare supports_request_batch=True; describe per-rank result queues, pre-dispatch wave rejection, and fail-closed partial-wave timeout from #5864. Signed-off-by: MuweiZ <zhengmuweixxn@gmail.com>

- Quickstart: add v0.27.0rc1 version requirement for AllGather commands; explain that #5864 bypasses supports_request_batch for DLO+AllGather+DP by running each rank's request through the single-request forward path. Note that no-AllGather DP is tracked in #5911 (still open). - no-AllGather memory scope: clarify that full host copy applies to pure DP, while TP/SP shards are already rank-local — in Quickstart, TL;DR, and B300 environment section. - DP-wave constraints: document the full batch-compatibility key (shape, CFG, steps, LoRA, extra_args, output count, quality mode); correct extra_args to 'must be JSON-identical' per source code (Gate 2). Describe per-rank result queues, pre-dispatch wave rejection, and fail-closed partial-wave timeout from #5864. - B300 environment: split Cosmos3 and MiniMax-H3 descriptions; move enforce_eager=True and pipeline_minimax_h3.py patch to MiniMax-H3 subsection. - Correct B300 versions: vLLM 0.24.0 (not 0.25.0), add vLLM-Omni build. - Fix total-RAM estimate: 124 GB / DP4 = ~296 GB (was ~280 GB). Co-authored-by: evanchueng <evanchueng@outlook.com> Signed-off-by: MuweiZ <zhengmuweixxn@gmail.com>

- Quickstart: add v0.27.0rc1 version requirement for AllGather commands; explain that #5864 bypasses supports_request_batch for DLO+AllGather+DP by running each rank's request through the single-request forward path. Note that no-AllGather DP is tracked in #5911 (still open). - no-AllGather memory scope: clarify that full host copy applies to pure DP, while TP/SP shards are already rank-local — in TL;DR, Quickstart, solution table, problem paragraph, and weight-sharding section. - DP-wave constraints: document the full batch-compatibility key (shape, CFG, steps, LoRA, extra_args, output count); correct extra_args to 'must be JSON-identical' per source code (Gate 2). Describe per-rank result queues, pre-dispatch wave rejection, and fail-closed partial-wave timeout from #5864. - B300 environment: split Cosmos3 and MiniMax-H3 descriptions; move vLLM/Omni versions, enforce_eager=True, and pipeline_minimax_h3.py patch to MiniMax-H3 subsection with environment.json.txt attribution. Cosmos3 paragraph retains only self-proven hardware/framework info. - Memory section: rename to 'cgroup-visible vs. Physical RAM'; clarify that physical RAM includes pinned DMA shards invisible to cgroup. - 400 GB extrapolation: add caveat that no 200B model was run. - Narrative: 'four cooperating techniques'; clarify DP multi-concurrency is throughput optimization, not memory feasibility. - Correct B300 versions: vLLM 0.24.0 (not 0.25.0), add vLLM-Omni build. - Fix total-RAM estimate: 124 GB / DP4 = ~296 GB (was ~280 GB). Co-authored-by: evanchueng <evanchueng@outlook.com> Signed-off-by: MuweiZ <zhengmuweixxn@gmail.com>

hsliuustc0106

I found seven remaining documentation, provenance, SEO, and accessibility issues on the current head. Details are inline.

hsliuustc0106 · 2026-08-16T00:45:18Z

+layout: post
+title: "Distributed Layerwise Offload: Scaling Toward 200B+ DiT Models Efficiently in vLLM-Omni"
+author: "vLLM-Omni Diffusion Team"
+summary: "Distributed Layerwise Offload shards and streams DiT weights across devices, serving a measured 124 GB Cosmos3 model on 64 GB HBM and estimating a path toward 200B+ models."


[P2] Add a description field that Jekyll SEO actually emits. The exact-head preview renders TL;DR as the meta description, Open Graph description, and JSON-LD description because jekyll-seo-tag does not consume this summary field. Please keep summary for the repository check, but also add the same text as description, or wire page.summary into the shared SEO template.

hsliuustc0106 · 2026-08-16T00:45:19Z

+    --dlo-no-use-allgather
+```
+
+The `--dlo-use-allgather` / `--dlo-no-use-allgather` flag controls whether weights are sharded (default: sharded). When disabled, each rank loads the standard loader's rank-local tensors — in pure-DP configurations this is a full model copy, while existing TP/SP shards are already rank-local and are reused as-is. This mode is useful when AllGather synchronization overhead outweighs the memory savings.


[P2] Do not claim that SP provides rank-local weight shards in no-AllGather mode. The standard loader may already produce TP-local tensors, but SP alone partitions sequences/activations rather than checkpoint weights. In v0.27.0rc1 the no-AllGather offloader forces its DLO shard size to one; only the AllGather path can select the SP group (source). Please change TP/SP to TP here and in the matching claims at lines 25 and 66, otherwise the post understates host RAM for SP-only deployments.

hsliuustc0106 · 2026-08-16T00:45:19Z

+
+For Hugging Face repo IDs (not local paths), we resolve the snapshot path first via `download_weights_from_hf()`, matching the pattern used by vLLM's existing DiffusersPipelineLoader.
+
+**What you gain.** Cold-start peak RSS drops from 178 GB to 47 GB for Cosmos3-Nano DP4 — a 73% reduction. The 178 GB baseline consists of 132 GB of private model copies, 33 GB of shared page cache, and about 13 GB of framework/transient overhead. The mmap page cache (1× model_size) is shared and read-only, and can be partially reclaimed by the OS under memory pressure.


[P2] Label the 178 GB → 47 GB result as a cgroup-visible host-memory peak, not peak RSS. The validation table uses these exact values for cgroup Peak and separately reports page cache and RSS. Please rename the repeated peak RSS claims (including the TL;DR and solution table), or provide distinct RSS measurements.

hsliuustc0106 · 2026-08-16T00:45:19Z

+
+**Why it works.** We maintain exactly two device buffers (slots), each sized to the largest block in the model. While the compute stream executes layer N (using slot 0), background streams prepare layer N+1 into slot 1:
+
+![DLO Double-Buffer Prefetch Pipeline](/assets/figures/2026-07-30-distributed-layerwise-offload/dlo_pipeline.gif)


[P2] Provide a pause or reduced-motion alternative for this animation. The GIF contains 35 frames over roughly 21 seconds and loops indefinitely, while the generated page embeds it as a plain <img> with no user control. Please use controlled media with pause/reduced-motion behavior, or provide a static SVG/PNG fallback.

hsliuustc0106 · 2026-08-16T00:45:19Z

+| dist_offload+AG (DP4, 4 req) | 1,020 | 3.22 | 3.5 GB | 12.4 GB | 3.3× |
+| dist_offload no-AG | 1,877 | 0.439 | 28.3 GB | 14.1 GB | -55% |
+
+AllGather overhead = 150 ms/step (72 ms stream switch + 10 ms HCCL + 68 ms Python dispatch), model-size independent. With 4 concurrent requests, this fixed cost is amortized 4×.


[P2] Scope the 150 ms/step overhead to the measured Nano DP4 run. The displayed Ascend performance data is Nano-only, while the described algorithm AllGathers every full layer on each denoising step; communication volume therefore varies with layer dimensions/count, participant count, and topology. The evidence does not support the general model-size independent claim.

hsliuustc0106 · 2026-08-16T00:45:19Z

+
+### NVIDIA B300 GPU Results
+
+To validate platform-agnosticism, we ran the same DLO stack on NVIDIA B300 SXM6 GPUs. The Cosmos3 tests below use Cosmos3-Super BF16 (124 GB), 4× NVIDIA B300 (physical GPUs 1,5,6,7), Python 3.12.3, PyTorch 2.11.0+cu130, CUDA 13.0, vLLM `0.23.0`, and vLLM-Omni main-branch commit [`9772bb32`](https://github.com/vllm-project/vllm-omni/commit/9772bb321f558a28c0dca1cb53b44aaf10e4ab69) (PR [#5397](https://github.com/vllm-project/vllm-omni/pull/5397)). The MiniMax-H3 subsection that follows documents its own vLLM/vLLM-Omni versions, `enforce_eager=True` flag, and a local pipeline patch; those details apply to the MiniMax-H3 study and are not assumed for the Cosmos3 runs.


[P2] Do not describe 9772bb32 as a main-branch commit. This SHA is a diverged PR snapshot, and #5397's final head contains later loader-gating and TP/mmap validation changes. Please label the benchmark revision as an unmerged PR snapshot and disclose that divergence, or rerun/cite a merged main/tag revision so readers do not interpret these results as validation of released code.

hsliuustc0106 · 2026-08-16T00:45:19Z

+|-------|:-------:|:------------------:|:----------------:|:----------:|
+| 33 GB | 4 | 47 GB | ~80 GB | ✓ |
+| 124 GB | 4 | 172 GB | ~296 GB | ✓ |
+| 185 GB | 4 | ~220 GB | ~420 GB | ✓ |


[P2] Reconcile this total-RAM estimate with the stated formula. Total physical RAM is defined above as cgroup-visible memory plus pinned shards totaling one model size, so ~220 GB + 185 GB = ~405 GB, not ~420 GB. Please correct the value or document the additional ~15 GB component.

- Quickstart: add v0.27.0rc1 version requirement for AllGather commands; explain that #5864 bypasses supports_request_batch for DLO+AllGather+DP by running each rank's request through the single-request forward path. Note that no-AllGather DP is tracked in #5911 (still open). - no-AllGather memory scope: clarify that full host copy applies to pure DP, while existing TP shards are already rank-local (SP partitions sequences, not weights) — in TL;DR, Quickstart, solution table, problem paragraph, and weight-sharding section. - DP-wave constraints: document the full batch-compatibility key (shape, CFG, steps, LoRA, extra_args, output count); correct extra_args to 'must be JSON-identical' per source code (Gate 2). Describe per-rank result queues, pre-dispatch wave rejection, and fail-closed partial-wave timeout from #5864. - B300 environment: split Cosmos3 and MiniMax-H3 descriptions; Cosmos3 uses vLLM 0.23.0 + unmerged PR-snapshot commit 9772bb32 (#5397); MiniMax-H3 documents its own versions, enforce_eager=True, and local pipeline patch from environment.json.txt. - Memory section: rename to 'cgroup-visible vs. Physical RAM'; clarify that physical RAM includes pinned DMA shards invisible to cgroup. - Label 178→47 GB as cgroup-visible peak, not peak RSS (4 locations). - 400 GB extrapolation: add caveat that no 200B model was run; fix 185 GB row total-RAM math (~405 GB, was ~420 GB). - Narrative: 'four cooperating techniques'; clarify DP multi-concurrency is throughput optimization, not memory feasibility. - AllGather overhead: scope to Cosmos3-Nano DP4; remove 'model-size independent' claim. - Add 'description' front matter field for jekyll-seo-tag. - Wrap pipeline GIF in <video controls> for pause/reduced-motion access. - Correct B300 versions: vLLM 0.24.0 (not 0.25.0), add vLLM-Omni build. - Fix total-RAM estimate: 124 GB / DP4 = ~296 GB (was ~280 GB). Co-authored-by: evanchueng <evanchueng@outlook.com> Signed-off-by: MuweiZ <zhengmuweixxn@gmail.com>

Signed-off-by: David Chen <530634352@qq.com>

Isotr0py · 2026-08-17T03:54:51Z

Maybe update the date? The blog is not at the top in preview:

Isotr0py · 2026-08-17T03:57:36Z

+
+## Quickstart
+
+> **Version requirement.** The two AllGather commands below require vLLM-Omni `v0.27.0rc1` or later with vLLM `0.27.0`. On the `v0.26.0` release, the Cosmos3 DLO+DP path rejects every request because the engine requires `supports_request_batch=True` for multi-request admission, which `Cosmos3OmniDiffusersPipeline` does not declare ([#5953](https://github.com/vllm-project/vllm-omni/issues/5953)). [#5864](https://github.com/vllm-project/vllm-omni/pull/5864) fixes this by bypassing the `supports_request_batch` requirement for DLO+AllGather+DP configurations: each DP rank runs its own request independently through the pipeline's single-request forward path, and the engine collects results from per-rank queues. The no-AllGather DP command is not covered by #5864; independent request dispatch for `--dlo-no-use-allgather` is tracked in [#5911](https://github.com/vllm-project/vllm-omni/pull/5911) (still open). The correctness fix in #5864 does not change the DLO weight-sharding or offload memory mechanism; each measurement section below reports its own environment.


I think it worths a TL;DR for version requirement as well? I think reader may just want to know which version to select for trying out of box.

Signed-off-by: David Chen <530634352@qq.com>

Isotr0py · 2026-08-17T07:22:43Z

This image's "safetensors page cache" block looks a bit weird. Seems some attatched ellipses are exported as well by mistake.

Signed-off-by: David Chen <530634352@qq.com>

evanchueng and others added 8 commits July 31, 2026 10:47

Blog: Distributed Layerwise Offload — Running 185GB Models on 64GB NP…

9de883f

…U/GPU Signed-off-by: evanchueng <evanchueng@outlook.com>

Fix author name: david6666666

e20a252

Signed-off-by: evanchueng <evanchueng@outlook.com>

Update authors: Jinyi Zhang, Weiqing Chen

c62f7c4

Signed-off-by: evanchueng <evanchueng@outlook.com>

Address DLO blog review feedback

ff1165d

Co-authored-by: evanchueng <evanchueng@outlook.com> Signed-off-by: David Chen <530634352@qq.com>

vercel Bot deployed to Preview August 10, 2026 02:31 View deployment

Merge upstream main into DLO blog update

1940f4a

Co-authored-by: evanchueng <evanchueng@outlook.com> Signed-off-by: David Chen <530634352@qq.com>

vercel Bot deployed to Preview August 10, 2026 02:35 View deployment

Add MiniMax-H3 B300 topology study

6c6d7c0

Co-authored-by: evanchueng <evanchueng@outlook.com> Co-authored-by: lishunyang12 <lishunyang12@163.com> Signed-off-by: David Chen <530634352@qq.com>

vercel Bot deployed to Preview August 10, 2026 07:50 View deployment

Add MiniMax-H3 multimodal Pareto frontier

ad3c579

Co-authored-by: evanchueng <evanchueng@outlook.com> Co-authored-by: lishunyang12 <lishunyang12@163.com> Signed-off-by: David Chen <530634352@qq.com>

vercel Bot deployed to Preview August 10, 2026 08:00 View deployment

hsliuustc0106 reviewed Aug 11, 2026

View reviewed changes

vercel Bot deployed to Preview August 14, 2026 09:11 View deployment

vercel Bot deployed to Preview August 14, 2026 09:32 View deployment

MuweiZ force-pushed the agent/dlo-blog-review-fixes branch from 908d214 to 2a73542 Compare August 14, 2026 09:41

vercel Bot deployed to Preview August 14, 2026 09:42 View deployment

MuweiZ force-pushed the agent/dlo-blog-review-fixes branch from 2a73542 to 03173ac Compare August 14, 2026 10:21

vercel Bot deployed to Preview August 14, 2026 10:22 View deployment

MuweiZ force-pushed the agent/dlo-blog-review-fixes branch from 03173ac to cd949f5 Compare August 14, 2026 10:56

vercel Bot deployed to Preview August 14, 2026 10:57 View deployment

hsliuustc0106 reviewed Aug 16, 2026

View reviewed changes

MuweiZ force-pushed the agent/dlo-blog-review-fixes branch from cd949f5 to 96f47fc Compare August 16, 2026 06:18

vercel Bot deployed to Preview August 16, 2026 06:18 View deployment

MuweiZ force-pushed the agent/dlo-blog-review-fixes branch from 96f47fc to 875e1b3 Compare August 16, 2026 06:26

vercel Bot deployed to Preview August 16, 2026 06:27 View deployment

MuweiZ force-pushed the agent/dlo-blog-review-fixes branch from 875e1b3 to a403d38 Compare August 16, 2026 06:40

vercel Bot deployed to Preview August 16, 2026 06:40 View deployment

MuweiZ force-pushed the agent/dlo-blog-review-fixes branch from a403d38 to ae3d50f Compare August 16, 2026 06:43

vercel Bot deployed to Preview August 16, 2026 06:44 View deployment

MuweiZ force-pushed the agent/dlo-blog-review-fixes branch from ae3d50f to 13d0802 Compare August 16, 2026 07:00

vercel Bot deployed to Preview August 16, 2026 07:01 View deployment

david6666666 marked this pull request as ready for review August 17, 2026 02:51

Clarify DLO double-buffer HBM scaling

025fe04

Signed-off-by: David Chen <530634352@qq.com>

vercel Bot deployed to Preview August 17, 2026 03:19 View deployment

Isotr0py reviewed Aug 17, 2026

View reviewed changes

Update DLO blog date and version guidance

a91548c

Signed-off-by: David Chen <530634352@qq.com>

vercel Bot deployed to Preview August 17, 2026 06:28 View deployment

Isotr0py approved these changes Aug 17, 2026

View reviewed changes

Improve DLO diagram label clarity

3c0895d

Signed-off-by: David Chen <530634352@qq.com>

vercel Bot deployed to Preview August 17, 2026 07:46 View deployment

Render DLO cache nodes as cylinders

f3c6763

Signed-off-by: David Chen <530634352@qq.com>

vercel Bot deployed to Preview August 17, 2026 08:41 View deployment

Merge branch 'main' into agent/dlo-blog-review-fixes

79fdb74

vercel Bot deployed to Preview August 17, 2026 08:43 View deployment

Isotr0py merged commit 258260b into vllm-project:main Aug 17, 2026
4 checks passed


		Only the primary rank within each DP replica (SP=0, TP=0, CFG=0, PP=0) replies, tagged with `dp_rank` for result matching. The executor collects responses via round-robin polling and sorts by `dp_rank` to match results to requests.

		A validation step rejects concurrent requests with different `num_inference_steps` — since AllGather is a collective, mismatched step counts would cause one rank to exit early while others hang.


		### NVIDIA B300 GPU Results

		To validate platform-agnosticism, we ran the same DLO stack on NVIDIA B300 SXM6 GPUs. All tests use Cosmos3-Super BF16 (124 GB), 4× NVIDIA B300 (physical GPUs 1,5,6,7), Python 3.12.3, PyTorch 2.11.0+cu130, CUDA 13.0, vLLM 0.25.0.


		For Hugging Face repo IDs (not local paths), we resolve the snapshot path first via `download_weights_from_hf()`, matching the pattern used by vLLM's existing DiffusersPipelineLoader.

		What you gain. Cold-start peak RSS drops from 178 GB to 47 GB for Cosmos3-Nano DP4 — a 73% reduction. The 178 GB baseline consists of 132 GB of private model copies, 33 GB of shared page cache, and about 13 GB of framework/transient overhead. The mmap page cache (1× model_size) is shared and read-only, and can be partially reclaimed by the OS under memory pressure.


		Why it works. We maintain exactly two device buffers (slots), each sized to the largest block in the model. While the compute stream executes layer N (using slot 0), background streams prepare layer N+1 into slot 1:

		![DLO Double-Buffer Prefetch Pipeline](/assets/figures/2026-07-30-distributed-layerwise-offload/dlo_pipeline.gif)


		## Quickstart

		> Version requirement. The two AllGather commands below require vLLM-Omni `v0.27.0rc1` or later with vLLM `0.27.0`. On the `v0.26.0` release, the Cosmos3 DLO+DP path rejects every request because the engine requires `supports_request_batch=True` for multi-request admission, which `Cosmos3OmniDiffusersPipeline` does not declare ([#5953](https://github.com/vllm-project/vllm-omni/issues/5953)). [#5864](https://github.com/vllm-project/vllm-omni/pull/5864) fixes this by bypassing the `supports_request_batch` requirement for DLO+AllGather+DP configurations: each DP rank runs its own request independently through the pipeline's single-request forward path, and the engine collects results from per-rank queues. The no-AllGather DP command is not covered by #5864; independent request dispatch for `--dlo-no-use-allgather` is tracked in [#5911](https://github.com/vllm-project/vllm-omni/pull/5911) (still open). The correctness fix in #5864 does not change the DLO weight-sharding or offload memory mechanism; each measurement section below reports its own environment.

Uh oh!

Conversation

david6666666 commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Review feedback addressed

MiniMax-H3 B300 addition

Validation

Commit history and co-authorship

Uh oh!

david6666666 commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hsliuustc0106 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hsliuustc0106 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

david6666666 commented Aug 10, 2026 •

edited

Loading

david6666666 commented Aug 10, 2026 •

edited

Loading