Skip to content

fix(inference): keep single-process model loads off the training dp mesh - #211

Merged
lfengad merged 2 commits into
mainfrom
fix/single-process-load-no-process-group
Aug 19, 2026
Merged

fix(inference): keep single-process model loads off the training dp mesh#211
lfengad merged 2 commits into
mainfrom
fix/single-process-load-no-process-group

Conversation

@lfengad

@lfengad lfengad commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Unblocks the five red checks on #208 (the 2026-08-19 i4 sync). Lands on main first; #208 then picks it up by updating/rebasing its release branch, which is also where the fix actually gets exercised — the GPU suite only triggers on pull_request: branches: [main], so on this PR it runs against code that does not yet contain the i4 change and can only show the fix is inert here.

All three touched files are OSS-only (none appear in .file_mapping.json), so the next i4 release will not clobber them and no imaginaire4 MR is needed.

Root cause 1 — single-process scripts are forced to build a device mesh

#208 flips ParallelDims.dp_enabled to True unconditionally under training semantics, so the FSDP2 wrap always has somewhere to install the MixedPrecisionPolicy. As a consequence build_meshes() always builds the dp mesh, even at world_size == 1, and init_device_mesh implicitly calls init_process_group(), which needs an env:// rendezvous. Reproduced directly against ParallelDims on #208's tree:

enable_inference_mode=False -> dp_enabled=True  -> ValueError: environment variable RANK expected, but not set
enable_inference_mode=True  -> dp_enabled=False -> no mesh built, no process group needed

Cosmos3OmniModel is only ever built for inference and export, including plain single-process runs with no torchrun rendezvous:

  • the conversion scripts (convert_model_to_dcp, convert_model_to_vlm_safetensors, _convert_model_to_diffusers) reach it via from_pretrained_dcp's default ParallelismConfig(), which is training semantics;
  • export_model constructs it directly from a training config.

convert_model_to_dcp.sh even runs with CUDA_VISIBLE_DEVICES= and COSMOS_DEVICE=cpu, so there is no rendezvous to inherit. This one cause accounts for generator-training-regression, reasoner-training-regression, reasoner-inference-smoke, training-smoke, and unittest's scripts/_test.py::test_level_0[convert_model_to_dcp] on #208 — all failed in fixture setup, not in the code under test.

The fix pins enable_inference_mode = True in Cosmos3OmniModel.__init__, next to the ema.enabled / activation_checkpointing.mode overrides the wrapper already applies for the same reason. Doing it in __init__ rather than in from_pretrained_dcp's default also covers export_model's direct construction. Real inference is unaffected: OmniInference._get_parallelism_config already passes enable_inference_mode=True. On main it is a no-op, since a single-rank dp_replicate=1, dp_shard=1 already leaves dp_enabled False.

Root cause 2 — OSS-only test does not know about the new LiDAR config field

As of #208 set_up_tokenizers reads config.lidar_tokenizer. omni_mot_model_test.py is OSS-only and builds its config from SimpleNamespace, so it raised AttributeError: 'types.SimpleNamespace' object has no attribute 'lidar_tokenizer'. Added lidar_tokenizer=None to both stand-in configs; the extra attribute is inert against main's tokenizer setup.

Verification

  • pytest cosmos_framework/model/generator/omni_mot_model_test.py — 2 failed then 2 passed on Release 2026-08-19 (i4 02f21ec8) #208's tree; 2 passed on this branch.
  • Root cause 1 reproduced directly against ParallelDims (output quoted above). The end-to-end convert_model_to_dcp run is left to Release 2026-08-19 (i4 02f21ec8) #208's GPU suite once it picks this up.
  • ruff check and ruff format --check clean on the three files.

🤖 Generated with Claude Code

Lands ahead of the i4 sync in #208, which flips `ParallelDims.dp_enabled`
to unconditionally True under training semantics so the FSDP2 wrap always
has somewhere to install the mixed-precision policy. `build_meshes` then
always builds the dp mesh, even at world_size 1, and `init_device_mesh`
implicitly calls `init_process_group()`, which needs an `env://`
rendezvous.

`Cosmos3OmniModel` is built for inference and export only, including
plain single-process runs with no torchrun rendezvous: the checkpoint
conversion scripts (`convert_model_to_dcp`,
`convert_model_to_vlm_safetensors`, `_convert_model_to_diffusers`) reach
it through `from_pretrained_dcp`'s default `ParallelismConfig()`, and
`export_model` constructs it directly from a training config. Both
inherit training semantics, and on #208 both died with

    ValueError: Error initializing torch.distributed using env://
    rendezvous: environment variable RANK expected, but not set

Pin `enable_inference_mode` alongside the other training-only features
the wrapper already disables. This is a no-op on main, where a
single-rank `dp_replicate=1, dp_shard=1` already leaves `dp_enabled`
False, and real inference paths are unaffected either way:
`OmniInference._get_parallelism_config` already passes True.

Also teach `omni_mot_model_test.py`'s stand-in config about
`lidar_tokenizer`, which `set_up_tokenizers` reads as of #208. The extra
`SimpleNamespace` attribute is inert against main's tokenizer setup.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@lfengad
lfengad force-pushed the fix/single-process-load-no-process-group branch from 58fa325 to fc45dbe Compare August 19, 2026 15:26
@lfengad
lfengad changed the base branch from release/2026-08-19-02f21ec8 to main August 19, 2026 15:27
@lfengad lfengad closed this Aug 19, 2026
@lfengad lfengad reopened this Aug 19, 2026
The i4 sync in #208 drops `rank0_only=False` from `IterSpeed.on_training_step_end`,
so the per-iteration line is emitted by rank 0 alone and no longer carries the
`[RANK n]` prefix:

    [RANK 0] Iteration 1: Hit counter: 1/50 | Loss: 0.2204 | Time: 75.78s   (before)
             Iteration 1: Hit counter: 1/50 | Loss: 0.2204 | Time: 76.09s   (after)

Both OSS-only test files anchor their loss regex on that prefix, so they parse
an empty series and fail before reaching what they actually assert:

    nano_training_smoke_test: expected 5 rank-0 losses, parsed []
    launch_regression_test:   No loss/grad-norm pairs found (losses=0, grads=10)

Make the prefix an optional capture and keep a match only when the rank is
absent (already rank-0-only) or zero, so one entry per step is parsed under
either format rather than nothing or every rank's copy. Verified against both
real CI logs: 5/5 and 10/10 rank-0 values, matching the values the green
baseline parsed from the prefixed format.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@lfengad
lfengad enabled auto-merge (squash) August 19, 2026 16:24
@lfengad
lfengad merged commit 84bd882 into main Aug 19, 2026
9 checks passed
@lfengad
lfengad deleted the fix/single-process-load-no-process-group branch August 19, 2026 16:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants