fix(inference): keep single-process model loads off the training dp mesh - #211
Merged
Conversation
Lands ahead of the i4 sync in #208, which flips `ParallelDims.dp_enabled` to unconditionally True under training semantics so the FSDP2 wrap always has somewhere to install the mixed-precision policy. `build_meshes` then always builds the dp mesh, even at world_size 1, and `init_device_mesh` implicitly calls `init_process_group()`, which needs an `env://` rendezvous. `Cosmos3OmniModel` is built for inference and export only, including plain single-process runs with no torchrun rendezvous: the checkpoint conversion scripts (`convert_model_to_dcp`, `convert_model_to_vlm_safetensors`, `_convert_model_to_diffusers`) reach it through `from_pretrained_dcp`'s default `ParallelismConfig()`, and `export_model` constructs it directly from a training config. Both inherit training semantics, and on #208 both died with ValueError: Error initializing torch.distributed using env:// rendezvous: environment variable RANK expected, but not set Pin `enable_inference_mode` alongside the other training-only features the wrapper already disables. This is a no-op on main, where a single-rank `dp_replicate=1, dp_shard=1` already leaves `dp_enabled` False, and real inference paths are unaffected either way: `OmniInference._get_parallelism_config` already passes True. Also teach `omni_mot_model_test.py`'s stand-in config about `lidar_tokenizer`, which `set_up_tokenizers` reads as of #208. The extra `SimpleNamespace` attribute is inert against main's tokenizer setup. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
lfengad
force-pushed
the
fix/single-process-load-no-process-group
branch
from
August 19, 2026 15:26
58fa325 to
fc45dbe
Compare
Dinghow
approved these changes
Aug 19, 2026
Xuanmeng-Zhang
approved these changes
Aug 19, 2026
The i4 sync in #208 drops `rank0_only=False` from `IterSpeed.on_training_step_end`, so the per-iteration line is emitted by rank 0 alone and no longer carries the `[RANK n]` prefix: [RANK 0] Iteration 1: Hit counter: 1/50 | Loss: 0.2204 | Time: 75.78s (before) Iteration 1: Hit counter: 1/50 | Loss: 0.2204 | Time: 76.09s (after) Both OSS-only test files anchor their loss regex on that prefix, so they parse an empty series and fail before reaching what they actually assert: nano_training_smoke_test: expected 5 rank-0 losses, parsed [] launch_regression_test: No loss/grad-norm pairs found (losses=0, grads=10) Make the prefix an optional capture and keep a match only when the rank is absent (already rank-0-only) or zero, so one entry per step is parsed under either format rather than nothing or every rank's copy. Verified against both real CI logs: 5/5 and 10/10 rank-0 values, matching the values the green baseline parsed from the prefixed format. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Unblocks the five red checks on #208 (the 2026-08-19 i4 sync). Lands on
mainfirst; #208 then picks it up by updating/rebasing its release branch, which is also where the fix actually gets exercised — the GPU suite only triggers onpull_request: branches: [main], so on this PR it runs against code that does not yet contain the i4 change and can only show the fix is inert here.All three touched files are OSS-only (none appear in
.file_mapping.json), so the next i4 release will not clobber them and no imaginaire4 MR is needed.Root cause 1 — single-process scripts are forced to build a device mesh
#208 flips
ParallelDims.dp_enabledtoTrueunconditionally under training semantics, so the FSDP2 wrap always has somewhere to install theMixedPrecisionPolicy. As a consequencebuild_meshes()always builds the dp mesh, even atworld_size == 1, andinit_device_meshimplicitly callsinit_process_group(), which needs anenv://rendezvous. Reproduced directly againstParallelDimson #208's tree:Cosmos3OmniModelis only ever built for inference and export, including plain single-process runs with no torchrun rendezvous:convert_model_to_dcp,convert_model_to_vlm_safetensors,_convert_model_to_diffusers) reach it viafrom_pretrained_dcp's defaultParallelismConfig(), which is training semantics;export_modelconstructs it directly from a training config.convert_model_to_dcp.sheven runs withCUDA_VISIBLE_DEVICES=andCOSMOS_DEVICE=cpu, so there is no rendezvous to inherit. This one cause accounts forgenerator-training-regression,reasoner-training-regression,reasoner-inference-smoke,training-smoke, andunittest'sscripts/_test.py::test_level_0[convert_model_to_dcp]on #208 — all failed in fixture setup, not in the code under test.The fix pins
enable_inference_mode = TrueinCosmos3OmniModel.__init__, next to theema.enabled/activation_checkpointing.modeoverrides the wrapper already applies for the same reason. Doing it in__init__rather than infrom_pretrained_dcp's default also coversexport_model's direct construction. Real inference is unaffected:OmniInference._get_parallelism_configalready passesenable_inference_mode=True. Onmainit is a no-op, since a single-rankdp_replicate=1, dp_shard=1already leavesdp_enabledFalse.Root cause 2 — OSS-only test does not know about the new LiDAR config field
As of #208
set_up_tokenizersreadsconfig.lidar_tokenizer.omni_mot_model_test.pyis OSS-only and builds its config fromSimpleNamespace, so it raisedAttributeError: 'types.SimpleNamespace' object has no attribute 'lidar_tokenizer'. Addedlidar_tokenizer=Noneto both stand-in configs; the extra attribute is inert against main's tokenizer setup.Verification
pytest cosmos_framework/model/generator/omni_mot_model_test.py— 2 failed then 2 passed on Release 2026-08-19 (i4 02f21ec8) #208's tree; 2 passed on this branch.ParallelDims(output quoted above). The end-to-endconvert_model_to_dcprun is left to Release 2026-08-19 (i4 02f21ec8) #208's GPU suite once it picks this up.ruff checkandruff format --checkclean on the three files.🤖 Generated with Claude Code