Release 2026-08-19 (i4 02f21ec8) - #208
Merged
Merged
Conversation
_source_commit: 02f21ec8097056f5b94570463c41fc7dcf958934-dirty _dest_commit: c14617c
lfengad
added a commit
that referenced
this pull request
Aug 19, 2026
Lands ahead of the i4 sync in #208, which flips `ParallelDims.dp_enabled` to unconditionally True under training semantics so the FSDP2 wrap always has somewhere to install the mixed-precision policy. `build_meshes` then always builds the dp mesh, even at world_size 1, and `init_device_mesh` implicitly calls `init_process_group()`, which needs an `env://` rendezvous. `Cosmos3OmniModel` is built for inference and export only, including plain single-process runs with no torchrun rendezvous: the checkpoint conversion scripts (`convert_model_to_dcp`, `convert_model_to_vlm_safetensors`, `_convert_model_to_diffusers`) reach it through `from_pretrained_dcp`'s default `ParallelismConfig()`, and `export_model` constructs it directly from a training config. Both inherit training semantics, and on #208 both died with ValueError: Error initializing torch.distributed using env:// rendezvous: environment variable RANK expected, but not set Pin `enable_inference_mode` alongside the other training-only features the wrapper already disables. This is a no-op on main, where a single-rank `dp_replicate=1, dp_shard=1` already leaves `dp_enabled` False, and real inference paths are unaffected either way: `OmniInference._get_parallelism_config` already passes True. Also teach `omni_mot_model_test.py`'s stand-in config about `lidar_tokenizer`, which `set_up_tokenizers` reads as of #208. The extra `SimpleNamespace` attribute is inert against main's tokenizer setup. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
lfengad
added a commit
that referenced
this pull request
Aug 19, 2026
The i4 sync in #208 drops `rank0_only=False` from `IterSpeed.on_training_step_end`, so the per-iteration line is emitted by rank 0 alone and no longer carries the `[RANK n]` prefix: [RANK 0] Iteration 1: Hit counter: 1/50 | Loss: 0.2204 | Time: 75.78s (before) Iteration 1: Hit counter: 1/50 | Loss: 0.2204 | Time: 76.09s (after) Both OSS-only test files anchor their loss regex on that prefix, so they parse an empty series and fail before reaching what they actually assert: nano_training_smoke_test: expected 5 rank-0 losses, parsed [] launch_regression_test: No loss/grad-norm pairs found (losses=0, grads=10) Make the prefix an optional capture and keep a match only when the rank is absent (already rank-0-only) or zero, so one entry per step is parsed under either format rather than nothing or every rank's copy. Verified against both real CI logs: 5/5 and 10/10 rank-0 values, matching the values the green baseline parsed from the prefixed format. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
lfengad
added a commit
that referenced
this pull request
Aug 19, 2026
…esh (#211) Unblocks the five red checks on #208 (the 2026-08-19 i4 sync). Lands on `main` first; #208 then picks it up by updating/rebasing its release branch, which is also where the fix actually gets exercised — the GPU suite only triggers on `pull_request: branches: [main]`, so on this PR it runs against code that does not yet contain the i4 change and can only show the fix is inert here. All three touched files are OSS-only (none appear in `.file_mapping.json`), so the next i4 release will not clobber them and no imaginaire4 MR is needed. ## Root cause 1 — single-process scripts are forced to build a device mesh #208 flips `ParallelDims.dp_enabled` to `True` unconditionally under training semantics, so the FSDP2 wrap always has somewhere to install the `MixedPrecisionPolicy`. As a consequence `build_meshes()` always builds the dp mesh, even at `world_size == 1`, and `init_device_mesh` implicitly calls `init_process_group()`, which needs an `env://` rendezvous. Reproduced directly against `ParallelDims` on #208's tree: ``` enable_inference_mode=False -> dp_enabled=True -> ValueError: environment variable RANK expected, but not set enable_inference_mode=True -> dp_enabled=False -> no mesh built, no process group needed ``` `Cosmos3OmniModel` is only ever built for inference and export, including plain single-process runs with no torchrun rendezvous: - the conversion scripts (`convert_model_to_dcp`, `convert_model_to_vlm_safetensors`, `_convert_model_to_diffusers`) reach it via `from_pretrained_dcp`'s default `ParallelismConfig()`, which is training semantics; - `export_model` constructs it directly from a training config. `convert_model_to_dcp.sh` even runs with `CUDA_VISIBLE_DEVICES=` and `COSMOS_DEVICE=cpu`, so there is no rendezvous to inherit. This one cause accounts for `generator-training-regression`, `reasoner-training-regression`, `reasoner-inference-smoke`, `training-smoke`, and `unittest`'s `scripts/_test.py::test_level_0[convert_model_to_dcp]` on #208 — all failed in fixture setup, not in the code under test. The fix pins `enable_inference_mode = True` in `Cosmos3OmniModel.__init__`, next to the `ema.enabled` / `activation_checkpointing.mode` overrides the wrapper already applies for the same reason. Doing it in `__init__` rather than in `from_pretrained_dcp`'s default also covers `export_model`'s direct construction. Real inference is unaffected: `OmniInference._get_parallelism_config` already passes `enable_inference_mode=True`. On `main` it is a no-op, since a single-rank `dp_replicate=1, dp_shard=1` already leaves `dp_enabled` False. ## Root cause 2 — OSS-only test does not know about the new LiDAR config field As of #208 `set_up_tokenizers` reads `config.lidar_tokenizer`. `omni_mot_model_test.py` is OSS-only and builds its config from `SimpleNamespace`, so it raised `AttributeError: 'types.SimpleNamespace' object has no attribute 'lidar_tokenizer'`. Added `lidar_tokenizer=None` to both stand-in configs; the extra attribute is inert against main's tokenizer setup. ## Verification - `pytest cosmos_framework/model/generator/omni_mot_model_test.py` — 2 failed then 2 passed on #208's tree; 2 passed on this branch. - Root cause 1 reproduced directly against `ParallelDims` (output quoted above). The end-to-end `convert_model_to_dcp` run is left to #208's GPU suite once it picks this up. - `ruff check` and `ruff format --check` clean on the three files. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
lfengad
enabled auto-merge (squash)
August 19, 2026 16:42
The VFM moved onto FSDP2 mixed precision. What changed is which layer owns the
FP32 master, not whether one exists:
before: sharded parameter BF16; FusedAdam holds the master itself, because
``_needs_master_weights`` is True for a BF16 parameter, and dispatches
to TE's ``multi_tensor_adam_capturable_master``.
after: ``fsdp_master_dtype`` (until now VLM-only -- ``set_up_model`` built the
net at ``self.precision``) makes the sharded parameter FP32 and installs
a MixedPrecisionPolicy that down-casts to BF16 for compute, so FusedAdam
needs no master and takes the plain capturable path.
Both arrangements accumulate in FP32 and run forward/backward in BF16, so they
are equivalent on paper. The series shifts anyway because the implementations
differ: a different TE kernel, and the BF16 down-cast moving from the optimizer
to the FSDP all-gather. This is also why ``dp_enabled`` became unconditional
under training semantics -- a degree-1 run still needs the wrap that carries the
policy.
got : [0.2242, 0.2141, 0.2428, 0.2259, 0.2607, 0.2553, 0.3318, 0.2252, 0.2035, 0.2609]
expected: [0.2242, 0.2141, 0.2429, 0.2259, 0.2608, 0.2555, 0.332, 0.2256, 0.2041, 0.2621]
iter-0 is unchanged (forward only, and an FP32 master loaded from BF16 weights
down-casts back losslessly); the gap grows monotonically to 1.2e-3 by iter 9,
past the 1e-3 tolerance. Two independent runs of this build agree to within
2e-4, so the shift is systematic rather than run-to-run noise, and the refreshed
series accepts both while still rejecting the pre-mixed-precision one. The
tolerance is left at 1e-3 rather than widened, so the guard keeps its
sensitivity to a real regression.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Dinghow
approved these changes
Aug 20, 2026
lfengad
approved these changes
Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated release from i4.
_source_commit:
02f21ec8097056f5b94570463c41fc7dcf958934-dirty_dest_commit (base):
c14617c2bc93dacbf69674fb964eec93182933d9