[Task Clean-up][Newton] Dexterous Part 1/8: Fix cubric fallback and visualizer teardown - #6411
Conversation
Cloner imports no longer create empty MuJoCo custom-frequency rows from ignored environment subtrees; the physics manager falls back from unvalidated cubric adapter versions that produced detached articulation links under Newton with Isaac RTX; visualization markers are torn down before interpreter shutdown to avoid destructor errors. Validated by full dexterous training runs on the Newton backend as part of the Task Clean-up campaign.
Greptile SummaryThis PR fixes three Newton backend bugs found during dexterous training: (1) empty MuJoCo custom-frequency rows created from ignored environment subtrees during USD import, (2) detached articulation links caused by accepting unvalidated cubric IAdapter minor versions, and (3) crashes during interpreter shutdown when
Confidence Score: 4/5Safe to merge; all three fixes are targeted, regression-tested, and have been validated by full dexterous training runs. The three fixes are well-scoped and backed by dedicated tests. The only notable concern is in source/isaaclab_newton/isaaclab_newton/cloner/newton_clone_utils.py — the regex path filter in _scope_filter deserves a second look for edge-case scenes with prim paths that share a prefix with the ignored subtrees. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A["_add_global_stage_to_builder(builder, stage, ignore_paths, schema_resolvers)"]
A --> B["Snapshot original usd_prim_filter\nfor each custom_frequency"]
B --> C["Wrap each existing filter\nwith _scope_filter (ignores ignored_paths)"]
C --> D["Intercept builder.add_custom_frequency\nvia instance setattr"]
D --> E["builder.add_usd(stage, ignore_paths,\nschema_resolvers)"]
E -->|New frequency registered| F["_add_scoped_custom_frequency:\nwrap filter, record original"]
E -->|Existing frequency re-registered| G["Check callbacks match;\nreturn early if same"]
E -->|Import error| H["Exception propagates to finally"]
E --> I["finally block:\nrestore all usd_prim_filter refs\ndelete instance override"]
H --> I
F --> I
G --> I
I --> J["Return stage_info"]
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
A["_add_global_stage_to_builder(builder, stage, ignore_paths, schema_resolvers)"]
A --> B["Snapshot original usd_prim_filter\nfor each custom_frequency"]
B --> C["Wrap each existing filter\nwith _scope_filter (ignores ignored_paths)"]
C --> D["Intercept builder.add_custom_frequency\nvia instance setattr"]
D --> E["builder.add_usd(stage, ignore_paths,\nschema_resolvers)"]
E -->|New frequency registered| F["_add_scoped_custom_frequency:\nwrap filter, record original"]
E -->|Existing frequency re-registered| G["Check callbacks match;\nreturn early if same"]
E -->|Import error| H["Exception propagates to finally"]
E --> I["finally block:\nrestore all usd_prim_filter refs\ndelete instance override"]
H --> I
F --> I
G --> I
I --> J["Return stage_info"]
Reviews (1): Last reviewed commit: "Fix Newton cloner rows, cubric fallback,..." | Re-trigger Greptile |
| ignored_patterns = tuple(re.compile(path) for path in ignore_paths) | ||
| original_filters = { | ||
| frequency_key: frequency.usd_prim_filter for frequency_key, frequency in builder.custom_frequencies.items() | ||
| } | ||
|
|
||
| def _scope_filter(callback): | ||
| if callback is None: | ||
| return None | ||
|
|
||
| def _filtered(prim: Usd.Prim, context: dict[str, Any]) -> bool: | ||
| prim_path = str(prim.GetPath()) | ||
| if any(pattern.match(prim_path) for pattern in ignored_patterns): |
There was a problem hiding this comment.
Regex path matching can produce false-positive exclusions
re.compile(path) uses the raw prim-path string as a regex pattern, and pattern.match(prim_path) anchors only at the start (not the end). This means /World/envs also matches /World/envs_global/..., and /World/envs/env_1 matches /World/envs/env_10/.... Any global prim whose path begins with one of the ignore_paths strings as a substring would be silently dropped from the custom-frequency traversal, even though Newton's own add_usd excludes only the intended exact subtrees.
For a correct prefix check, replace the regex compile and match with a plain string prefix comparison that also requires a / separator or an exact match (e.g. prim_path == path or prim_path.startswith(path + "/")), or at minimum use re.compile(re.escape(path) + r"(/|$)") to prevent mid-segment matches.
| # Load everything except the env subtrees (ground plane, lights, etc.) | ||
| ignore_paths = [path for _, path in env_paths] | ||
| builder.add_usd(stage, ignore_paths=ignore_paths, schema_resolvers=schema_resolvers) | ||
| from isaaclab_newton.cloner.newton_clone_utils import _add_global_stage_to_builder |
There was a problem hiding this comment.
Inline import of private cross-module function
_add_global_stage_to_builder is now used in three separate modules (replicate.py, visualization_builder.py, and here), which gives it de-facto public cross-module status. The leading underscore and this inline from import are inconsistent with that reach. Moving the import to the top-level imports of newton_manager.py (alongside the other newton_clone_utils imports already used in the file) would make the dependency explicit, and promoting the function to a non-underscore name would align its visibility with its actual API surface.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
It is consumed by three modules, so the leading underscore and the physics manager's inline import were misleading; imports move to the module top.
The audit guard treated any newer major as compatible; require the exact expected major.minor instead.
newton-physics/newton#3406 landed the fix upstream: the custom-frequency USD traversal now honors ignore_paths, so the local import-scoping workaround (add_global_stage_to_builder) is removed and its three call sites use builder.add_usd directly. The pin bump pulls in MuJoCo and mujoco-warp 3.10. Also requires the exact expected interface major.minor in the cubric audit.
Newton moves to current main (9af5a9f4181, 1.5.0.dev0), whose custom-frequency USD traversal honors ignore_paths. The pin requires the MuJoCo 3.10 stack while isaacsim-core exact-pins mujoco-warp 3.8, so [tool.uv] override-dependencies force mujoco/mujoco-warp 3.10 for every requester, and the newton-usd-schemas floor rises to 0.4.0 per newton main's [importers] extra. The uv-run pyproject test asserts the new overrides mirror the versions table.
AntoineRichard
left a comment
There was a problem hiding this comment.
Couple of small things otherwise it looks good
| # MuJoCo-stack overrides mirror the table (they outrank isaacsim-core's exact pins). | ||
| assert f"mujoco-warp{versions['mujoco_warp']}" in overrides | ||
| assert f"mujoco{versions['mujoco']}" in overrides | ||
|
|
There was a problem hiding this comment.
Is this needed? I can see why but does it belongs in this PR?
There was a problem hiding this comment.
There's a fix on newton side that's merged: newton-physics/newton#3406. The main problem is that it was pinned to older mjcwarp and updating newton needs newer version.
This can be done on a separate PR to update to the latest, at least resolve the dep issue I think.
There was a problem hiding this comment.
I didn't know that was a thing, that looks lovely /s
There was a problem hiding this comment.
Changes are reasonable otherwise.
There was a problem hiding this comment.
I think this is also in another PRs. Worth checking out like the solver coupling PR. https://github.com/isaac-sim/IsaacLab/pull/5834/changes#diff-1b68f4d0395901f033293c2fbb2f489b98135587b9e94bbe7fb560bff647498f
There was a problem hiding this comment.
I think it's different context though in the same file.
…eanup-dex-part01 # Conflicts: # source/isaaclab_newton/test/cloner/test_rename_builder_labels.py
…' into jichuanh/task-cleanup-dex-part01
…nager runtime (#6412) ## Summary - Fixes OVPhysX actuator joint indices to follow the common actuator indexing contract. - Fixes OVPhysX initialization alongside Kit by reusing Kit's registered PhysX schema provider. - Fixes the OVPhysX manager to support both the declared public runtime API and the current runtime API. - Regression tests included. Validated by full dexterous training runs on the OVPhysX backend; split out of the lumped validation branch #6324 (Part 2 of 11). ## Dependencies - None. ## Series review map Full integrated diff + training/validation evidence: the lumped validation PR #6324 (DO-NOT-MERGE). | Part | PR | |---|---| | Docs: regenerate the environment overview table | #6410 | | Part 1/11: Newton runtime fixes (cloner rows, cubric fallback, viz teardown) | #6411 | | **Part 2/11: OVPhysX runtime fixes (this PR)** | #6412 | | Part 3/11: success-rate metrics for the Direct reorientation tasks | #6413 | | Part 4/11: RSL-RL training for the handover Direct task | #6414 | | Part 5/11: success-rate support in the benchmark utilities | #6415 | | Part 6/11: renderer presets for the Direct camera task | #6416 | | Part 7/11: OVPhysX presets for the dexterous tasks | #6417 | | Part 8/11: Allegro manager counterpart | #6418 | | Part 9/11: Shadow + OpenAI manager counterparts | #6419 | | Part 10/11: Shadow camera manager counterpart | #6420 | | Part 11/11: Shadow handover manager counterpart | #6421 | --- ### Exact changes in this PR - OVPhysX backend changes + tests: 1f7a433
|
|
||
| Newton's built-in USD import honors ``ignore_paths``, but its custom-frequency | ||
| traversal currently does not. MuJoCo frequencies are also registered from | ||
| inside :meth:`ModelBuilder.add_usd`, so both existing and newly registered |
There was a problem hiding this comment.
Major: Register custom frequencies before attempting to scope them
The fresh global builders used by these call sites have no MuJoCo custom frequencies, and the repository's pinned Newton ModelBuilder.add_usd() does not call add_custom_frequency() despite this statement. Consequently original_filters is empty and the method interception never sees a registration. The newly added test_global_import_does_not_create_tendon_rows_from_ignored_envs fails with KeyError: 'mujoco:tendon', so this workaround does not establish the behavior it claims. Configure the global builder with the active solver's custom attributes before importing, preferably through the same shared builder setup used for source builders. Once registration is explicit, delete the speculative add_custom_frequency monkeypatch and duplicate callback-identity checks; the pinned importer has no path that needs them. Keep the regression assertion proving one global tendon row plus the replicated source rows.
There was a problem hiding this comment.
This will be removed once newton bump is done. Fix is already merged on newton side
| def remove_group(self, group_id: str) -> None: | ||
| self.removed_groups.append(group_id) | ||
|
|
||
| marker = object.__new__(newton_markers.NewtonVisualizationMarkers) |
There was a problem hiding this comment.
Minor: Exercise the constructor-to-shutdown registry lifecycle
Constructing the marker with object.__new__ and assigning _registry manually bypasses the new constructor behavior that caches sim.vis_marker_registry. This test would still pass if that assignment were later removed, even though real markers would again have no registry available once SimulationContext is torn down. Monkeypatch SimulationContext.instance() to return a fake context, instantiate NewtonVisualizationMarkers normally with a minimal marker config, then make instance() return None before calling close() twice. That directly protects registration, cached ownership, shutdown cleanup, and idempotency in one test.
There was a problem hiding this comment.
Adopted in 3443004 — the test now constructs NewtonVisualizationMarkers normally against a monkeypatched SimulationContext (a fake context exposing vis_marker_registry), then flips instance() to None and calls close() twice, asserting registration, cached ownership, shutdown cleanup, and idempotency as you described.
…eanup-dex-part01 # Conflicts: # source/isaaclab_newton/isaaclab_newton/cloner/newton_clone_utils.py
Newton commit 81cdcfc (merged via the develop pin bump) contains the upstream fix that makes ModelBuilder.add_usd honor ignore_paths in the custom-frequency traversal (newton#3406), so the scoped-import shim and its tests are no longer needed: the three call sites go back to direct add_usd calls. The behavior stays regression-tested inside Newton.
Address the review finding that the teardown test bypassed the constructor's registry caching: construct NewtonVisualizationMarkers normally against a monkeypatched SimulationContext, tear the context down, and close twice — covering registration, cached ownership, shutdown cleanup, and idempotency in one test.
…eorientation Direct tasks (#6413) ## Review Map - **Exact changes**: a stacked PR's page shows the cumulative diff of its dependency chain; the link pins the commit range that is the PR's own contribution. - Links pin specific SHAs and can go stale after a branch update — the table on #6324 is refreshed first. | PR | Status | Depends on | Exact changes | |---|---|---|---| | #6411 Part 1/9: Newton cloner/cubric/visualizer fixes |  | — | merged | | #6412 Part 2/9: OVPhysX articulation + manager runtime |  | — | merged | | 📌 #6413 Part 3/9: Reorient Direct, torch (this PR) |  | — | [changes](https://github.com/isaac-sim/IsaacLab/pull/6413/changes/f4895f0f9ee..d29afc75e71) | | #6414 Part 4/9: MARL-to-single-agent fix + handover/camera Direct |  | #6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6414/changes/d29afc75e71..b10a84948f8) | | #6418 Part 5/9: Reorient manager counterparts |  | #6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6418/changes/d29afc75e71..707d37f8f99) | | #6421 Part 6/9: Handover + camera manager counterparts |  | #6413, #6414, #6418 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6421/changes/707d37f8f99..b50ac8906fc) | | #6410 Part 7/9: Environment overview docs |  | #6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6410/changes/b50ac8906fc..af259c0778d) | | #6415 Part 8/9: Benchmark success-rate utilities |  | #6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6415/changes/b50ac8906fc..c7f2f019d8b) | | #6582 Part 9/9: Warp variants → experimental (draft; merges last) |  | #6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6582/changes/d29afc75e71..167c28578b3) | | #6324 [DO-NOT-MERGE] Lumped validation reference |  | ALL | — | ## Summary - Adds a behavioral `Metrics/success_rate` signal (goal-reach streaks per episode) and threshold-independent episode orientation-error diagnostics to the Direct reorientation environments; success gates task health, reward stays diagnostic. - Task logic is **torch-first** per the mainline convention (plain torch buffers; `.torch` accessors only at the core-lib boundary). This supersedes the earlier warp-first revision of this PR; the warp implementation moved to `isaaclab_tasks_experimental` (#6582). - Lands shared helpers in `isaaclab_tasks.core.utils` (`EpisodeErrorRecorder`, `sample_joint_positions_within_limits`) with torch math tests; fixes hand resets below lower joint limits. ## Stacking - First stack PR; based on `develop`. ## Validation - Kit-free torch math + utils tests green; Direct state training on Newton: success rate 0.85–1.00, reward ≈3600–3900 at 2000 iterations, frame-verified policy video. Full integrated evidence: #6324. ## Review history - The earlier warp-era review rounds on this PR are superseded by the torch restack (2026-07-16); the warp implementation and its review-polished kernels live on in #6582.
… and enable handover Direct RSL-RL (#6414) ## Review Map - **Exact changes**: a stacked PR's page shows the cumulative diff of its dependency chain; the link pins the commit range that is the PR's own contribution. This PR (#6414) is now rebased directly onto `develop`, so its own **Files changed** tab is its contribution. - Links pin specific SHAs and can go stale after a branch update — the table on #6324 is refreshed first. | PR | Status | Depends on | Exact changes | |---|---|---|---| | #6411 Part 1/9: Newton cloner/cubric/visualizer fixes |  | — | merged | | #6412 Part 2/9: OVPhysX articulation + manager runtime |  | — | merged | | #6413 Part 3/9: Reorient Direct, torch |  | — | merged | | 📌 #6414 Part 4/9: MARL-to-single-agent fix + handover/camera Direct (this PR) |  | — (on develop) | Files changed tab | | #6418 Part 5/9: Reorient manager counterparts |  | #6413 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6418/changes/d29afc75e71..707d37f8f99) | | #6421 Part 6/9: Handover + camera manager counterparts |  | #6414, #6418 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6421/changes/707d37f8f99..b50ac8906fc) | | #6410 Part 7/9: Environment overview docs |  | #6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6410/changes/b50ac8906fc..af259c0778d) | | #6415 Part 8/9: Benchmark success-rate utilities |  | #6421 | [changes](https://github.com/isaac-sim/IsaacLab/pull/6415/changes/b50ac8906fc..c7f2f019d8b) | | #6582 Part 9/9: Warp variants → experimental (draft; merges last) |  | — | [changes](https://github.com/isaac-sim/IsaacLab/pull/6582/changes/d29afc75e71..167c28578b3) | | #6324 [DO-NOT-MERGE] Lumped validation reference |  | ALL | — | ## Summary Converts the two-hand Shadow Hand **handover** task from multi-agent (MARL) to **single-agent** (enabling RSL-RL training), fixes the underlying MARL→single-agent observation bridge, and consolidates the shared Shadow Hand definitions into the asset. - **MARL → single-agent bridge (general, broader than this task).** Single-agent RL libraries train `DirectMARLEnv` tasks via `multi_agent_to_single_agent`; the bridge dropped the latest observations from the public buffer. Fixed generally — RSL-RL observations read from the env-owned `obs_buf`, stored by `reset` like `step` on all env bases (`DirectRLEnv`, `DirectMARLEnv`, the experimental warp base). Every MARL task + single-agent runner benefits; handover is the first consumer. - **Handover Direct → single-agent + RSL-RL.** Single-agent conversion, RSL-RL runner config, shared identity in `handover_common`, success-rate reward metrics, and a fix for its Newton construction failure (see below). - **Shadow Hand Newton robot → the asset.** Moved the Newton (MJWarp) robot cfg into `isaaclab_assets` as `SHADOW_HAND_NEWTON_CFG` (beside `SHADOW_HAND_CFG`); reorient and handover consume it as **siblings** (neither imports the other). Reorient uses the default gains; handover overrides only the finger stiffness/damping to `20/2` for its catch. - **Shadow camera cleanup.** Renamed the camera Direct modules to `shadow_hand_direct_camera_env[_cfg]`; moved cube-keypoint math to the shared, unit-tested `reorient/mdp/observations.py` (replacing the per-env `compute_keypoints` shim); and added OVPhysX physics presets to the handover and camera Direct environments. The camera env config is otherwise unchanged from develop. ### Behavior changes **Reorient-on-Newton is unchanged.** The Newton hand definition moves **verbatim** into `SHADOW_HAND_NEWTON_CFG` — same joint expressions, effort limits, gains, friction and armature — so its config diff vs `develop` is purely the move, and the Direct env config has no diff at all. Handover changes in three ways: - Its **default physics backend moves from PhysX to Newton** (MJWarp), matching the reorientation tasks. Pass `physics=physx` for the previous backend. - Its `distal_passive` actuator group is **removed**. It targeted `robot0_(FF|MF|RF|LF)J0` — joints that **do not exist** on the Newton asset (its fingers are numbered `J1`–`J4`/`J5`), so constructing the environment raised `No joints found for actuator group`. The `fingers` group already drives those joints, and the comment justifying the group (USD-baked `stiffness=286/damping=57`) does not apply to this asset, which authors no drive stiffness at all. - Two orientation fixes: the goal orientation was initialized to a 180-degree rotation instead of identity, and the Newton root rotation replaced the asset's baked rotation instead of composing with it, leaving both palms rotated 90 degrees. Two pre-existing discrepancies were found while verifying against the asset and are left for a follow-up, since each is a behavior change needing its own validation: the config sets `armature=2e-3` where the asset authors `0.0002`, and the per-finger `J1`/`J2` pair is coupled by a fixed tendon that the MJWarp solver currently skips — the actuator gains are what hold that pair together today, which is why the `J4` knuckle-abduction joints cannot simply be added to the driven set. ## Notes - Rebased onto **latest develop**; clean 3-commit history: `Add single-agent flattening for MARL Direct environments` / `Convert Shadow Hand handover to single-agent Direct with RSL-RL` / `Clean up Shadow reorient/camera tasks and consolidate the Newton hand`. ## Validation - Kit-free suite green: handover, keypoint math, the MARL adapter, and new checks pinning the RSL-RL wrapper observation contract (verified to fail against the previous implementation). - Camera Direct reaches training takeoff under `rsl_rl` (frame-verified videos); its env config now matches develop's established vision env. - **Reorient-on-Newton vs `develop`** (`rsl_rl`, 2048 envs, 1500 iterations, 4 runs per config): this branch spans 524–664 reward, `develop` spans 654–822 — overlapping distributions whose per-config spreads (140–168) exceed the gap between them. The move is additionally verified at the config level: the Newton robot configuration is character-for-character identical to `develop`'s. - **Handover-on-Newton** trains end-to-end on this branch (reward 1314), which the `develop` config cannot do at all — it fails during construction. Full evidence: #6324.
Review Map
Summary
ignore_pathsworkaround for custom-frequency USD traversal; it becomes redundant once the Newton pin advance (Pin Newton to v1.4.0 and override the Isaac Sim MuJoCo pins #6584) merges — this PR then only needs a rebase.Stacking
develop.Review history