Skip to content

[Task Clean-up][Newton] Dexterous Part 1/8: Fix cubric fallback and visualizer teardown - #6411

Merged
hujc7 merged 19 commits into
isaac-sim:developfrom
hujc7:jichuanh/task-cleanup-dex-part01
Jul 22, 2026
Merged

[Task Clean-up][Newton] Dexterous Part 1/8: Fix cubric fallback and visualizer teardown#6411
hujc7 merged 19 commits into
isaac-sim:developfrom
hujc7:jichuanh/task-cleanup-dex-part01

Conversation

@hujc7

@hujc7 hujc7 commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Review Map

PR Status Depends on Exact changes
#6410 [Docs] Environment overview regen
📌 #6411 Part 1/8: Newton cloner/cubric/visualizer fixes (this PR)
#6412 Part 2/8: OVPhysX articulation + manager runtime
#6413 Part 3/8: Reorient Direct, torch
#6414 Part 4/8: MARL-to-single-agent fix + handover/camera Direct #6413 changes
#6418 Part 5/8: Reorient manager counterparts #6413 changes
#6421 Part 6/8: Handover + camera manager counterparts #6413, #6414, #6418 changes
#6415 Part 7/8: Benchmark success-rate utilities + docs #6413, #6414, #6418, #6421 changes
#6582 Part 8/8: Warp variants → experimental (draft; merges last) #6413 changes
#6324 [DO-NOT-MERGE] Lumped validation reference ALL

Summary

  • Fixes Newton cloner label rows, the cubric IAdapter version audit (exact-match fallback to the CPU hierarchy path), and visualizer teardown.
  • Retains an in-tree ignore_paths workaround for custom-frequency USD traversal; it becomes redundant once the Newton pin advance (Pin Newton to v1.4.0 and override the Isaac Sim MuJoCo pins #6584) merges — this PR then only needs a rebase.

Stacking

  • Independent; based on develop.

Review history

Cloner imports no longer create empty MuJoCo custom-frequency rows from
ignored environment subtrees; the physics manager falls back from
unvalidated cubric adapter versions that produced detached articulation
links under Newton with Isaac RTX; visualization markers are torn down
before interpreter shutdown to avoid destructor errors.

Validated by full dexterous training runs on the Newton backend as part
of the Task Clean-up campaign.
@github-actions github-actions Bot added the isaac-lab Related to Isaac Lab team label Jul 8, 2026
@greptile-apps

greptile-apps Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes three Newton backend bugs found during dexterous training: (1) empty MuJoCo custom-frequency rows created from ignored environment subtrees during USD import, (2) detached articulation links caused by accepting unvalidated cubric IAdapter minor versions, and (3) crashes during interpreter shutdown when close() re-queried a potentially-torn-down SimulationContext.

  • Cloner fix: _add_global_stage_to_builder monkey-patches builder.add_custom_frequency during add_usd to scope MuJoCo-frequency prim filters to exclude ignored subtrees; a try/finally block guarantees filter restoration on failure.
  • Cubric fix: _verify_iadapter_version now requires an exact version match (minor != _IA_EXPECTED_MINOR) instead of the previous semver-compatible check (minor < _IA_EXPECTED_MINOR), since higher minor versions produced incorrect transform behavior with Isaac RTX.
  • Visualizer fix: NewtonVisualizationMarkers caches the registry reference at __init__ time so close() never calls SimulationContext.instance(); double-close is made idempotent. All three fixes have dedicated regression tests.

Confidence Score: 4/5

Safe to merge; all three fixes are targeted, regression-tested, and have been validated by full dexterous training runs.

The three fixes are well-scoped and backed by dedicated tests. The only notable concern is in _add_global_stage_to_builder, where re.compile(path).match(prim_path) is used for path filtering without anchoring at the end or escaping metacharacters — this could cause /World/envs to mistakenly exclude prims under a path like /World/envs_global. In the current usage patterns of IsaacLab this doesn't trigger, but it's a subtle correctness gap worth addressing.

source/isaaclab_newton/isaaclab_newton/cloner/newton_clone_utils.py — the regex path filter in _scope_filter deserves a second look for edge-case scenes with prim paths that share a prefix with the ignored subtrees.

Important Files Changed

Filename Overview
source/isaaclab_newton/isaaclab_newton/cloner/newton_clone_utils.py Adds _add_global_stage_to_builder which patches builder.add_custom_frequency to scope MuJoCo-frequency filters during USD import; restore logic via try/finally is correct, but the regex-based path filter uses re.match without escaping or end-anchoring which can over-exclude prims.
source/isaaclab_newton/isaaclab_newton/cloner/replicate.py Swaps builder.add_usd for _add_global_stage_to_builder to fix empty custom-frequency rows from ignored env subtrees; straightforward call-site change.
source/isaaclab_newton/isaaclab_newton/physics/_cubric.py Tightens IAdapter version check from semver-compatible (minor >=) to exact-match (minor ==), removing the proceed-under-semver-contract path that produced detached articulation links with Isaac RTX.
source/isaaclab_newton/isaaclab_newton/physics/newton_manager.py Applies the same _add_global_stage_to_builder fix for the non-clone single-builder path; the import is done inline inside the else block rather than at the module top.
source/isaaclab_visualizers/isaaclab_visualizers/newton/newton_visualization_markers.py Caches vis_marker_registry at init time instead of re-querying SimulationContext during close(), preventing AttributeError when module globals are torn down during interpreter shutdown; double-close is also made idempotent.
source/isaaclab_newton/test/cloner/test_rename_builder_labels.py Adds TestIgnoredCloneCustomFrequencies with two tests: one verifying tendon rows are not created from ignored env subtrees, another verifying filter restoration after import failure; coverage is thorough.
source/isaaclab_newton/test/physics/test_cubric.py New regression test for the exact-match version change; constructs an in-memory ctypes plugin descriptor and asserts that v0.2 is rejected while v0.1 is accepted.
source/isaaclab_visualizers/test/test_newton_adapter.py Adds test_newton_marker_close_does_not_query_simulation_context verifying the shutdown-safe close() path; double-close and idempotency are both exercised.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["_add_global_stage_to_builder(builder, stage, ignore_paths, schema_resolvers)"]
    A --> B["Snapshot original usd_prim_filter\nfor each custom_frequency"]
    B --> C["Wrap each existing filter\nwith _scope_filter (ignores ignored_paths)"]
    C --> D["Intercept builder.add_custom_frequency\nvia instance setattr"]
    D --> E["builder.add_usd(stage, ignore_paths,\nschema_resolvers)"]
    E -->|New frequency registered| F["_add_scoped_custom_frequency:\nwrap filter, record original"]
    E -->|Existing frequency re-registered| G["Check callbacks match;\nreturn early if same"]
    E -->|Import error| H["Exception propagates to finally"]
    E --> I["finally block:\nrestore all usd_prim_filter refs\ndelete instance override"]
    H --> I
    F --> I
    G --> I
    I --> J["Return stage_info"]
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A["_add_global_stage_to_builder(builder, stage, ignore_paths, schema_resolvers)"]
    A --> B["Snapshot original usd_prim_filter\nfor each custom_frequency"]
    B --> C["Wrap each existing filter\nwith _scope_filter (ignores ignored_paths)"]
    C --> D["Intercept builder.add_custom_frequency\nvia instance setattr"]
    D --> E["builder.add_usd(stage, ignore_paths,\nschema_resolvers)"]
    E -->|New frequency registered| F["_add_scoped_custom_frequency:\nwrap filter, record original"]
    E -->|Existing frequency re-registered| G["Check callbacks match;\nreturn early if same"]
    E -->|Import error| H["Exception propagates to finally"]
    E --> I["finally block:\nrestore all usd_prim_filter refs\ndelete instance override"]
    H --> I
    F --> I
    G --> I
    I --> J["Return stage_info"]
Loading

Reviews (1): Last reviewed commit: "Fix Newton cloner rows, cubric fallback,..." | Re-trigger Greptile

Comment on lines +36 to +47
ignored_patterns = tuple(re.compile(path) for path in ignore_paths)
original_filters = {
frequency_key: frequency.usd_prim_filter for frequency_key, frequency in builder.custom_frequencies.items()
}

def _scope_filter(callback):
if callback is None:
return None

def _filtered(prim: Usd.Prim, context: dict[str, Any]) -> bool:
prim_path = str(prim.GetPath())
if any(pattern.match(prim_path) for pattern in ignored_patterns):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Regex path matching can produce false-positive exclusions

re.compile(path) uses the raw prim-path string as a regex pattern, and pattern.match(prim_path) anchors only at the start (not the end). This means /World/envs also matches /World/envs_global/..., and /World/envs/env_1 matches /World/envs/env_10/.... Any global prim whose path begins with one of the ignore_paths strings as a substring would be silently dropped from the custom-frequency traversal, even though Newton's own add_usd excludes only the intended exact subtrees.

For a correct prefix check, replace the regex compile and match with a plain string prefix comparison that also requires a / separator or an exact match (e.g. prim_path == path or prim_path.startswith(path + "/")), or at minimum use re.compile(re.escape(path) + r"(/|$)") to prevent mid-segment matches.

Comment on lines +1295 to +1297
# Load everything except the env subtrees (ground plane, lights, etc.)
ignore_paths = [path for _, path in env_paths]
builder.add_usd(stage, ignore_paths=ignore_paths, schema_resolvers=schema_resolvers)
from isaaclab_newton.cloner.newton_clone_utils import _add_global_stage_to_builder

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Inline import of private cross-module function

_add_global_stage_to_builder is now used in three separate modules (replicate.py, visualization_builder.py, and here), which gives it de-facto public cross-module status. The leading underscore and this inline from import are inconsistent with that reach. Moving the import to the top-level imports of newton_manager.py (alongside the other newton_clone_utils imports already used in the file) would make the dependency explicit, and promoting the function to a non-underscore name would align its visibility with its actual API surface.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

hujc7 added 5 commits July 8, 2026 20:54
It is consumed by three modules, so the leading underscore and the
physics manager's inline import were misleading; imports move to the
module top.
The audit guard treated any newer major as compatible; require the
exact expected major.minor instead.
newton-physics/newton#3406 landed the fix upstream: the custom-frequency
USD traversal now honors ignore_paths, so the local import-scoping
workaround (add_global_stage_to_builder) is removed and its three call
sites use builder.add_usd directly. The pin bump pulls in MuJoCo and
mujoco-warp 3.10. Also requires the exact expected interface major.minor
in the cubric audit.
Newton moves to current main (9af5a9f4181, 1.5.0.dev0), whose
custom-frequency USD traversal honors ignore_paths. The pin requires
the MuJoCo 3.10 stack while isaacsim-core exact-pins mujoco-warp 3.8,
so [tool.uv] override-dependencies force mujoco/mujoco-warp 3.10 for
every requester, and the newton-usd-schemas floor rises to 0.4.0 per
newton main's [importers] extra. The uv-run pyproject test asserts
the new overrides mirror the versions table.

@AntoineRichard AntoineRichard left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple of small things otherwise it looks good

Comment on lines +104 to +107
# MuJoCo-stack overrides mirror the table (they outrank isaacsim-core's exact pins).
assert f"mujoco-warp{versions['mujoco_warp']}" in overrides
assert f"mujoco{versions['mujoco']}" in overrides

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this needed? I can see why but does it belongs in this PR?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a fix on newton side that's merged: newton-physics/newton#3406. The main problem is that it was pinned to older mjcwarp and updating newton needs newer version.
This can be done on a separate PR to update to the latest, at least resolve the dep issue I think.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't know that was a thing, that looks lovely /s

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes are reasonable otherwise.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is also in another PRs. Worth checking out like the solver coupling PR. https://github.com/isaac-sim/IsaacLab/pull/5834/changes#diff-1b68f4d0395901f033293c2fbb2f489b98135587b9e94bbe7fb560bff647498f

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's different context though in the same file.

@hujc7 hujc7 changed the title [Task Clean-up][Newton] Dexterous Part 1/11: Fix cloner rows, cubric fallback, and visualizer teardown [Task Clean-up][Newton] Dexterous Part 1: Fix cloner rows, cubric fallback, and visualizer teardown Jul 17, 2026
@hujc7 hujc7 changed the title [Task Clean-up][Newton] Dexterous Part 1: Fix cloner rows, cubric fallback, and visualizer teardown [Task Clean-up][Newton] Dexterous Part 1/11: Fix cloner rows, cubric fallback, and visualizer teardown Jul 17, 2026
hujc7 added a commit that referenced this pull request Jul 17, 2026
…nager runtime (#6412)

## Summary

- Fixes OVPhysX actuator joint indices to follow the common actuator
indexing contract.
- Fixes OVPhysX initialization alongside Kit by reusing Kit's registered
PhysX schema provider.
- Fixes the OVPhysX manager to support both the declared public runtime
API and the current runtime API.
- Regression tests included. Validated by full dexterous training runs
on the OVPhysX backend; split out of the lumped validation branch #6324
(Part 2 of 11).

## Dependencies

- None.

## Series review map

Full integrated diff + training/validation evidence: the lumped
validation PR #6324
(DO-NOT-MERGE).

| Part | PR |
|---|---|
| Docs: regenerate the environment overview table |
#6410 |
| Part 1/11: Newton runtime fixes (cloner rows, cubric fallback, viz
teardown) | #6411 |
| **Part 2/11: OVPhysX runtime fixes (this PR)** |
#6412 |
| Part 3/11: success-rate metrics for the Direct reorientation tasks |
#6413 |
| Part 4/11: RSL-RL training for the handover Direct task |
#6414 |
| Part 5/11: success-rate support in the benchmark utilities |
#6415 |
| Part 6/11: renderer presets for the Direct camera task |
#6416 |
| Part 7/11: OVPhysX presets for the dexterous tasks |
#6417 |
| Part 8/11: Allegro manager counterpart |
#6418 |
| Part 9/11: Shadow + OpenAI manager counterparts |
#6419 |
| Part 10/11: Shadow camera manager counterpart |
#6420 |
| Part 11/11: Shadow handover manager counterpart |
#6421 |


---
### Exact changes in this PR

- OVPhysX backend changes + tests:
1f7a433
@hujc7 hujc7 changed the title [Task Clean-up][Newton] Dexterous Part 1/11: Fix cloner rows, cubric fallback, and visualizer teardown [Task Clean-up][Newton] Dexterous Part 1/8: Fix cloner rows, cubric fallback, and visualizer teardown Jul 17, 2026

Newton's built-in USD import honors ``ignore_paths``, but its custom-frequency
traversal currently does not. MuJoCo frequencies are also registered from
inside :meth:`ModelBuilder.add_usd`, so both existing and newly registered

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major: Register custom frequencies before attempting to scope them

The fresh global builders used by these call sites have no MuJoCo custom frequencies, and the repository's pinned Newton ModelBuilder.add_usd() does not call add_custom_frequency() despite this statement. Consequently original_filters is empty and the method interception never sees a registration. The newly added test_global_import_does_not_create_tendon_rows_from_ignored_envs fails with KeyError: 'mujoco:tendon', so this workaround does not establish the behavior it claims. Configure the global builder with the active solver's custom attributes before importing, preferably through the same shared builder setup used for source builders. Once registration is explicit, delete the speculative add_custom_frequency monkeypatch and duplicate callback-identity checks; the pinned importer has no path that needs them. Keep the regression assertion proving one global tendon row plus the replicated source rows.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be removed once newton bump is done. Fix is already merged on newton side

def remove_group(self, group_id: str) -> None:
self.removed_groups.append(group_id)

marker = object.__new__(newton_markers.NewtonVisualizationMarkers)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: Exercise the constructor-to-shutdown registry lifecycle

Constructing the marker with object.__new__ and assigning _registry manually bypasses the new constructor behavior that caches sim.vis_marker_registry. This test would still pass if that assignment were later removed, even though real markers would again have no registry available once SimulationContext is torn down. Monkeypatch SimulationContext.instance() to return a fake context, instantiate NewtonVisualizationMarkers normally with a minimal marker config, then make instance() return None before calling close() twice. That directly protects registration, cached ownership, shutdown cleanup, and idempotency in one test.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adopted in 3443004 — the test now constructs NewtonVisualizationMarkers normally against a monkeypatched SimulationContext (a fake context exposing vis_marker_registry), then flips instance() to None and calls close() twice, asserting registration, cached ownership, shutdown cleanup, and idempotency as you described.

hujc7 added 2 commits July 21, 2026 12:01
…eanup-dex-part01

# Conflicts:
#	source/isaaclab_newton/isaaclab_newton/cloner/newton_clone_utils.py
Newton commit 81cdcfc (merged via the develop pin bump) contains the
upstream fix that makes ModelBuilder.add_usd honor ignore_paths in the
custom-frequency traversal (newton#3406), so the scoped-import shim and
its tests are no longer needed: the three call sites go back to direct
add_usd calls. The behavior stays regression-tested inside Newton.
@hujc7 hujc7 changed the title [Task Clean-up][Newton] Dexterous Part 1/8: Fix cloner rows, cubric fallback, and visualizer teardown [Task Clean-up][Newton] Dexterous Part 1/8: Fix cubric fallback and visualizer teardown Jul 21, 2026
@hujc7
hujc7 requested a review from a team July 21, 2026 19:03
hujc7 added 3 commits July 21, 2026 12:15
Address the review finding that the teardown test bypassed the
constructor's registry caching: construct NewtonVisualizationMarkers
normally against a monkeypatched SimulationContext, tear the context
down, and close twice — covering registration, cached ownership,
shutdown cleanup, and idempotency in one test.
@hujc7
hujc7 merged commit cb50838 into isaac-sim:develop Jul 22, 2026
65 of 66 checks passed
hujc7 added a commit that referenced this pull request Jul 24, 2026
…eorientation Direct tasks (#6413)

## Review Map

- **Exact changes**: a stacked PR's page shows the cumulative diff of
its dependency chain; the link pins the commit range that is the PR's
own contribution.
- Links pin specific SHAs and can go stale after a branch update — the
table on #6324 is refreshed first.

| PR | Status | Depends on | Exact changes |
|---|---|---|---|
| #6411 Part 1/9: Newton cloner/cubric/visualizer fixes |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6411?label=)
| — | merged |
| #6412 Part 2/9: OVPhysX articulation + manager runtime |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6412?label=)
| — | merged |
| 📌 #6413 Part 3/9: Reorient Direct, torch (this PR) |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6413?label=)
| — |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6413/changes/f4895f0f9ee..d29afc75e71)
|
| #6414 Part 4/9: MARL-to-single-agent fix + handover/camera Direct |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6414?label=)
| #6413 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6414/changes/d29afc75e71..b10a84948f8)
|
| #6418 Part 5/9: Reorient manager counterparts |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6418?label=)
| #6413 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6418/changes/d29afc75e71..707d37f8f99)
|
| #6421 Part 6/9: Handover + camera manager counterparts |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6421?label=)
| #6413, #6414, #6418 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6421/changes/707d37f8f99..b50ac8906fc)
|
| #6410 Part 7/9: Environment overview docs |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6410?label=)
| #6421 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6410/changes/b50ac8906fc..af259c0778d)
|
| #6415 Part 8/9: Benchmark success-rate utilities |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6415?label=)
| #6421 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6415/changes/b50ac8906fc..c7f2f019d8b)
|
| #6582 Part 9/9: Warp variants → experimental (draft; merges last) |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6582?label=)
| #6413 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6582/changes/d29afc75e71..167c28578b3)
|
| #6324 [DO-NOT-MERGE] Lumped validation reference |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6324?label=)
| ALL | — |

## Summary

- Adds a behavioral `Metrics/success_rate` signal (goal-reach streaks
per episode) and threshold-independent episode orientation-error
diagnostics to the Direct reorientation environments; success gates task
health, reward stays diagnostic.
- Task logic is **torch-first** per the mainline convention (plain torch
buffers; `.torch` accessors only at the core-lib boundary). This
supersedes the earlier warp-first revision of this PR; the warp
implementation moved to `isaaclab_tasks_experimental`
(#6582).
- Lands shared helpers in `isaaclab_tasks.core.utils`
(`EpisodeErrorRecorder`, `sample_joint_positions_within_limits`) with
torch math tests; fixes hand resets below lower joint limits.

## Stacking

- First stack PR; based on `develop`.

## Validation

- Kit-free torch math + utils tests green; Direct state training on
Newton: success rate 0.85–1.00, reward ≈3600–3900 at 2000 iterations,
frame-verified policy video. Full integrated evidence: #6324.

## Review history

- The earlier warp-era review rounds on this PR are superseded by the
torch restack (2026-07-16); the warp implementation and its
review-polished kernels live on in
#6582.
hujc7 added a commit that referenced this pull request Jul 29, 2026
… and enable handover Direct RSL-RL (#6414)

## Review Map

- **Exact changes**: a stacked PR's page shows the cumulative diff of
its dependency chain; the link pins the commit range that is the PR's
own contribution. This PR (#6414) is now rebased directly onto
`develop`, so its own **Files changed** tab is its contribution.
- Links pin specific SHAs and can go stale after a branch update — the
table on #6324 is refreshed first.

| PR | Status | Depends on | Exact changes |
|---|---|---|---|
| #6411 Part 1/9: Newton cloner/cubric/visualizer fixes |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6411?label=)
| — | merged |
| #6412 Part 2/9: OVPhysX articulation + manager runtime |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6412?label=)
| — | merged |
| #6413 Part 3/9: Reorient Direct, torch |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6413?label=)
| — | merged |
| 📌 #6414 Part 4/9: MARL-to-single-agent fix + handover/camera Direct
(this PR) |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6414?label=)
| — (on develop) | Files changed tab |
| #6418 Part 5/9: Reorient manager counterparts |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6418?label=)
| #6413 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6418/changes/d29afc75e71..707d37f8f99)
|
| #6421 Part 6/9: Handover + camera manager counterparts |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6421?label=)
| #6414, #6418 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6421/changes/707d37f8f99..b50ac8906fc)
|
| #6410 Part 7/9: Environment overview docs |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6410?label=)
| #6421 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6410/changes/b50ac8906fc..af259c0778d)
|
| #6415 Part 8/9: Benchmark success-rate utilities |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6415?label=)
| #6421 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6415/changes/b50ac8906fc..c7f2f019d8b)
|
| #6582 Part 9/9: Warp variants → experimental (draft; merges last) |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6582?label=)
| — |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6582/changes/d29afc75e71..167c28578b3)
|
| #6324 [DO-NOT-MERGE] Lumped validation reference |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6324?label=)
| ALL | — |

## Summary

Converts the two-hand Shadow Hand **handover** task from multi-agent
(MARL) to **single-agent** (enabling RSL-RL training), fixes the
underlying MARL→single-agent observation bridge, and consolidates the
shared Shadow Hand definitions into the asset.

- **MARL → single-agent bridge (general, broader than this task).**
Single-agent RL libraries train `DirectMARLEnv` tasks via
`multi_agent_to_single_agent`; the bridge dropped the latest
observations from the public buffer. Fixed generally — RSL-RL
observations read from the env-owned `obs_buf`, stored by `reset` like
`step` on all env bases (`DirectRLEnv`, `DirectMARLEnv`, the
experimental warp base). Every MARL task + single-agent runner benefits;
handover is the first consumer.
- **Handover Direct → single-agent + RSL-RL.** Single-agent conversion,
RSL-RL runner config, shared identity in `handover_common`, success-rate
reward metrics, and a fix for its Newton construction failure (see
below).
- **Shadow Hand Newton robot → the asset.** Moved the Newton (MJWarp)
robot cfg into `isaaclab_assets` as `SHADOW_HAND_NEWTON_CFG` (beside
`SHADOW_HAND_CFG`); reorient and handover consume it as **siblings**
(neither imports the other). Reorient uses the default gains; handover
overrides only the finger stiffness/damping to `20/2` for its catch.
- **Shadow camera cleanup.** Renamed the camera Direct modules to
`shadow_hand_direct_camera_env[_cfg]`; moved cube-keypoint math to the
shared, unit-tested `reorient/mdp/observations.py` (replacing the
per-env `compute_keypoints` shim); and added OVPhysX physics presets to
the handover and camera Direct environments. The camera env config is
otherwise unchanged from develop.

### Behavior changes

**Reorient-on-Newton is unchanged.** The Newton hand definition moves
**verbatim** into `SHADOW_HAND_NEWTON_CFG` — same joint expressions,
effort limits, gains, friction and armature — so its config diff vs
`develop` is purely the move, and the Direct env config has no diff at
all.

Handover changes in three ways:

- Its **default physics backend moves from PhysX to Newton** (MJWarp),
matching the reorientation tasks. Pass `physics=physx` for the previous
backend.
- Its `distal_passive` actuator group is **removed**. It targeted
`robot0_(FF|MF|RF|LF)J0` — joints that **do not exist** on the Newton
asset (its fingers are numbered `J1`–`J4`/`J5`), so constructing the
environment raised `No joints found for actuator group`. The `fingers`
group already drives those joints, and the comment justifying the group
(USD-baked `stiffness=286/damping=57`) does not apply to this asset,
which authors no drive stiffness at all.
- Two orientation fixes: the goal orientation was initialized to a
180-degree rotation instead of identity, and the Newton root rotation
replaced the asset's baked rotation instead of composing with it,
leaving both palms rotated 90 degrees.

Two pre-existing discrepancies were found while verifying against the
asset and are left for a follow-up, since each is a behavior change
needing its own validation: the config sets `armature=2e-3` where the
asset authors `0.0002`, and the per-finger `J1`/`J2` pair is coupled by
a fixed tendon that the MJWarp solver currently skips — the actuator
gains are what hold that pair together today, which is why the `J4`
knuckle-abduction joints cannot simply be added to the driven set.

## Notes

- Rebased onto **latest develop**; clean 3-commit history: `Add
single-agent flattening for MARL Direct environments` / `Convert Shadow
Hand handover to single-agent Direct with RSL-RL` / `Clean up Shadow
reorient/camera tasks and consolidate the Newton hand`.

## Validation

- Kit-free suite green: handover, keypoint math, the MARL adapter, and
new checks pinning the RSL-RL wrapper observation contract (verified to
fail against the previous implementation).
- Camera Direct reaches training takeoff under `rsl_rl` (frame-verified
videos); its env config now matches develop's established vision env.
- **Reorient-on-Newton vs `develop`** (`rsl_rl`, 2048 envs, 1500
iterations, 4 runs per config): this branch spans 524–664 reward,
`develop` spans 654–822 — overlapping distributions whose per-config
spreads (140–168) exceed the gap between them. The move is additionally
verified at the config level: the Newton robot configuration is
character-for-character identical to `develop`'s.
- **Handover-on-Newton** trains end-to-end on this branch (reward 1314),
which the `develop` config cannot do at all — it fails during
construction. Full evidence: #6324.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

infrastructure isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants