Skip to content

[DO-NOT-MERGE][Task Clean-up] Dexterous: lumped validation branch (series reference) - #6324

Open
hujc7 wants to merge 75 commits into
isaac-sim:developfrom
hujc7:jichuanh/dexterous-env-convergence-codex
Open

[DO-NOT-MERGE][Task Clean-up] Dexterous: lumped validation branch (series reference)#6324
hujc7 wants to merge 75 commits into
isaac-sim:developfrom
hujc7:jichuanh/dexterous-env-convergence-codex

Conversation

@hujc7

@hujc7 hujc7 commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

DO NOT MERGE — integrated reference branch for the dexterous task-cleanup series; the reviewable stack is in the Review Map below. Per-part details live in the part PRs.

Update (2026-07-16) — torch-first rework: the mainline task code was reworked torch-first per the stable task convention; the warp implementation moved to isaaclab_tasks_experimental (#6582, preserved from this branch's history).

Update (2026-07-17) — restack + validation: all parts rebuilt on the torch tree and CI-green; the Newton pin split out to #6584. Torch re-validation: Direct state SR 0.85–1.00 @2000 it; camera rsl_rl takeoff (reward peak 1222, SR peak 1.0) @1500 it — both with frame-verified videos.

Update (2026-07-18) — lump review round propagated: Parts 5/6/8 histories rebuilt with the review folded in (they had no human review yet); Parts 3/4 received append commits; Part 7 merged the rebuilt base. Union-of-parts verified byte-identical to this branch. The Newton pin remnants left this branch and live only in #6584.

Review Map

PR Status Depends on Exact changes
#6411 Part 1/9: Newton cloner/cubric/visualizer fixes merged
#6412 Part 2/9: OVPhysX articulation + manager runtime merged
#6413 Part 3/9: Reorient Direct, torch changes
#6414 Part 4/9: MARL-to-single-agent fix + handover/camera Direct #6413 changes
#6418 Part 5/9: Reorient manager counterparts #6413 changes
#6421 Part 6/9: Handover + camera manager counterparts #6413, #6414, #6418 changes
#6410 Part 7/9: Environment overview docs #6421 changes
#6415 Part 8/9: Benchmark success-rate utilities #6421 changes
#6582 Part 9/9: Warp variants → experimental (draft; merges last) #6413 changes
📌 #6324 [DO-NOT-MERGE] Lumped validation reference (this PR) ALL

Summary

  • Manager-based counterparts, boolean success metrics (Metrics/success_rate), and PhysX/Newton/OVPhysX presets for the assigned dexterous tasks (Allegro + Shadow reorientation, OpenAI FF/LSTM, camera, handover).
  • Fixes handover construction on Newton (renamed distal joints) and hand resets below joint limits.
  • The campaign table below is the warp-era record; rows refresh as torch reruns land.
  • Per-part scope, review updates, and validation: see each part PR (Review Map above).

Direct vs manager training reference (qualifying campaign, seed 42)

Cell format: reward|eplen|success_rate|{trained}iter (final Train/mean_reward, Train/mean_episode_length, Metrics/success_rate).

Iteration budgets: 500 (Allegro), 3000 (all other rows). Every cell's source run consumed its full budget — verified cell-by-cell against the runs' event files (all three metrics of each cell match one run's final-iteration scalars). Cells read reward | episode length | success rate | iterations actually trained.

Task Backend Direct Manager
Reorient Allegro PhysX 340 | 259 | 0.63 | 500iter 212 | 285 | 0.23 | 500iter
Reorient Allegro Newton 267 | 254 | 0.48 | 500iter 203 | 295 | 0.17 | 500iter
Reorient Allegro OVPhysX⁴ 308 | 252 | 0.68 | 500iter 165 | 286 | 0.18 | 500iter
Reorient Shadow PhysX 5725 | 553 | 1.00 | 3000iter 6066 | 566 | 0.99 | 3000iter
Reorient Shadow Newton 3783 | 523 | 0.85 | 3000iter 3866 | 534 | 0.86 | 3000iter
Reorient Shadow OVPhysX⁴ 3337 | 434 | 1.00 | 3000iter 4062 | 479 | 0.97 | 3000iter
Shadow OpenAI FF PhysX 5422 | 480 | 1.00 | 3000iter 3688 | 599 | 0.94 | 3000iter
Shadow OpenAI FF Newton 2762 | 356 | 0.67 | 3000iter 2779 | 507 | 0.75 | 3000iter
Shadow OpenAI FF OVPhysX⁴ ⁵ 317 | 162 | 0.62 | 3000iter 198 | 238 | 0.67 | 3000iter
Shadow OpenAI LSTM PhysX 5640 | 605 | 1.00 | 3000iter 3811 | 809 | 0.93 | 3000iter
Shadow OpenAI LSTM Newton 3441 | 479 | 0.75 | 3000iter 2278 | 503 | 0.76 | 3000iter
Shadow OpenAI LSTM OVPhysX⁴ ⁵ -16 | 261 | 0.33 | 3000iter 41 | 285 | 0.42 | 3000iter
Shadow Camera PhysX 224 | 564 | 0.00² | 3000iter 243 | 595 | 0.00² | 3000iter
Shadow Camera Newton 1307 | 503 | 0.83 | 3000iter³ 164 | 457 | 0.00² | 3000iter
Shadow Handover PhysX 1634 | 441 | 1.00 | 3000iter 724 | 367 | 0.99 | 3000iter
Shadow Handover Newton 1471 | 412 | 1.00 | 3000iter¹ 1415 | 400 | 0.93 | 3000iter¹
Shadow Handover OVPhysX⁴ 1613 | 445 | 1.00 | 3000iter 1619 | 446 | 1.00 | 3000iter

¹ Post-fix handover-Newton rows (Part 4's Newton fixes), full-budget qualifying runs on the warp-first environment code.
² Same success-rate definition as the other reorientation rows. Camera learning takes off late (reward ~400 plateau until a sharp transition; PhysX evidence crosses reward 1000 near iter ~1000, Newton near iter ~2300 — see ³); these rows' policies remained pre-takeoff at the recorded budget, so success at the 0.1 rad tolerance reads 0.00 and reward is the tracked pre-takeoff signal. The criterion itself is unchanged and proven reachable.
³ Full-budget qualifying run on the current branch head (seed 42, rgb_depth preset, Newton renderer, 1225 envs, rl_games workflow with its asymmetric actor-critic recipe): crossed reward 1000 at iter ~2300, tail-50 reward 1307, success rate 0.83 with ≥0.3 sustained for 43 consecutive iterations. Note the workflow difference: the other camera cells (and the rest of the table) are rsl_rl runs; the camera task has so far reached takeoff only under the rl_games recipe (also PhysX evidence: reward 1018, success 0.91 by iter ~1025), while the rsl_rl camera recipe remains pre-takeoff at full budget — a training-recipe sensitivity to be reconciled, not a physics/backend gap. Superseded on the torch tree: the torch-rework rsl_rl camera run reaches takeoff (reward > 1000, sr peak 1.0) by ~1500 iterations — the rsl_rl/rl_games recipe sensitivity no longer reproduces after the rework.
⁴ OVPhysX rows validated 2026-07-21/22 (same protocol: seed 42, 8192 envs, rsl_rl, full budget; env pins ovphysx==0.4.13 + omniverseclient 2.72.1 — the paired versions published on pypi.nvidia.com). First OVPhysX validation of these tasks.
⁵ The OpenAI variants' stock stepping (dt=1/60, decimation 3) is unstable on the OVPhysX runtime: grasp contact is lost and the cube free-falls from reset (~9-step episodes from iteration 0), while Kit-PhysX trains the identical config. Probes isolate the physics step size as the sole trigger; every other ovphysx-enabled task in the tree steps at 1/120 or finer. These cells run at dt=1/120, decimation 2 (the state rows' stepping). Success rate is the cross-backend comparison metric for these rows: the finer stepping triples the per-episode step count, so cumulative reward and episode length are not comparable to the dt=1/60 reference cells.
Reward scales differ per family; the Direct-vs-manager pair per row is the comparison that matters.

Torch re-validation (post-rework). Verified runs on the torch tree; the table below remains the warp-era campaign record.

Task Workflow Torch result Warp-era, same recipe
Reorient Shadow Direct Newton rsl_rl 3600–3900 | sr 0.85–1.00 | 2000it 3783 | 0.85 | 3000it
Shadow Camera Direct Newton rsl_rl peak 1222 | sr peak 1.0 | 1500it (takeoff) pre-takeoff @3000it²
Shadow Camera Manager Newton rsl_rl 2023 | sr 0.87 | 3000it (takeoff) 0.00 @3000it² (pre-takeoff)

Phase-2 requirement fulfillment

🟢 Met   🟡 Partial / queued   🔴 Gap   ⚪ Not applicable

Overall Phase-2 status Count Share of 392 applicable requirements
🟢 Met 291 74%
🟡 Partial / queued 101 26%
🔴 Gap 0 0%
Estimated progress 291 met + 0.5 × 101 partial 87%

118 N/A cells are excluded. Success-related acceptance criteria (P2-C3–C7 and dependent P2-D1–D2) are deferred pending definition; no provisional threshold is treated as a requirement. Reward and episode length remain diagnostic only. Previously validated evidence is retained wherever the code path is unchanged by this diff (the OVPhysX path always used the production assets); only artifacts tied to the removed asset are queued for re-run. Sim-to-sim and sim-to-real are later waves and excluded. Current PR head: 5d7ddbbee82.

Allegro cube reorientation — 🟢 49 · 🟡 15 · 88%
Registered variant 🟢 🟡 🔴 Progress
Isaac-Reorient-Cube-Allegro-Direct 18 7 0 5 86%
Isaac-Reorient-Cube-Allegro 18 7 0 5 86%
Isaac-Reorient-Cube-Allegro-Play 13 1 0 16 96%
AreaID · RequirementDirect trainManager trainManager Play
StructureP2-S1 · Assigned registration accounted for🟢 Registered and assigned🟢 Registered and assigned🟢 Registered and assigned
P2-S2 · Manager-based training environment🟢 Manager train exists for family🟢 Manager train exists for family🟢 Manager train exists for family
P2-S3 · Direct/Manager contracts comparable🟢 Focused parity checks pass🟢 Focused parity checks pass🟢 Focused parity checks pass
P2-S4 · Rendering identified by Camera name⚪ Non-rendering⚪ Non-rendering⚪ Non-rendering
P2-S5 · Play/Eval implementation consolidated⚪ Not a Play/Eval variant⚪ Not a Play/Eval variant🟡 Public Play ID retained; generic consolidation pending
Assets & backendsP2-B1 · New asset evaluated; fallback justified⚪ Dropped: new-asset adoption removed from the consolidated standard⚪ Dropped: new-asset adoption removed from the consolidated standard⚪ Dropped: new-asset adoption removed from the consolidated standard
P2-B2 · PhysX / Newton / OV presets resolve🟢 All three physics presets resolve🟢 All three physics presets resolve🟢 All three physics presets resolve
P2-B3 · PhysX constructs and trains🟢 Exact-head PhysX train smoke and full-budget training pass🟢 Exact-head PhysX train smoke and full-budget training pass🟢 Deterministic 200-step PhysX checkpoint trace passes on restored assets
P2-B3 · Newton constructs and trains🟢 Exact-head Newton train smoke and full-budget training pass🟢 Exact-head Newton train smoke and full-budget training pass🟢 Deterministic 200-step Newton checkpoint trace passes on restored assets
P2-B3 · OVPhysX constructs and trains🟢 Exact-head OVPhysX train smoke and full-budget training pass🟢 Exact-head OVPhysX train smoke and full-budget training pass🟢 Exact 50-step OVPhysX checkpoint trace passes (path unchanged by asset restoration)
P2-B4 · Supported renderer matrix works⚪ Non-camera⚪ Non-camera⚪ Non-camera
P2-B5 · Camera framing and ground are usable⚪ Non-camera⚪ Non-camera⚪ Non-camera
SuccessP2-C1 · RSL-RL configuration provided🟢 RSL-RL config resolves🟢 RSL-RL config resolves⚪ Not a training variant
P2-C2 · Success rate logged efficiently🟢 Success metric emitted through training extras🟢 Success metric emitted through training extras⚪ Not a training variant
P2-C3 · Success criterion defined and configured🟡 Acceptance criterion pending definition🟡 Acceptance criterion pending definition⚪ Not a training variant
P2-C4 · Success met within iteration/time budget🟡 Acceptance run deferred until criterion is defined🟡 Acceptance run deferred until criterion is defined⚪ Not a training variant
P2-C5 · Multi-seed sanity check🟡 Multi-seed acceptance deferred with criterion🟡 Multi-seed acceptance deferred with criterion⚪ Not a training variant
P2-C6 · Policy replay meets success criterion🟡 Policy acceptance replay deferred with criterion🟡 Policy acceptance replay deferred with criterion⚪ Not a qualifying-checkpoint row
P2-C7 · Two-reviewer behavior validation🟡 Behavior acceptance review deferred with criterion🟡 Behavior acceptance review deferred with criterion⚪ Validated at task-family policy level
P2-C8 · Reward and episode length tracked (non-gating)🟢 Exact M2 events contain reward and episode length🟢 Exact M2 events contain reward and episode length⚪ Not a training variant
QualityP2-Q1 · Task-code hygiene🟢 Retained diff hygiene review passes🟢 Retained diff hygiene review passes🟢 Retained diff hygiene review passes
P2-Q2 · Independent package dependencies🟢 Wheel metadata covers direct imports, including Franka-soft deformable helpers🟢 Wheel metadata covers direct imports, including Franka-soft deformable helpers🟢 Wheel metadata covers direct imports, including Franka-soft deformable helpers
P2-Q3 · Variants and selections documented🟢 Environment overview lists IDs and selections🟢 Environment overview lists IDs and selections🟢 Environment overview lists IDs and selections
P2-Q4 · Open-issue audit🟢 Audit complete; only actually resolved issues close🟢 Audit complete; only actually resolved issues close🟢 Audit complete; only actually resolved issues close
P2-Q5 · Automated contract checks🟢 Registration/parity/preset/asset/package checks pass🟢 Registration/parity/preset/asset/package checks pass🟢 Registration/parity/preset/asset/package checks pass
DeliveryP2-D1 · Current-asset qualifying checkpoint🟡 Candidate checkpoints archived for all backends (SHA256); qualification deferred🟡 Candidate checkpoints archived for all backends (SHA256); qualification deferred⚪ Not a training variant
P2-D2 · Replay/video and publication artifacts🟡 Playback traces and bundles re-prepared on restored assets; acceptance deferred🟡 Playback traces and bundles re-prepared on restored assets; acceptance deferred⚪ Not a qualifying-checkpoint row
P2-D3 · Runtime performance recorded🟢 Steps/s, duration, and peak memory re-recorded on restored assets🟢 Steps/s, duration, and peak memory re-recorded on restored assets⚪ No training or rendering workload
P2-D4 · Dashboard metrics recorded🟢 Event, throughput, and peak-memory bundle re-recorded on restored assets🟢 Event, throughput, and peak-memory bundle re-recorded on restored assets⚪ Playback-only
P2-D5 · Recurring benchmark schema compatible🟢 Program-wide M2 result schema validated🟢 Program-wide M2 result schema validated🟢 Program-wide M2 result schema validated
Shadow state cube reorientation — 🟢 36 · 🟡 14 · 86%
Registered variant 🟢 🟡 🔴 Progress
Isaac-Reorient-Cube-Shadow-Direct 18 7 0 5 86%
Isaac-Reorient-Cube-Shadow 18 7 0 5 86%
AreaID · RequirementDirectManager
StructureP2-S1 · Assigned registration accounted for🟢 Registered and assigned🟢 Registered and assigned
P2-S2 · Manager-based training environment🟢 Manager train exists for family🟢 Manager train exists for family
P2-S3 · Direct/Manager contracts comparable🟢 Focused parity checks pass🟢 Focused parity checks pass
P2-S4 · Rendering identified by Camera name⚪ Non-rendering⚪ Non-rendering
P2-S5 · Play/Eval implementation consolidated⚪ Not a Play/Eval variant⚪ Not a Play/Eval variant
Assets & backendsP2-B1 · New asset evaluated; fallback justified⚪ Dropped: new-asset adoption removed from the consolidated standard⚪ Dropped: new-asset adoption removed from the consolidated standard
P2-B2 · PhysX / Newton / OV presets resolve🟢 All three physics presets resolve🟢 All three physics presets resolve
P2-B3 · PhysX constructs and trains🟢 Exact-head PhysX train smoke and full-budget training pass🟢 Exact-head PhysX train smoke and full-budget training pass
P2-B3 · Newton constructs and trains🟢 Exact-head Newton train smoke and full-budget training pass🟢 Exact-head Newton train smoke and full-budget training pass
P2-B3 · OVPhysX constructs and trains🟢 Train smoke and corrected task-frame runtime probes pass🟢 Train smoke and corrected task-frame runtime probes pass
P2-B4 · Supported renderer matrix works⚪ Non-camera⚪ Non-camera
P2-B5 · Camera framing and ground are usable⚪ Non-camera⚪ Non-camera
SuccessP2-C1 · RSL-RL configuration provided🟢 RSL-RL config resolves🟢 RSL-RL config resolves
P2-C2 · Success rate logged efficiently🟢 Success metric emitted through training extras🟢 Success metric emitted through training extras
P2-C3 · Success criterion defined and configured🟡 Acceptance criterion pending definition🟡 Acceptance criterion pending definition
P2-C4 · Success met within iteration/time budget🟡 Acceptance run deferred until criterion is defined🟡 Acceptance run deferred until criterion is defined
P2-C5 · Multi-seed sanity check🟡 Multi-seed acceptance deferred with criterion🟡 Multi-seed acceptance deferred with criterion
P2-C6 · Policy replay meets success criterion🟡 Policy acceptance replay deferred with criterion🟡 Policy acceptance replay deferred with criterion
P2-C7 · Two-reviewer behavior validation🟡 Behavior acceptance review deferred with criterion🟡 Behavior acceptance review deferred with criterion
P2-C8 · Reward and episode length tracked (non-gating)🟢 Exact M2 events contain reward and episode length🟢 Exact M2 events contain reward and episode length
QualityP2-Q1 · Task-code hygiene🟢 Retained diff hygiene review passes🟢 Retained diff hygiene review passes
P2-Q2 · Independent package dependencies🟢 Wheel metadata covers direct imports, including Franka-soft deformable helpers🟢 Wheel metadata covers direct imports, including Franka-soft deformable helpers
P2-Q3 · Variants and selections documented🟢 Environment overview lists IDs and selections🟢 Environment overview lists IDs and selections
P2-Q4 · Open-issue audit🟢 Audit complete; only actually resolved issues close🟢 Audit complete; only actually resolved issues close
P2-Q5 · Automated contract checks🟢 Registration/parity/preset/asset/package checks pass🟢 Registration/parity/preset/asset/package checks pass
DeliveryP2-D1 · Current-asset qualifying checkpoint🟡 Candidate checkpoints archived for all backends (SHA256); qualification deferred🟡 Candidate checkpoints archived for all backends (SHA256); qualification deferred
P2-D2 · Replay/video and publication artifacts🟡 Replay/publication acceptance deferred with success criteria🟡 Replay/publication acceptance deferred with success criteria
P2-D3 · Runtime performance recorded🟢 Steps/s, duration, and peak memory re-recorded on restored assets🟢 Steps/s, duration, and peak memory re-recorded on restored assets
P2-D4 · Dashboard metrics recorded🟢 Event, throughput, and peak-memory bundle re-recorded on restored assets🟢 Event, throughput, and peak-memory bundle re-recorded on restored assets
P2-D5 · Recurring benchmark schema compatible🟢 Program-wide M2 result schema validated🟢 Program-wide M2 result schema validated
Shadow OpenAI FF cube reorientation — 🟢 36 · 🟡 14 · 86%
Registered variant 🟢 🟡 🔴 Progress
Isaac-Reorient-Cube-Shadow-OpenAI-FF-Direct 18 7 0 5 86%
Isaac-Reorient-Cube-Shadow-OpenAI-FF 18 7 0 5 86%
AreaID · RequirementDirectManager
StructureP2-S1 · Assigned registration accounted for🟢 Registered and assigned🟢 Registered and assigned
P2-S2 · Manager-based training environment🟢 Manager train exists for family🟢 Manager train exists for family
P2-S3 · Direct/Manager contracts comparable🟢 Focused parity checks pass🟢 Focused parity checks pass
P2-S4 · Rendering identified by Camera name⚪ Non-rendering⚪ Non-rendering
P2-S5 · Play/Eval implementation consolidated⚪ Not a Play/Eval variant⚪ Not a Play/Eval variant
Assets & backendsP2-B1 · New asset evaluated; fallback justified⚪ Dropped: new-asset adoption removed from the consolidated standard⚪ Dropped: new-asset adoption removed from the consolidated standard
P2-B2 · PhysX / Newton / OV presets resolve🟢 All three physics presets resolve🟢 All three physics presets resolve
P2-B3 · PhysX constructs and trains🟢 Exact-head PhysX train smoke and full-budget training pass🟢 Exact-head PhysX train smoke and full-budget training pass
P2-B3 · Newton constructs and trains🟢 Exact-head Newton train smoke and full-budget training pass🟢 Exact-head Newton train smoke and full-budget training pass
P2-B3 · OVPhysX constructs and trains🟢 Train smoke and corrected task-frame runtime probes pass🟢 Train smoke and corrected task-frame runtime probes pass
P2-B4 · Supported renderer matrix works⚪ Non-camera⚪ Non-camera
P2-B5 · Camera framing and ground are usable⚪ Non-camera⚪ Non-camera
SuccessP2-C1 · RSL-RL configuration provided🟢 RSL-RL config resolves🟢 RSL-RL config resolves
P2-C2 · Success rate logged efficiently🟢 Success metric emitted through training extras🟢 Success metric emitted through training extras
P2-C3 · Success criterion defined and configured🟡 Acceptance criterion pending definition🟡 Acceptance criterion pending definition
P2-C4 · Success met within iteration/time budget🟡 Acceptance run deferred until criterion is defined🟡 Acceptance run deferred until criterion is defined
P2-C5 · Multi-seed sanity check🟡 Multi-seed acceptance deferred with criterion🟡 Multi-seed acceptance deferred with criterion
P2-C6 · Policy replay meets success criterion🟡 Policy acceptance replay deferred with criterion🟡 Policy acceptance replay deferred with criterion
P2-C7 · Two-reviewer behavior validation🟡 Behavior acceptance review deferred with criterion🟡 Behavior acceptance review deferred with criterion
P2-C8 · Reward and episode length tracked (non-gating)🟢 Exact M2 events contain reward and episode length🟢 Exact M2 events contain reward and episode length
QualityP2-Q1 · Task-code hygiene🟢 Retained diff hygiene review passes🟢 Retained diff hygiene review passes
P2-Q2 · Independent package dependencies🟢 Wheel metadata covers direct imports, including Franka-soft deformable helpers🟢 Wheel metadata covers direct imports, including Franka-soft deformable helpers
P2-Q3 · Variants and selections documented🟢 Environment overview lists IDs and selections🟢 Environment overview lists IDs and selections
P2-Q4 · Open-issue audit🟢 Audit complete; only actually resolved issues close🟢 Audit complete; only actually resolved issues close
P2-Q5 · Automated contract checks🟢 Registration/parity/preset/asset/package checks pass🟢 Registration/parity/preset/asset/package checks pass
DeliveryP2-D1 · Current-asset qualifying checkpoint🟡 Checkpoint qualification deferred with success criteria🟡 Checkpoint qualification deferred with success criteria
P2-D2 · Replay/video and publication artifacts🟡 Replay/publication acceptance deferred with success criteria🟡 Replay/publication acceptance deferred with success criteria
P2-D3 · Runtime performance recorded🟢 Steps/s, duration, and peak memory re-recorded on restored assets🟢 Steps/s, duration, and peak memory re-recorded on restored assets
P2-D4 · Dashboard metrics recorded🟢 Event, throughput, and peak-memory bundle re-recorded on restored assets🟢 Event, throughput, and peak-memory bundle re-recorded on restored assets
P2-D5 · Recurring benchmark schema compatible🟢 Program-wide M2 result schema validated🟢 Program-wide M2 result schema validated
Shadow OpenAI LSTM cube reorientation — 🟢 36 · 🟡 14 · 86%
Registered variant 🟢 🟡 🔴 Progress
Isaac-Reorient-Cube-Shadow-OpenAI-LSTM-Direct 18 7 0 5 86%
Isaac-Reorient-Cube-Shadow-OpenAI-LSTM 18 7 0 5 86%
AreaID · RequirementDirectManager
StructureP2-S1 · Assigned registration accounted for🟢 Registered and assigned🟢 Registered and assigned
P2-S2 · Manager-based training environment🟢 Manager train exists for family🟢 Manager train exists for family
P2-S3 · Direct/Manager contracts comparable🟢 Focused parity checks pass🟢 Focused parity checks pass
P2-S4 · Rendering identified by Camera name⚪ Non-rendering⚪ Non-rendering
P2-S5 · Play/Eval implementation consolidated⚪ Not a Play/Eval variant⚪ Not a Play/Eval variant
Assets & backendsP2-B1 · New asset evaluated; fallback justified⚪ Dropped: new-asset adoption removed from the consolidated standard⚪ Dropped: new-asset adoption removed from the consolidated standard
P2-B2 · PhysX / Newton / OV presets resolve🟢 All three physics presets resolve🟢 All three physics presets resolve
P2-B3 · PhysX constructs and trains🟢 Exact-head PhysX train smoke and full-budget training pass🟢 Exact-head PhysX train smoke and full-budget training pass
P2-B3 · Newton constructs and trains🟢 Exact-head Newton train smoke and full-budget training pass🟢 Exact-head Newton train smoke and full-budget training pass
P2-B3 · OVPhysX constructs and trains🟢 Train smoke and corrected task-frame runtime probes pass🟢 Train smoke and corrected task-frame runtime probes pass
P2-B4 · Supported renderer matrix works⚪ Non-camera⚪ Non-camera
P2-B5 · Camera framing and ground are usable⚪ Non-camera⚪ Non-camera
SuccessP2-C1 · RSL-RL configuration provided🟢 RSL-RL config resolves🟢 RSL-RL config resolves
P2-C2 · Success rate logged efficiently🟢 Success metric emitted through training extras🟢 Success metric emitted through training extras
P2-C3 · Success criterion defined and configured🟡 Acceptance criterion pending definition🟡 Acceptance criterion pending definition
P2-C4 · Success met within iteration/time budget🟡 Acceptance run deferred until criterion is defined🟡 Acceptance run deferred until criterion is defined
P2-C5 · Multi-seed sanity check🟡 Multi-seed acceptance deferred with criterion🟡 Multi-seed acceptance deferred with criterion
P2-C6 · Policy replay meets success criterion🟡 Policy acceptance replay deferred with criterion🟡 Policy acceptance replay deferred with criterion
P2-C7 · Two-reviewer behavior validation🟡 Behavior acceptance review deferred with criterion🟡 Behavior acceptance review deferred with criterion
P2-C8 · Reward and episode length tracked (non-gating)🟢 Exact M2 events contain reward and episode length🟢 Exact M2 events contain reward and episode length
QualityP2-Q1 · Task-code hygiene🟢 Retained diff hygiene review passes🟢 Retained diff hygiene review passes
P2-Q2 · Independent package dependencies🟢 Wheel metadata covers direct imports, including Franka-soft deformable helpers🟢 Wheel metadata covers direct imports, including Franka-soft deformable helpers
P2-Q3 · Variants and selections documented🟢 Environment overview lists IDs and selections🟢 Environment overview lists IDs and selections
P2-Q4 · Open-issue audit🟢 Audit complete; only actually resolved issues close🟢 Audit complete; only actually resolved issues close
P2-Q5 · Automated contract checks🟢 Registration/parity/preset/asset/package checks pass🟢 Registration/parity/preset/asset/package checks pass
DeliveryP2-D1 · Current-asset qualifying checkpoint🟡 Checkpoint qualification deferred with success criteria🟡 Checkpoint qualification deferred with success criteria
P2-D2 · Replay/video and publication artifacts🟡 Replay/publication acceptance deferred with success criteria🟡 Replay/publication acceptance deferred with success criteria
P2-D3 · Runtime performance recorded🟢 Steps/s, duration, and peak memory re-recorded on restored assets🟢 Steps/s, duration, and peak memory re-recorded on restored assets
P2-D4 · Dashboard metrics recorded🟢 Event, throughput, and peak-memory bundle re-recorded on restored assets🟢 Event, throughput, and peak-memory bundle re-recorded on restored assets
P2-D5 · Recurring benchmark schema compatible🟢 Program-wide M2 result schema validated🟢 Program-wide M2 result schema validated
Shadow Camera cube reorientation — 🟢 100 · 🟡 28 · 89%
Registered variant 🟢 🟡 🔴 Progress
Isaac-Reorient-Cube-Shadow-Camera-Direct 19 9 0 2 84%
Isaac-Reorient-Cube-Shadow-Camera 19 9 0 2 84%
Isaac-Reorient-Cube-Shadow-Camera-Direct-Play 15 3 0 12 92%
Isaac-Reorient-Cube-Shadow-Camera-Play 15 3 0 12 92%
Isaac-Reorient-Cube-Shadow-Camera-Benchmark-Direct 16 2 0 12 94%
Isaac-Reorient-Cube-Shadow-Camera-Benchmark 16 2 0 12 94%
AreaID · RequirementDirect trainManager trainDirect PlayManager PlayDirect benchmarkManager benchmark
StructureP2-S1 · Assigned registration accounted for🟢 Registered and assigned🟢 Registered and assigned🟢 Registered and assigned🟢 Registered and assigned🟢 Registered and assigned🟢 Registered and assigned
P2-S2 · Manager-based training environment🟢 Manager train exists for family🟢 Manager train exists for family🟢 Manager train exists for family🟢 Manager train exists for family🟢 Manager train exists for family🟢 Manager train exists for family
P2-S3 · Direct/Manager contracts comparable🟢 Focused parity checks pass🟢 Focused parity checks pass🟢 Focused parity checks pass🟢 Focused parity checks pass🟢 Focused parity checks pass🟢 Focused parity checks pass
P2-S4 · Rendering identified by Camera name🟢 Camera in public ID🟢 Camera in public ID🟢 Camera in public ID🟢 Camera in public ID🟢 Camera in public ID🟢 Camera in public ID
P2-S5 · Play/Eval implementation consolidated⚪ Not a Play/Eval variant⚪ Not a Play/Eval variant🟡 Public Play ID retained; generic consolidation pending🟡 Public Play ID retained; generic consolidation pending⚪ Not a Play/Eval variant⚪ Not a Play/Eval variant
Assets & backendsP2-B1 · New asset evaluated; fallback justified⚪ Dropped: new-asset adoption removed from the consolidated standard⚪ Dropped: new-asset adoption removed from the consolidated standard⚪ Dropped: new-asset adoption removed from the consolidated standard⚪ Dropped: new-asset adoption removed from the consolidated standard⚪ Dropped: new-asset adoption removed from the consolidated standard⚪ Dropped: new-asset adoption removed from the consolidated standard
P2-B2 · PhysX / Newton / OV presets resolve🟢 All three physics presets resolve🟢 All three physics presets resolve🟢 All three physics presets resolve🟢 All three physics presets resolve🟢 All three physics presets resolve🟢 All three physics presets resolve
P2-B3 · PhysX constructs and trains🟢 Exact-head PhysX train smoke and full-budget training pass🟢 Exact-head PhysX train smoke and full-budget training pass🟢 PhysX replay and evaluation video exported from restored-asset checkpoint🟢 PhysX replay and evaluation video exported from restored-asset checkpoint🟢 PhysX benchmark bundle recorded on restored assets🟢 PhysX benchmark bundle recorded on restored assets
P2-B3 · Newton constructs and trains🟢 Exact-head Newton train smoke and full-budget training pass🟢 Exact-head Newton train smoke and full-budget training pass🟢 Newton replay and evaluation video exported from restored-asset checkpoint🟢 Newton replay and evaluation video exported from restored-asset checkpoint🟢 Newton benchmark bundle recorded on restored assets🟢 Newton benchmark bundle recorded on restored assets
P2-B3 · OVPhysX constructs and trains🟢 Train smoke and corrected task-frame runtime probes pass🟢 Train smoke and corrected task-frame runtime probes pass🟢 Exact OVPhysX replay and video pass (path unchanged by asset restoration)🟢 Exact OVPhysX replay and video pass (path unchanged by asset restoration)🟢 OVPhysX benchmark bundle passes (path unchanged by asset restoration)🟢 OVPhysX benchmark bundle passes (path unchanged by asset restoration)
P2-B4 · Supported renderer matrix works🟡 7/8 pairs exit clean; Newton+OVRTX teardown gap🟡 7/8 pairs exit clean; Newton+OVRTX teardown gap🟡 7/8 pairs exit clean; Newton+OVRTX teardown gap🟡 7/8 pairs exit clean; Newton+OVRTX teardown gap🟡 7/8 pairs exit clean; Newton+OVRTX teardown gap🟡 7/8 pairs exit clean; Newton+OVRTX teardown gap
P2-B5 · Camera framing and ground are usable🟡 8/8 frame routes captured; human framing gate pending🟡 8/8 frame routes captured; human framing gate pending🟡 8/8 frame routes captured; human framing gate pending🟡 8/8 frame routes captured; human framing gate pending🟡 8/8 frame routes captured; human framing gate pending🟡 8/8 frame routes captured; human framing gate pending
SuccessP2-C1 · RSL-RL configuration provided🟢 RSL-RL config resolves🟢 RSL-RL config resolves⚪ Not a training variant⚪ Not a training variant⚪ Not a training variant⚪ Not a training variant
P2-C2 · Success rate logged efficiently🟢 Success metric emitted through training extras🟢 Success metric emitted through training extras⚪ Not a training variant⚪ Not a training variant⚪ Not a training variant⚪ Not a training variant
P2-C3 · Success criterion defined and configured🟡 Acceptance criterion pending definition🟡 Acceptance criterion pending definition⚪ Not a training variant⚪ Not a training variant⚪ Not a training variant⚪ Not a training variant
P2-C4 · Success met within iteration/time budget🟡 Acceptance run deferred until criterion is defined🟡 Acceptance run deferred until criterion is defined⚪ Not a training variant⚪ Not a training variant⚪ Not a training variant⚪ Not a training variant
P2-C5 · Multi-seed sanity check🟡 Multi-seed acceptance deferred with criterion🟡 Multi-seed acceptance deferred with criterion⚪ Not a training variant⚪ Not a training variant⚪ Not a training variant⚪ Not a training variant
P2-C6 · Policy replay meets success criterion🟡 Policy acceptance replay deferred with criterion🟡 Policy acceptance replay deferred with criterion⚪ Not a qualifying-checkpoint row⚪ Not a qualifying-checkpoint row⚪ Not a qualifying-checkpoint row⚪ Not a qualifying-checkpoint row
P2-C7 · Two-reviewer behavior validation🟡 Behavior acceptance review deferred with criterion🟡 Behavior acceptance review deferred with criterion⚪ Validated at task-family policy level⚪ Validated at task-family policy level⚪ Validated at task-family policy level⚪ Validated at task-family policy level
P2-C8 · Reward and episode length tracked (non-gating)🟢 Exact M2 events contain reward and episode length🟢 Exact M2 events contain reward and episode length⚪ Not a training variant⚪ Not a training variant⚪ Not a training variant⚪ Not a training variant
QualityP2-Q1 · Task-code hygiene🟢 Retained diff hygiene review passes🟢 Retained diff hygiene review passes🟢 Retained diff hygiene review passes🟢 Retained diff hygiene review passes🟢 Retained diff hygiene review passes🟢 Retained diff hygiene review passes
P2-Q2 · Independent package dependencies🟢 Wheel metadata covers direct imports, including Franka-soft deformable helpers🟢 Wheel metadata covers direct imports, including Franka-soft deformable helpers🟢 Wheel metadata covers direct imports, including Franka-soft deformable helpers🟢 Wheel metadata covers direct imports, including Franka-soft deformable helpers🟢 Wheel metadata covers direct imports, including Franka-soft deformable helpers🟢 Wheel metadata covers direct imports, including Franka-soft deformable helpers
P2-Q3 · Variants and selections documented🟢 Environment overview lists IDs and selections🟢 Environment overview lists IDs and selections🟢 Environment overview lists IDs and selections🟢 Environment overview lists IDs and selections🟢 Environment overview lists IDs and selections🟢 Environment overview lists IDs and selections
P2-Q4 · Open-issue audit🟢 Audit complete; only actually resolved issues close🟢 Audit complete; only actually resolved issues close🟢 Audit complete; only actually resolved issues close🟢 Audit complete; only actually resolved issues close🟢 Audit complete; only actually resolved issues close🟢 Audit complete; only actually resolved issues close
P2-Q5 · Automated contract checks🟢 Registration/parity/preset/asset/package checks pass🟢 Registration/parity/preset/asset/package checks pass🟢 Registration/parity/preset/asset/package checks pass🟢 Registration/parity/preset/asset/package checks pass🟢 Registration/parity/preset/asset/package checks pass🟢 Registration/parity/preset/asset/package checks pass
DeliveryP2-D1 · Current-asset qualifying checkpoint🟡 Checkpoint qualification deferred with success criteria🟡 Checkpoint qualification deferred with success criteria⚪ Not a training variant⚪ Not a training variant⚪ Not a training variant⚪ Not a training variant
P2-D2 · Replay/video and publication artifacts🟡 Replay/publication acceptance deferred with success criteria🟡 Replay/publication acceptance deferred with success criteria⚪ Not a qualifying-checkpoint row⚪ Not a qualifying-checkpoint row⚪ Not a qualifying-checkpoint row⚪ Not a qualifying-checkpoint row
P2-D3 · Runtime performance recorded🟢 Steps/s, duration, and peak memory re-recorded on restored assets🟢 Steps/s, duration, and peak memory re-recorded on restored assets🟢 Steps/s, duration, and peak memory re-recorded on restored assets🟢 Steps/s, duration, and peak memory re-recorded on restored assets🟢 Steps/s, duration, and peak memory re-recorded on restored assets🟢 Steps/s, duration, and peak memory re-recorded on restored assets
P2-D4 · Dashboard metrics recorded🟢 Event, throughput, and peak-memory bundle re-recorded on restored assets🟢 Event, throughput, and peak-memory bundle re-recorded on restored assets⚪ Playback-only⚪ Playback-only🟢 Two-step benchmark event and runtime bundle recorded🟢 Two-step benchmark event and runtime bundle recorded
P2-D5 · Recurring benchmark schema compatible🟢 Program-wide M2 result schema validated🟢 Program-wide M2 result schema validated🟢 Program-wide M2 result schema validated🟢 Program-wide M2 result schema validated🟢 Program-wide M2 result schema validated🟢 Program-wide M2 result schema validated
Shadow handover — 🟢 34 · 🟡 16 · 84%
Registered variant 🟢 🟡 🔴 Progress
Isaac-Shadow-Handover-Direct 17 8 0 5 84%
Isaac-Shadow-Handover 17 8 0 5 84%
AreaID · RequirementDirectManager
StructureP2-S1 · Assigned registration accounted for🟢 Registered and assigned🟢 Registered and assigned
P2-S2 · Manager-based training environment🟢 Manager train exists for family🟢 Manager train exists for family
P2-S3 · Direct/Manager contracts comparable🟢 Focused parity checks pass🟢 Focused parity checks pass
P2-S4 · Rendering identified by Camera name⚪ Non-rendering⚪ Non-rendering
P2-S5 · Play/Eval implementation consolidated⚪ Not a Play/Eval variant⚪ Not a Play/Eval variant
Assets & backendsP2-B1 · New asset evaluated; fallback justified⚪ Dropped: new-asset adoption removed from the consolidated standard⚪ Dropped: new-asset adoption removed from the consolidated standard
P2-B2 · PhysX / Newton / OV presets resolve🟢 All three physics presets resolve🟢 All three physics presets resolve
P2-B3 · PhysX constructs and trains🟢 Exact-head PhysX train smoke and full-budget training pass🟢 Exact-head PhysX train smoke and full-budget training pass
P2-B3 · Newton constructs and trains🟡 Constructs and trains mechanically; convergence blocked by upstream asset joint renumbering🟡 Constructs and trains mechanically; convergence blocked by upstream asset joint renumbering
P2-B3 · OVPhysX constructs and trains🟢 Train smoke and corrected task-frame runtime probes pass🟢 Train smoke and corrected task-frame runtime probes pass
P2-B4 · Supported renderer matrix works⚪ Non-camera⚪ Non-camera
P2-B5 · Camera framing and ground are usable⚪ Non-camera⚪ Non-camera
SuccessP2-C1 · RSL-RL configuration provided🟢 RSL-RL config resolves🟢 RSL-RL config resolves
P2-C2 · Success rate logged efficiently🟢 Success metric emitted through training extras🟢 Success metric emitted through training extras
P2-C3 · Success criterion defined and configured🟡 Acceptance criterion pending definition🟡 Acceptance criterion pending definition
P2-C4 · Success met within iteration/time budget🟡 Acceptance run deferred until criterion is defined🟡 Acceptance run deferred until criterion is defined
P2-C5 · Multi-seed sanity check🟡 Multi-seed acceptance deferred with criterion🟡 Multi-seed acceptance deferred with criterion
P2-C6 · Policy replay meets success criterion🟡 Policy acceptance replay deferred with criterion🟡 Policy acceptance replay deferred with criterion
P2-C7 · Two-reviewer behavior validation🟡 Behavior acceptance review deferred with criterion🟡 Behavior acceptance review deferred with criterion
P2-C8 · Reward and episode length tracked (non-gating)🟢 Exact M2 events contain reward and episode length🟢 Exact M2 events contain reward and episode length
QualityP2-Q1 · Task-code hygiene🟢 Retained diff hygiene review passes🟢 Retained diff hygiene review passes
P2-Q2 · Independent package dependencies🟢 Wheel metadata covers direct imports, including Franka-soft deformable helpers🟢 Wheel metadata covers direct imports, including Franka-soft deformable helpers
P2-Q3 · Variants and selections documented🟢 Environment overview lists IDs and selections🟢 Environment overview lists IDs and selections
P2-Q4 · Open-issue audit🟢 Audit complete; only actually resolved issues close🟢 Audit complete; only actually resolved issues close
P2-Q5 · Automated contract checks🟢 Registration/parity/preset/asset/package checks pass🟢 Registration/parity/preset/asset/package checks pass
DeliveryP2-D1 · Current-asset qualifying checkpoint🟡 Checkpoint qualification deferred with success criteria🟡 Checkpoint qualification deferred with success criteria
P2-D2 · Replay/video and publication artifacts🟡 Replay/publication acceptance deferred with success criteria🟡 Replay/publication acceptance deferred with success criteria
P2-D3 · Runtime performance recorded🟢 Steps/s, duration, and peak memory re-recorded on restored assets🟢 Steps/s, duration, and peak memory re-recorded on restored assets
P2-D4 · Dashboard metrics recorded🟢 Event, throughput, and peak-memory bundle re-recorded on restored assets🟢 Event, throughput, and peak-memory bundle re-recorded on restored assets
P2-D5 · Recurring benchmark schema compatible🟢 Program-wide M2 result schema validated🟢 Program-wide M2 result schema validated

@github-actions github-actions Bot added isaac-lab Related to Isaac Lab team asset New asset feature or request labels Jul 2, 2026
@hujc7
hujc7 force-pushed the jichuanh/dexterous-env-convergence-codex branch from 23c2ef8 to 99025bd Compare July 2, 2026 10:08
@hujc7 hujc7 changed the title [Newton] Stabilize dexterous handover and camera training [Newton] Stabilize dexterous hand tasks across backends Jul 2, 2026
@hujc7
hujc7 force-pushed the jichuanh/dexterous-env-convergence-codex branch from 507b309 to 43ede30 Compare July 2, 2026 22:41
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Jul 2, 2026
@hujc7 hujc7 changed the title [Newton] Stabilize dexterous hand tasks across backends [DO-NOT-MERGE][Task Clean-up] Dexterous: lumped validation branch (split into Parts 1-11) Jul 8, 2026
@hujc7
hujc7 marked this pull request as ready for review July 11, 2026 01:19
@greptile-apps

greptile-apps Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds the dexterous task convergence work across direct and manager environments. The main changes are:

  • Manager versions for Allegro, Shadow reorientation, camera, OpenAI, and handover tasks.
  • Warp-backed reward, observation, action, and success-metric paths.
  • OVPhysX runtime and articulation updates for new backend presets.
  • RSL-RL wrapper and MARL utility updates for shared observation buffers.
  • Regenerated docs, benchmark configs, tests, and camera goldens.

Confidence Score: 4/5

The handover manager reset observation path needs a fix before merging.

  • A newly reset handover manager episode can receive the terminal action from the previous episode.
  • The reward, reorientation, MARL wrapper, and OVPhysX changes reviewed here otherwise line up with their intended direct or backend contracts.

source/isaaclab_tasks/isaaclab_tasks/core/handover/mdp/observations.py

Important Files Changed

Filename Overview
source/isaaclab_tasks/isaaclab_tasks/core/handover/mdp/observations.py Adds manager handover observation terms; the reset action masking can expose previous-episode actions on the first post-reset observation.
source/isaaclab_tasks/isaaclab_tasks/core/handover/mdp/rewards.py Adds Warp-backed handover reward and success metrics with scaling intended to match the Direct summed reward path.
source/isaaclab_tasks/isaaclab_tasks/core/reorient/mdp/rewards.py Moves reorientation reward and success accounting into Warp kernels and caller-owned buffers.
source/isaaclab_tasks/isaaclab_tasks/core/reorient/reorient_direct_env.py Rewrites direct reorientation actuation, observations, rewards, and reset bookkeeping to use Warp-backed state.
source/isaaclab_ovphysx/isaaclab_ovphysx/physics/ovphysx_manager.py Adds compatibility handling for current and legacy OVPhysX runtime APIs.
source/isaaclab_ovphysx/isaaclab_ovphysx/assets/articulation/articulation.py Updates actuator joint-id handling and actuator write paths for OVPhysX articulations.
source/isaaclab/isaaclab/envs/utils/marl.py Adds observation-buffer and episode-length forwarding for MARL conversion wrappers.

Reviews (1): Last reviewed commit: "Calibrate camera benchmark gates from fu..." | Re-trigger Greptile

episode_length_buf = getattr(env, "episode_length_buf", None)
if reset_actions is None or action_name not in reset_actions or episode_length_buf is None:
return raw_action
return torch.where((episode_length_buf == 0).unsqueeze(-1), reset_actions[action_name], raw_action)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Terminal Action Leaks Forward

When a manager handover env autoresets, reset_handover_state stores the terminal raw action and episode_length_buf stays 0 for the first observation of the new episode. This branch then reports the previous episode’s action for the newly reset state, while the Direct handover path reports the current step action, so RSL-RL can train on stale actions at episode boundaries.

hujc7 added 2 commits July 16, 2026 17:44
Add a behavioral Metrics/success_rate signal (goal-reach streaks per
episode) and threshold-independent episode orientation-error diagnostics
to the Direct reorientation environments, with the shared helpers in
isaaclab_tasks.core.utils and torch math tests. The task logic is
torch-first per the mainline convention; success gates task health while
reward stays diagnostic. Also fix hand resets that could initialize
joints below their lower position limits.
Add the RSL-RL runner configuration for the Shadow handover Direct task
and success-rate metrics on the torch-first path, fix handover
construction on Newton (renamed distal joints), and land the camera
Direct renderer presets with configuration validation. RSL-RL
observations now read from the public environment-owned obs_buf on all
Direct env bases (reset stores the buffer like step), replacing the
adapter-side private hook.
hujc7 added 8 commits July 22, 2026 09:25
The lower layer's registration revert merged forward and un-renamed
the camera entry points here, where the camera modules do carry the
renamed filenames. Pin the registration block back to the final
state.
…eanup-dex-part11

# Conflicts:
#	source/isaaclab_tasks/isaaclab_tasks/core/reorient/config/shadow_hand/__init__.py
hujc7 added a commit that referenced this pull request Jul 22, 2026
…isualizer teardown (#6411)

## Review Map

- **Exact changes**: a stacked PR's page shows the cumulative diff of
its dependency chain; the link pins the commit range that is the PR's
own contribution.
- Links pin specific SHAs and can go stale after a branch update — the
table on #6324 is refreshed first.

| PR | Status | Depends on | Exact changes |
|---|---|---|---|
| #6410 [Docs] Environment overview regen |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6410?label=)
| — | — |
| 📌 #6411 Part 1/8: Newton cloner/cubric/visualizer fixes (this PR) |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6411?label=)
| — | — |
| #6412 Part 2/8: OVPhysX articulation + manager runtime |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6412?label=)
| — | — |
| #6413 Part 3/8: Reorient Direct, torch |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6413?label=)
| — | — |
| #6414 Part 4/8: MARL-to-single-agent fix + handover/camera Direct |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6414?label=)
| #6413 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6414/changes/6e8a63e4e028b2d43676ea30c446b9dc9068c7b5..5cb00e7cb5cc813b202521272e043007cd255194)
|
| #6418 Part 5/8: Reorient manager counterparts |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6418?label=)
| #6413 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6418/changes/79f87501ac4c81de93a71dab00dc443da62113aa..e7c9a9a3fae3a7972b0c5165ae683abffb7d0e0f)
|
| #6421 Part 6/8: Handover + camera manager counterparts |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6421?label=)
| #6413, #6414, #6418 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6421/changes/01c9f4d8c5c35a5688b2a5bb90209e16b8f81b99..835a5815ec49b11aada1d20a76c177054505e6e7)
|
| #6415 Part 7/8: Benchmark success-rate utilities + docs |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6415?label=)
| #6413, #6414, #6418, #6421 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6415/changes/e1abb6b1790ccc57af42551eebccf743633f1f13..d6348539aa8032d9668c20a8fea462c5d88d3af9)
|
| #6582 Part 8/8: Warp variants → experimental (draft; merges last) |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6582?label=)
| #6413 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6582/changes/21dbb1769c4e30c8e9e5b0f563c2dae24c230349..83e1587cadd9712a60615ed2a3cb2d177c2ac24d)
|
| #6324 [DO-NOT-MERGE] Lumped validation reference |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6324?label=)
| ALL | — |

## Summary

- Fixes Newton cloner label rows, the cubric IAdapter version audit
(exact-match fallback to the CPU hierarchy path), and visualizer
teardown.
- Retains an in-tree `ignore_paths` workaround for custom-frequency USD
traversal; it becomes redundant once the Newton pin advance
(#6584) merges — this PR then
only needs a rebase.

## Stacking

- Independent; based on `develop`.

## Review history

- Approved. The Newton pin + MuJoCo overrides were split out to
#6584 via revert commits
(2026-07-17) so this PR's CI runs against develop's pins.
hujc7 added 17 commits July 23, 2026 13:26
Applies the P3-owned share of the lump review-response commits:

- Default the dexterous tasks to newton_mjwarp (from 3f9ce324).
- Drop solver defaults matching the backend + inline the scene preset
  via preset() instead of a wrapper class (from 29e189bf).
- Simplify the Direct reorientation reward computation (from bfb73735).
- Source the actuated-joint and fingertip body-name lists from the
  robot assets instead of the config module (from 34676102).

Deferred to their owning layers: the core/utils relocation (S17) lands
at the manager layer where reorient/mdp/events.py is introduced, and the
handover default/preset changes land with the handover part.
P4-owned share of the lump review-response commits:

- Default the handover task to newton_mjwarp and add the ovphysx object
  preset alias (from 3f9ce324, 29e189bf).
- Source the handover joint/body name lists from the robot assets
  (from 34676102).

S17 (core/utils relocation) defers to the manager layer where
reorient/mdp/events.py is introduced.
Rebuilt from the reviewed lump so the manager counterparts land with
the review rounds folded in: inline section values per file (drift
guarded by the value-parity test incl. sim), identity from the common
modules, the OpenAI variant in its own module, and no sim mixins.

Lump review commits folded: 2c22af0, 792e400, 42675b6,
1f05f85, c6140f9, 173e9dc, 2727616, 1732493,
3659a33, ee272c9, cdaadac.
P5-owned share of the lump review-response commits:

- Relocate the shared reset/rotation helpers into reorient/mdp/events.py
  and EpisodeErrorRecorder into reorient/mdp/rewards.py, removing the
  core/utils.py module (S17, from 428297c2).
- Default the reorient manager cfgs to newton_mjwarp and source the
  hand name lists from the robot assets (from 3f9ce324, 34676102).
- Repoint the Direct env to the relocated helpers.
Rebuilds P6 on the updated P4+P5, folding the P6-owned share:

- Repoint handover to the relocated reorient.mdp helpers after the
  core/utils.py removal (S17 handover side, from 428297c2).
- Inherits N2/preset/reward/names fixes via the rebuilt P4 and P5.
…6415-fold

# Conflicts:
#	source/isaaclab_tasks/isaaclab_tasks/core/handover/handover_manager_env_cfg.py
#	source/isaaclab_tasks/isaaclab_tasks/core/handover/mdp/events.py
#	source/isaaclab_tasks/isaaclab_tasks/core/handover/mdp/rewards.py
#	source/isaaclab_tasks/isaaclab_tasks/core/reorient/config/allegro_hand/allegro_hand_manager_env_cfg.py
#	source/isaaclab_tasks/isaaclab_tasks/core/reorient/config/shadow_hand/shadow_hand_camera_manager_env_cfg.py
#	source/isaaclab_tasks/isaaclab_tasks/core/reorient/config/shadow_hand/shadow_hand_manager_env_cfg.py
#	source/isaaclab_tasks/isaaclab_tasks/core/reorient/config/shadow_hand/shadow_hand_openai_manager_env_cfg.py
#	source/isaaclab_tasks/isaaclab_tasks/core/reorient/mdp/events.py
#	source/isaaclab_tasks/isaaclab_tasks/core/reorient/mdp/rewards.py
Move the warp implementation of the Shadow reorientation and handover
Direct environments (kernels, environments, kernel behavior tests) from
the mainline task package to isaaclab_tasks_experimental, following the
torch-first mainline convention. The variants register under
-Direct-Warp-v0 IDs and reuse the stable configurations and agents. The
Allegro warp variant keeps its existing legacy registration.
Mirror the mainline rename (reorient_reward) in the experimental warp
package: the launcher is family-scoped, not paradigm-scoped, and the
package already lives under direct/. Pure rename; no behavioral
change.
Repoints the shadow-hand env.rst source links to the renamed _direct_
modules and lists ovphysx on the OpenAI-FF illustrated row (C1), completing
the dexterous environment docs in the series.
The environment overview updates for the dexterous series belong in the
dedicated docs PR (isaac-sim#6410), not the benchmark part. Reverts env.rst to the
base version here.
Documents the dexterous reorientation and handover tasks' newton/ovphysx
physics backends in the comprehensive table, and repoints the shadow-hand
source links to the renamed _direct_ config modules. Stacked on Part 6 so
every referenced task/module exists; inserted as the series' docs update.
@hujc7
hujc7 force-pushed the jichuanh/dexterous-env-convergence-codex branch from 44fbfa5 to 1c2e62f Compare July 23, 2026 22:20
The vision env renders through the Isaac RTX tiled camera, whose render
products require the Fabric cloning path. The Newton backend disables
Fabric cloning, so under Newton the rgb annotator has no render products
at num_envs > 1 and the default RGB/depth/semantic render fails.

Override the camera env's backend PresetCfgs to default to PhysX in
__post_init__; Newton stays selectable via physics=newton_mjwarp.
hujc7 added a commit that referenced this pull request Jul 24, 2026
…eorientation Direct tasks (#6413)

## Review Map

- **Exact changes**: a stacked PR's page shows the cumulative diff of
its dependency chain; the link pins the commit range that is the PR's
own contribution.
- Links pin specific SHAs and can go stale after a branch update — the
table on #6324 is refreshed first.

| PR | Status | Depends on | Exact changes |
|---|---|---|---|
| #6411 Part 1/9: Newton cloner/cubric/visualizer fixes |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6411?label=)
| — | merged |
| #6412 Part 2/9: OVPhysX articulation + manager runtime |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6412?label=)
| — | merged |
| 📌 #6413 Part 3/9: Reorient Direct, torch (this PR) |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6413?label=)
| — |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6413/changes/f4895f0f9ee..d29afc75e71)
|
| #6414 Part 4/9: MARL-to-single-agent fix + handover/camera Direct |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6414?label=)
| #6413 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6414/changes/d29afc75e71..b10a84948f8)
|
| #6418 Part 5/9: Reorient manager counterparts |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6418?label=)
| #6413 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6418/changes/d29afc75e71..707d37f8f99)
|
| #6421 Part 6/9: Handover + camera manager counterparts |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6421?label=)
| #6413, #6414, #6418 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6421/changes/707d37f8f99..b50ac8906fc)
|
| #6410 Part 7/9: Environment overview docs |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6410?label=)
| #6421 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6410/changes/b50ac8906fc..af259c0778d)
|
| #6415 Part 8/9: Benchmark success-rate utilities |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6415?label=)
| #6421 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6415/changes/b50ac8906fc..c7f2f019d8b)
|
| #6582 Part 9/9: Warp variants → experimental (draft; merges last) |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6582?label=)
| #6413 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6582/changes/d29afc75e71..167c28578b3)
|
| #6324 [DO-NOT-MERGE] Lumped validation reference |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6324?label=)
| ALL | — |

## Summary

- Adds a behavioral `Metrics/success_rate` signal (goal-reach streaks
per episode) and threshold-independent episode orientation-error
diagnostics to the Direct reorientation environments; success gates task
health, reward stays diagnostic.
- Task logic is **torch-first** per the mainline convention (plain torch
buffers; `.torch` accessors only at the core-lib boundary). This
supersedes the earlier warp-first revision of this PR; the warp
implementation moved to `isaaclab_tasks_experimental`
(#6582).
- Lands shared helpers in `isaaclab_tasks.core.utils`
(`EpisodeErrorRecorder`, `sample_joint_positions_within_limits`) with
torch math tests; fixes hand resets below lower joint limits.

## Stacking

- First stack PR; based on `develop`.

## Validation

- Kit-free torch math + utils tests green; Direct state training on
Newton: success rate 0.85–1.00, reward ≈3600–3900 at 2000 iterations,
frame-verified policy video. Full integrated evidence: #6324.

## Review history

- The earlier warp-era review rounds on this PR are superseded by the
torch restack (2026-07-16); the warp implementation and its
review-polished kernels live on in
#6582.
hujc7 added a commit that referenced this pull request Jul 29, 2026
… and enable handover Direct RSL-RL (#6414)

## Review Map

- **Exact changes**: a stacked PR's page shows the cumulative diff of
its dependency chain; the link pins the commit range that is the PR's
own contribution. This PR (#6414) is now rebased directly onto
`develop`, so its own **Files changed** tab is its contribution.
- Links pin specific SHAs and can go stale after a branch update — the
table on #6324 is refreshed first.

| PR | Status | Depends on | Exact changes |
|---|---|---|---|
| #6411 Part 1/9: Newton cloner/cubric/visualizer fixes |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6411?label=)
| — | merged |
| #6412 Part 2/9: OVPhysX articulation + manager runtime |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6412?label=)
| — | merged |
| #6413 Part 3/9: Reorient Direct, torch |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6413?label=)
| — | merged |
| 📌 #6414 Part 4/9: MARL-to-single-agent fix + handover/camera Direct
(this PR) |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6414?label=)
| — (on develop) | Files changed tab |
| #6418 Part 5/9: Reorient manager counterparts |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6418?label=)
| #6413 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6418/changes/d29afc75e71..707d37f8f99)
|
| #6421 Part 6/9: Handover + camera manager counterparts |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6421?label=)
| #6414, #6418 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6421/changes/707d37f8f99..b50ac8906fc)
|
| #6410 Part 7/9: Environment overview docs |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6410?label=)
| #6421 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6410/changes/b50ac8906fc..af259c0778d)
|
| #6415 Part 8/9: Benchmark success-rate utilities |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6415?label=)
| #6421 |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6415/changes/b50ac8906fc..c7f2f019d8b)
|
| #6582 Part 9/9: Warp variants → experimental (draft; merges last) |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6582?label=)
| — |
[changes](https://github.com/isaac-sim/IsaacLab/pull/6582/changes/d29afc75e71..167c28578b3)
|
| #6324 [DO-NOT-MERGE] Lumped validation reference |
![](https://img.shields.io/github/pulls/detail/state/isaac-sim/IsaacLab/6324?label=)
| ALL | — |

## Summary

Converts the two-hand Shadow Hand **handover** task from multi-agent
(MARL) to **single-agent** (enabling RSL-RL training), fixes the
underlying MARL→single-agent observation bridge, and consolidates the
shared Shadow Hand definitions into the asset.

- **MARL → single-agent bridge (general, broader than this task).**
Single-agent RL libraries train `DirectMARLEnv` tasks via
`multi_agent_to_single_agent`; the bridge dropped the latest
observations from the public buffer. Fixed generally — RSL-RL
observations read from the env-owned `obs_buf`, stored by `reset` like
`step` on all env bases (`DirectRLEnv`, `DirectMARLEnv`, the
experimental warp base). Every MARL task + single-agent runner benefits;
handover is the first consumer.
- **Handover Direct → single-agent + RSL-RL.** Single-agent conversion,
RSL-RL runner config, shared identity in `handover_common`, success-rate
reward metrics, and a fix for its Newton construction failure (see
below).
- **Shadow Hand Newton robot → the asset.** Moved the Newton (MJWarp)
robot cfg into `isaaclab_assets` as `SHADOW_HAND_NEWTON_CFG` (beside
`SHADOW_HAND_CFG`); reorient and handover consume it as **siblings**
(neither imports the other). Reorient uses the default gains; handover
overrides only the finger stiffness/damping to `20/2` for its catch.
- **Shadow camera cleanup.** Renamed the camera Direct modules to
`shadow_hand_direct_camera_env[_cfg]`; moved cube-keypoint math to the
shared, unit-tested `reorient/mdp/observations.py` (replacing the
per-env `compute_keypoints` shim); and added OVPhysX physics presets to
the handover and camera Direct environments. The camera env config is
otherwise unchanged from develop.

### Behavior changes

**Reorient-on-Newton is unchanged.** The Newton hand definition moves
**verbatim** into `SHADOW_HAND_NEWTON_CFG` — same joint expressions,
effort limits, gains, friction and armature — so its config diff vs
`develop` is purely the move, and the Direct env config has no diff at
all.

Handover changes in three ways:

- Its **default physics backend moves from PhysX to Newton** (MJWarp),
matching the reorientation tasks. Pass `physics=physx` for the previous
backend.
- Its `distal_passive` actuator group is **removed**. It targeted
`robot0_(FF|MF|RF|LF)J0` — joints that **do not exist** on the Newton
asset (its fingers are numbered `J1`–`J4`/`J5`), so constructing the
environment raised `No joints found for actuator group`. The `fingers`
group already drives those joints, and the comment justifying the group
(USD-baked `stiffness=286/damping=57`) does not apply to this asset,
which authors no drive stiffness at all.
- Two orientation fixes: the goal orientation was initialized to a
180-degree rotation instead of identity, and the Newton root rotation
replaced the asset's baked rotation instead of composing with it,
leaving both palms rotated 90 degrees.

Two pre-existing discrepancies were found while verifying against the
asset and are left for a follow-up, since each is a behavior change
needing its own validation: the config sets `armature=2e-3` where the
asset authors `0.0002`, and the per-finger `J1`/`J2` pair is coupled by
a fixed tendon that the MJWarp solver currently skips — the actuator
gains are what hold that pair together today, which is why the `J4`
knuckle-abduction joints cannot simply be added to the driven set.

## Notes

- Rebased onto **latest develop**; clean 3-commit history: `Add
single-agent flattening for MARL Direct environments` / `Convert Shadow
Hand handover to single-agent Direct with RSL-RL` / `Clean up Shadow
reorient/camera tasks and consolidate the Newton hand`.

## Validation

- Kit-free suite green: handover, keypoint math, the MARL adapter, and
new checks pinning the RSL-RL wrapper observation contract (verified to
fail against the previous implementation).
- Camera Direct reaches training takeoff under `rsl_rl` (frame-verified
videos); its env config now matches develop's established vision env.
- **Reorient-on-Newton vs `develop`** (`rsl_rl`, 2048 envs, 1500
iterations, 4 runs per config): this branch spans 524–664 reward,
`develop` spans 654–822 — overlapping distributions whose per-config
spreads (140–168) exceed the gap between them. The move is additionally
verified at the config level: the Newton robot configuration is
character-for-character identical to `develop`'s.
- **Handover-on-Newton** trains end-to-end on this branch (reward 1314),
which the `develop` config cannot do at all — it fails during
construction. Full evidence: #6324.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

asset New asset feature or request documentation Improvements or additions to documentation infrastructure isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant