Add XR camera picture-in-picture feedback

[hougantc-nvda]

Description

Adds low-latency XR camera picture-in-picture (PiP) feedback to IsaacTeleop so teleoperators can see the same task-configured camera views that are captured in demonstration data.

The implementation:

• Selects existing task cameras declaratively through ordered IsaacTeleopCfg.xr_camera_feeds.
• Supports manual, horizontal, vertical, and grid layouts with viewer-start, head-locked, and world placement.
• Disables PiP when no feeds are configured or external cameras are disabled.
• Shares the task-configured feed path between teleop_se3_agent.py and record_demos.py.
• Keeps CPU physics and camera pose state on CPU while Isaac RTX produces camera pixels on cuda:0.
• Uses MultiBackendRendererCfg for the GR1T2 and G1 reference cameras so other renderer backends remain valid.
• Loads Kit Scene UI lazily so kitless runs warn and continue without PiP or task-camera mutation.
• Uses responsive DLSS Ray Reconstruction on Isaac Sim 6.1+ and falls back to classic DLSS on older supported runtimes.
• Limits XR camera PiP to one environment while preserving existing multi-environment behavior for IsaacTeleop XR without PiP.

Core changes are limited to two capabilities unavailable through existing public APIs: separating camera pixel output from the simulation device, and applying camera-local RTX settings when the renderer creates its render product. This keeps CPU physics and camera state on CPU while producing CUDA pixels without a readback, and avoids changing the XR headset renderer globally. Existing renderer behavior is preserved by default.

Canonical example:

./isaaclab.sh -p scripts/environments/teleoperation/teleop_se3_agent.py \
  --task IsaacContrib-PickPlace-GR1T2-Abs \
  --xr \
  --device cpu

Type of change

• New feature (non-breaking change which adds functionality)
• Documentation update

Screenshots

Quest 3 and IsaacTeleop Web Client acceptance was completed. A headset demonstration video was recorded separately and can be attached to the PR.

Validation

• pre-commit run --all-files
• 214 focused renderer contract/config, camera-feed, layout, Scene UI, task preset, control-event, and teleoperation-environment tests
• 2 Isaac Sim GPU integration tests, including real CPU camera state with RGB output on cuda:0
• Full Sphinx documentation build with warnings treated as errors
• Quest 3 and IsaacTeleop Web Client acceptance of the PiP path

Checklist

• I have read and understood the contribution guidelines
• I have run the pre-commit checks with ./isaaclab.sh --format
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump extension.toml — CI handles that)
• I have added my name to the CONTRIBUTORS.md or my name already exists there

[greptile-apps]

Greptile Summary

Adds task-configured XR camera picture-in-picture feedback throughout the teleoperation stack.

• Separates camera pixel-output placement from simulation and camera-state devices.
• Adds camera-local RTX/DLSS settings and version-aware responsive denoising.
• Adds Scene UI panel presentation, layouts, placement modes, reset refresh, and task presets.
• Integrates PiP lifecycle management into teleoperation and demonstration-recording entrypoints.

Confidence Score: 4/5

The PR appears safe to merge, with a non-blocking robustness gap when optional Scene UI runtime initialization fails after its modules import successfully.

The core camera-output and rendering changes are internally consistent, but the two entrypoints currently allow a late PiP initialization exception to terminate the parent workflow rather than disabling the optional presentation layer.

Files Needing Attention: source/isaaclab_teleop/isaaclab_teleop/camera_feed.py, scripts/environments/teleoperation/teleop_se3_agent.py, scripts/tools/record_demos.py

Important Files Changed

Filename	Overview
source/isaaclab_teleop/isaaclab_teleop/camera_feed.py	Adds feed validation, layout, binding, frame publication, and reset-refresh lifecycle; runtime presenter failures are not converted to the advertised optional-feature fallback.
source/isaaclab_teleop/isaaclab_teleop/camera_feed_kit_scene_ui.py	Implements viewer-start, head-locked, and world-space Scene UI panels with direct CUDA image upload.
source/isaaclab/isaaclab/sensors/camera/camera.py	Splits renderer pixel-output allocation from CPU/GPU camera pose and intrinsic state.
source/isaaclab_physx/isaaclab_physx/renderers/isaac_rtx_renderer.py	Adds RTX camera output-device resolution and per-render-product DLSS settings.
scripts/environments/teleoperation/teleop_se3_agent.py	Integrates feed preparation, responsive denoising, binding, and reset refresh into interactive teleoperation.
scripts/tools/record_demos.py	Integrates the same camera-feed lifecycle into demonstration recording.
source/isaaclab_teleop/isaaclab_teleop/isaac_teleop_cfg.py	Adds declarative per-feed and layout configuration classes.

Sequence Diagram

sequenceDiagram
    participant Task as Task configuration
    participant Script as Teleop/record script
    participant Env as Isaac Lab environment
    participant Camera as Camera sensor
    participant RTX as Isaac RTX renderer
    participant PiP as Scene UI presenter

    Task->>Script: XR feed names and layout
    Script->>Script: Prepare and validate feed session
    Script->>Env: Construct environment
    Env->>Camera: Initialize selected cameras
    Camera->>RTX: Create render product on output device
    Script->>PiP: Bind feed session
    loop Rendered frames
        RTX->>Camera: Produce RGBA output
        PiP->>Camera: Read persistent RGBA buffer
        PiP->>PiP: Upload image and place panel
    end

Loading

_{Reviews (1): Last reviewed commit: "Add XR camera picture-in-picture feedbac..." | Re-trigger Greptile}

[isaaclab-review-bot]

Isaac Lab Review Bot

The XR PiP design cleanly separates camera pixel-output placement from simulation state and adds render-product-local DLSS controls. Two fixes remain: the new README example uses a repository-unsupported invocation path, and the pre-6.1 Ray Reconstruction fallback only runs during PiP preparation even though reference task cameras enable Ray Reconstruction unconditionally.

• Design and architecture: The renderer boundary for output-device resolution and render-product-local settings is appropriate. However, compatibility handling for Isaac Sim versions before 6.1 is confined to _XrCameraFeedSession.prepare(). Consequently, the same task cameras retain Ray Reconstruction in non-XR, disabled-PiP, and kitless paths, bypassing the stated ghosting fallback. Resolve the effective setting in the RTX renderer or another path shared by every camera consumer.
• API: The public XrCameraFeedCfg and XrCameraFeedLayoutCfg types are exported and integrated into the API and feature documentation. No unsupported public API removal or compatibility break was established.
• Implementation: Camera state and image-output allocation paths, post-annotator render-product setting order, reset rebinding, and control reset latching were traced. Update the new README command from ./isaaclab.sh -p to uv run python per repository workflow rules, and ensure the pre-6.1 DLSS fallback also applies when reference cameras run outside an active PiP session.

Minor fixes needed. Posted 2 actionable findings inline.

Automated review; human maintainers own approval decisions.