Add XR camera picture-in-picture feedback

docs/source/api/lab_teleop/isaaclab_teleop.rst

@@ -11,6 +11,8 @@ isaaclab_teleop
 
     IsaacTeleopCfg
     IsaacTeleopDevice
+    XrCameraFeedCfg
+    XrCameraFeedLayoutCfg
     HapticFeedbackCfg
     ControllerHapticFeedbackCfg
     GloveHapticFeedbackCfg
@@ -40,6 +42,15 @@ Configuration
 .. autoclass:: XrAnchorRotationMode
     :members:
 
+XR Camera Feedback
+------------------
+
+.. autoclass:: XrCameraFeedCfg
+    :members:
+
+.. autoclass:: XrCameraFeedLayoutCfg
+    :members:
+
 Device
 ------
 

docs/source/features/isaac_teleop.rst

@@ -949,6 +949,10 @@ Key ``IsaacTeleopCfg`` fields:
 * ``pipeline_builder`` -- callable that returns an ``OutputCombiner`` with an ``"action"`` output.
 * ``retargeters_to_tune`` -- optional callable returning retargeters to expose in the live tuning UI.
 * ``xr_cfg`` -- :class:`~isaaclab_teleop.XrCfg` for anchor configuration (see below).
+* ``xr_camera_feeds`` -- ordered selection and per-feed panel settings for existing task cameras.
+  The list is empty by default, so tasks opt in to PiP explicitly.
+* ``xr_camera_feed_layout`` -- viewer reference, fixed world pose, and manual, horizontal, vertical,
+  or grid panel packing.
 * ``plugins`` -- list of Isaac Teleop plugin configurations (e.g. Manus).
 * ``sim_device`` -- torch device string (default ``"cuda:0"``).
 * ``retargeting_execution`` -- IsaacTeleop retargeting execution settings.
@@ -1152,8 +1156,150 @@ XR device's view.
 
 .. tip::
 
-   When using XR, call :func:`~isaaclab_teleop.remove_camera_configs` on your env config to strip
-   camera sensors. Additional cameras cause GPU contention and degrade XR performance.
+   Camera sensors add GPU cost. Strip them with ``--disable_external_cameras`` when the workflow
+   needs neither image observations nor XR camera feedback. That flag also skips configured PiP
+   feeds.
+
+
+XR Camera Feedback
+------------------
+
+An ordered list of :class:`~isaaclab_teleop.XrCameraFeedCfg` objects selects existing task scene
+cameras. The manager publishes each new RGBA frame after rendering, while
+:class:`~isaaclab_teleop.XrCameraFeedLayoutCfg` places the panels manually or in horizontal,
+vertical, and grid layouts. ``IsaacContrib-PickPlace-GR1T2-Abs`` and
+``IsaacContrib-PickPlace-Locomanipulation-G1-Abs`` are the primary reference examples.
+
+``teleop_se3_agent.py`` and ``record_demos.py`` show every enabled feed when an IsaacTeleop-enabled
+environment runs with ``--xr``. PiP is absent unless the task explicitly selects an existing
+``CameraCfg`` through ``xr_camera_feeds``. In the reference examples, the selected
+``robot_pov_cam`` is also a policy image observation, so the normal demonstration recorder stores
+the same view shown to the operator. GR1T2 demonstrates a fixed task camera; Locomanipulation uses
+the same fixed task-camera placement pattern for G1. The NutPour and ExhaustPipe GR1T2
+teleoperation tasks also present their existing recorded ``robot_pov_cam``:
+
+.. code-block:: bash
+
+   uv run python scripts/environments/teleoperation/teleop_se3_agent.py \
+       --task IsaacContrib-PickPlace-GR1T2-Abs \
+       --xr --device cpu
+
+   uv run python scripts/environments/teleoperation/teleop_se3_agent.py \
+       --task IsaacContrib-PickPlace-Locomanipulation-G1-Abs \
+       --xr --device cpu
+
+XR camera PiP currently supports exactly one environment. When a task has enabled PiP feeds,
+startup rejects ``--num_envs`` values other than ``1``; IsaacTeleop XR behavior without PiP is
+unchanged.
+
+On Isaac Sim 6.1 and newer, camera-feed setup enables responsive DLSS Ray Reconstruction denoising
+automatically. For every camera consumer, the Isaac RTX renderer falls back to classic DLSS on
+earlier runtimes to avoid the temporal ghosting in their Ray Reconstruction implementation.
+
+Camera selection
+~~~~~~~~~~~~~~~~
+
+Tasks declare their default selection through ``IsaacTeleopCfg.xr_camera_feeds``:
+
+.. code-block:: python
+
+   from isaaclab_teleop import IsaacTeleopCfg, XrCameraFeedCfg
+
+   self.isaac_teleop = IsaacTeleopCfg(
+       pipeline_builder=_build_my_pipeline,
+       xr_camera_feeds=[
+           XrCameraFeedCfg(camera_name="left_wrist_camera"),
+           XrCameraFeedCfg(camera_name="overview_camera", enabled=False),
+       ],
+   )
+
+For a recorded training view, define the named ``CameraCfg`` in the task scene and a matching
+``mdp.image`` term in ``observations.policy``. The normal recorder then stores that observation,
+while the PiP declaration above only selects it for presentation. Enabled task-declared entries
+control camera selection, panel count, display order, and automatic-layout order without mutating
+the reusable task configuration.
+
+Placement
+~~~~~~~~~
+
+The default :attr:`~isaaclab_teleop.XrCameraFeedLayoutCfg.placement` is ``"viewer_start"``. The
+presenter waits for the first valid headset pose, captures its eye position and upright yaw, then
+places the layout 0.8 m ahead at eye height. That pose remains fixed in the world while the user
+moves. Panels hide when the XR display disconnects and capture a new starting pose after reconnect;
+resetting the environment does not recenter them.
+
+Three placement references are available:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 22 78
+
+   * - Placement
+     - Behavior
+   * - ``viewer_start``
+     - Capture the first valid eye position and upright yaw, then keep the layout world-static.
+       This is the default.
+   * - ``head_locked``
+     - Follow the current headset position and orientation with the configured offset and distance.
+   * - ``world``
+     - Use ``world_position_m`` and ``world_orientation_xyzw`` as a fixed pose in the Isaac Lab USD
+       stage world. ``distance_m`` is unused.
+
+.. code-block:: python
+
+   from isaaclab_teleop import XrCameraFeedLayoutCfg
+
+   # Default: fixed in the world at the user's starting eye pose.
+   self.isaac_teleop.xr_camera_feed_layout = XrCameraFeedLayoutCfg()
+
+   # Follow the headset and pack feeds left to right.
+   self.isaac_teleop.xr_camera_feed_layout = XrCameraFeedLayoutCfg(
+       placement="head_locked",
+       mode="horizontal",
+       distance_m=0.8,
+   )
+
+   # Place a grid in an Isaac Lab Z-up world. This example assumes an eye at
+   # (0, 0, 1.6) looking along world +Y.
+   self.isaac_teleop.xr_camera_feed_layout = XrCameraFeedLayoutCfg(
+       placement="world",
+       mode="grid",
+       world_position_m=(0.0, 0.8, 1.6),
+       world_orientation_xyzw=(0.7071067812, 0.0, 0.0, 0.7071067812),
+       max_columns=2,
+   )
+
+``manual`` mode preserves each feed's ``offset_m`` and ``distance_m``. ``horizontal``, ``vertical``,
+and ``grid`` modes pack enabled feeds around ``center_offset_m`` using ``panel_gap_m``; grid mode
+also honors ``max_columns``. Offsets are measured in the selected layout plane. With
+``viewer_start`` and ``head_locked``, ``distance_m`` places that plane in front of the viewer
+reference. In manual mode, feeds with identical offsets overlap; select an automatic mode or assign
+distinct feed offsets when showing multiple panels.
+
+With ``world``, the explicit pose is measured in the Isaac Lab USD stage world in meters, not
+OpenXR physical space. Isaac Lab stages are Z-up. ``world_orientation_xyzw`` maps panel-local
+coordinates into that world: local +X is image right, local +Y is image up, and local +Z points
+from the readable face toward the viewer. Feed and automatic-layout offsets are applied in the
+panel's local XY plane.
+
+Disable cameras and PiP
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Set :attr:`~isaaclab_teleop.XrCameraFeedCfg.enabled` to ``False`` to suppress one feed, or set
+``xr_camera_feeds=[]`` to suppress all PiP. The list is empty by default. These choices do not
+remove camera sensors independently owned by the task.
+
+``--disable_external_cameras`` is the master camera-rendering switch for ``teleop_se3_agent.py`` and
+``record_demos.py``. It strips camera sensors and suppresses every configured PiP feed.
+
+Kit Scene UI presentation
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+PiP presentation uses Kit Scene UI and ``SpatialSource`` placement. Kit imports are deferred until
+an enabled feed is requested. If the Scene UI modules cannot be imported, the scripts log a warning
+and continue without PiP; task-owned cameras and recording observations remain unchanged. This keeps
+the camera selection configuration usable when a future kitless entry point no longer provides
+Scene UI. Configuration, camera-buffer, and panel-initialization errors still fail during startup.
 
 
 .. _isaac-teleop-haptics:
@@ -1557,8 +1703,10 @@ Optimize XR Performance
 
    The flag strips the environment's camera sensors (equivalent to calling
    :func:`~isaaclab_teleop.remove_camera_configs` on the env config) and selects a lighter Kit
-   experience. Omit it to keep cameras enabled (the default) -- required when recording camera
-   observations, or when you want ``teleop_replay_agent.py`` to mirror the production render load.
+   experience. For ``teleop_se3_agent.py`` it is also the master PiP gate: task-default and
+   task-configured feeds are ignored and no camera panel is created. Omit it to keep cameras enabled
+   (the default) -- required when recording camera observations, showing XR camera feedback, or when
+   you want ``teleop_replay_agent.py`` to mirror the production render load.
 
 .. dropdown:: Run headless (skip the local viewport)
    :open:
@@ -1683,6 +1831,8 @@ See the :ref:`isaaclab_teleop-api` for full class and function documentation:
 * :class:`~isaaclab_teleop.HapticFeedbackReceiver`
 * :class:`~isaaclab_teleop.HapticFeedbackDriver`
 * :func:`~isaaclab_teleop.create_haptic_feedback_driver`
+* :class:`~isaaclab_teleop.XrCameraFeedCfg`
+* :class:`~isaaclab_teleop.XrCameraFeedLayoutCfg`
 * :class:`~isaaclab_teleop.XrCfg`
 * :class:`~isaaclab_teleop.XrAnchorRotationMode`
 

scripts/environments/teleoperation/teleop_se3_agent.py

@@ -272,6 +272,14 @@ def main() -> None:  # noqa: C901
         not teleop_device_explicitly_set and hasattr(env_cfg, "isaac_teleop") and env_cfg.isaac_teleop is not None
     )
 
+    from isaaclab_teleop.camera_feed import _XrCameraFeedSession
+
+    camera_feed_session = _XrCameraFeedSession.prepare(
+        env_cfg,
+        enabled=args_cli.xr and use_isaac_teleop,
+        camera_rendering_enabled=not args_cli.disable_external_cameras,
+    )
+
     # XR-rendering setup (camera removal + DLSS) is only needed for the Kit XR
     # path. Without --xr, IsaacTeleop runs standalone (I/O only) and renders
     # normally, so gate on --xr alone.
@@ -286,7 +294,14 @@ def main() -> None:  # noqa: C901
     if _rtx_rendering_requested(args_cli):
         _ensure_replicator_loaded()
         apply_isaac_rtx_global_settings(
-            IsaacRtxRendererGlobalSettingsCfg(antialiasing_mode="DLSS"),
+            IsaacRtxRendererGlobalSettingsCfg(
+                antialiasing_mode="DLSS",
+                carb_settings=(
+                    {"/rtx/dldenoiser/responsiveDenoising": True}
+                    if camera_feed_session.requires_responsive_denoising
+                    else None
+                ),
+            ),
         )
 
     try:
@@ -493,6 +508,7 @@ def run_loop():
                     if should_reset_recording_instance:
                         env.reset()
                         teleop_interface.reset()
+                        camera_feed_session.refresh()
                         should_reset_recording_instance = False
                         print("Environment reset complete")
             except Exception as e:
@@ -502,10 +518,11 @@ def run_loop():
     # Run the teleoperation loop
     # IsaacTeleop requires a context manager, native devices don't
     if use_isaac_teleop:
-        with teleop_interface:
+        with teleop_interface, camera_feed_session.bind(env):
             run_loop()
     else:
-        run_loop()
+        with camera_feed_session.bind(env):
+            run_loop()
 
     # close the simulator
     env.close()

scripts/tools/record_demos.py

@@ -34,11 +34,15 @@
 # Standard library imports
 import argparse
 import contextlib
+from typing import TYPE_CHECKING
 
 # Isaac Lab AppLauncher
 from isaaclab.app import AppLauncher
 from isaaclab.utils.string import list_intersection, string_to_callable
 
+if TYPE_CHECKING:
+    from isaaclab_teleop.camera_feed import _XrCameraFeedSession
+
 # add argparse arguments
 parser = argparse.ArgumentParser(description="Record demonstrations for Isaac Lab environments.")
 parser.add_argument("--task", type=str, required=True, help="Name of the task.")
@@ -575,6 +579,7 @@ def run_simulation_loop(  # noqa: C901
     teleop_interface: object | None,
     success_term: object | None,
     rate_limiter: RateLimiter | None,
+    camera_feed_session: "_XrCameraFeedSession",
     use_isaac_teleop: bool = False,
 ) -> int:
     """Run the main simulation loop for collecting demonstrations.
@@ -588,6 +593,7 @@ def run_simulation_loop(  # noqa: C901
         teleop_interface: Optional teleop interface (will be created if None)
         success_term: The success termination object or None if not available
         rate_limiter: Optional rate limiter to control simulation speed
+        camera_feed_session: Shared XR camera-feed lifecycle
         use_isaac_teleop: Whether to use IsaacTeleop stack
 
     Returns:
@@ -694,7 +700,7 @@ def inner_loop():
         if use_isaac_teleop:
             from isaaclab_teleop import poll_control_events
 
-        with contextlib.suppress(KeyboardInterrupt), torch.inference_mode():
+        with contextlib.suppress(KeyboardInterrupt), torch.inference_mode(), camera_feed_session.bind(env):
             while simulation_app.is_running():
                 # Get teleop command (may be None while waiting for session start)
                 action = teleop_interface.advance()
@@ -764,6 +770,7 @@ def inner_loop():
                     success_step_count = handle_reset(
                         env, success_step_count, instruction_display, label_text, teleop_interface
                     )
+                    camera_feed_session.refresh()
                     should_reset_recording_instance = False
 
                 # Check if simulation is stopped
@@ -804,6 +811,20 @@ def main() -> None:
     global env_cfg  # Make env_cfg available to setup_teleop_device
     env_cfg, success_term, use_isaac_teleop = create_environment_config(output_dir, output_file_name)
 
+    from isaaclab_teleop.camera_feed import _XrCameraFeedSession
+
+    camera_feed_session = _XrCameraFeedSession.prepare(
+        env_cfg,
+        enabled=args_cli.xr and use_isaac_teleop,
+        camera_rendering_enabled=not args_cli.disable_external_cameras,
+    )
+    if camera_feed_session.requires_responsive_denoising:
+        apply_isaac_rtx_global_settings(
+            IsaacRtxRendererGlobalSettingsCfg(
+                carb_settings={"/rtx/dldenoiser/responsiveDenoising": True},
+            )
+        )
+
     # With --xr, rate limiting is achieved via OpenXR and the XR visualization
     # manager is installed. Without --xr (including standalone IsaacTeleop I/O),
     # fall back to the software rate limiter and skip the XR viz stack.
@@ -820,7 +841,9 @@ def main() -> None:
     env = create_environment(env_cfg)
 
     # Run simulation loop
-    current_recorded_demo_count = run_simulation_loop(env, None, success_term, rate_limiter, use_isaac_teleop)
+    current_recorded_demo_count = run_simulation_loop(
+        env, None, success_term, rate_limiter, camera_feed_session, use_isaac_teleop
+    )
 
     # Clean up
     env.close()

source/isaaclab/changelog.d/xr-camera-feedback.minor.rst

@@ -0,0 +1,5 @@
+Added
+^^^^^
+
+* Allowed renderers to select a camera pixel-output device independently from camera pose and
+  simulation state.

source/isaaclab/isaaclab/renderers/base_renderer.py

@@ -44,6 +44,21 @@ def initialize(self) -> None:
         """Post-physics one-time initialization hook. Called only once."""
         return
 
+    def resolve_camera_output_device(self, simulation_device: str) -> str:
+        """Resolve the device used for persistent camera pixel outputs.
+
+        The default keeps outputs on the simulation device. A renderer may
+        override this when its pixel production is independent of physics and
+        camera pose storage.
+
+        Args:
+            simulation_device: Device used by simulation and camera state.
+
+        Returns:
+            Device used by the render specification and image output buffers.
+        """
+        return simulation_device
+
     def prepare_cameras(self, stage: Any, spec: CameraRenderSpec) -> None:
         """Pre-render per-camera setup the backend needs.