[Bug] XR + cameras: scene renders white/untextured

[MihaiAnca13]

Describe the bug

Two problems hit while setting up Quest 3 / CloudXR teleoperation to record visuomotor
demonstrations with --xr --enable_cameras.

1. Scene renders white/untextured as soon as XR starts (main bug). In my own task, a
DirectRLEnv with two Camera sensors configured as CameraCfg(spawn=None) bound to camera prims
already authored inside a referenced USD asset, the Kit viewport shows the scene correctly until I
press Start XR, at which point both the viewport and the headset go blank white. Stopping XR
restores the correct render.

Geometry appears to render as white/untextured rather than being culled: with dome-light texture
randomisation on (stage.GetPrimAtPath("/World/Light").GetAttribute("inputs:texture:file").Set(dome_light_texture)), the headset shows the HDRI backdrop; with it off, the view is flat white.

This is not limited to the XR views. The RTX camera outputs are affected too, so demonstrations
recorded in this state would contain unusable images. Dumping camera RGB to disk PNG while XR is active gives:

• wrist camera: uniformly white
• ceiling camera: HDRI dome and none of the scene geometry

2. Erratic arm motion in Isaac-Stack-Cube-Galbot-Left-Arm-Gripper-Visuomotor-v0. This shipped
task renders correctly in the headset with cameras enabled, but the arm swings wildly as soon as I
move the controller instead of tracking my hand. I report it because it appears to be the only
shipped task combining cameras with XR teleop, so it is the natural reference for visuomotor data
collection over VR.

Steps to reproduce

Problem 2 (fully reproducible from the repo):

USE_RELATIVE_MODE=False ./isaaclab.sh -p scripts/tools/record_demos.py \
  --task Isaac-Stack-Cube-Galbot-Left-Arm-Gripper-Visuomotor-v0 \
  --viz kit --xr --enable_cameras \
  --dataset_file ./datasets/galbot_xr.hdf5 --num_demos 1

USE_RELATIVE_MODE=False is required: stack_rmp_rel_env_cfg.py:48 defaults it to "True", and
relative mode sets isaac_teleop = None (line 65), leaving no XR pipeline. The headset connects and
the scene renders; moving the controller produces wild arm motion.

Problem 1: I cannot share the environment (proprietary cell asset), so instead here is what
differs from the two configurations that do work correctly:

Configuration	XR + cameras	Result
Isaac-Stack-Cube-Galbot-...-Visuomotor-v0 (ManagerBased, spawned PinholeCameraCfg, 4 x 256²)	yes	renders correctly
My task with cameras removed (DirectRLEnv, same asset, same dome light)	XR, no cameras	renders correctly
My task with cameras (DirectRLEnv, CameraCfg(spawn=None), 2 x 910x512, data_types=["rgb"])	yes	white

Ruled out by experiment:

• XR anchor / eye height. The scene is not merely out of view, it is not anywhere.
• Anchor quaternion convention. XrCfg.anchor_rot is xyzw, which is what I pass.
• RenderCfg.rendering_mode. Mine sets "balanced"; skipping the preset changes nothing. It
  does remove the DLSS increasing input dimensions warning, since
  rendering_modes/balanced.kit sets rtx.post.dlss.execMode = 1.
• Render-target size. 910x512 down to 256x256 (matching Galbot) behaves identically.
• VRAM pressure. About 10.9 of 12.3 GB in use before XR starts; disabling dome-texture
  randomisation frees roughly 4.4 GB (5.8 GB free) with no change and no allocation failures logged.

System Info

• Commit: ffff603eafc (tag v3.0.0-beta2.patch1)
• Isaac Sim Version: 6.0.1-rc.7+release.42383.32955d8d.gl (pip isaacsim 6.0.1.0)
• OS: Ubuntu 24.04.4 LTS (kernel 6.8.0-136-generic)
• GPU: RTX 4080 Laptop GPU (12282 MiB)
• CUDA: 13.2 (driver runtime); torch built against 12.8
• GPU Driver: 595.84

Additional: CloudXR Runtime 6.2.0 (auto-launched by isaaclab_teleop), Meta Quest 3 with the
CloudXR JS client (cloudxrjs profile), experience file apps/isaaclab.python.xr.openxr.kit.

Additional context

Two further robustness issues found while debugging. Both are reproducible without my asset and are
independent of the above.

a) --xr silently forces the simulation onto CPU, hanging camera tasks indefinitely.
app_launcher.py:1003-1005:

if self._xr and not device_explicitly_passed:
    # If no device is specified, default to the CPU device if we are running in XR
    device = "cpu"

b) Reading camera data before the first render launches an out-of-bounds Warp kernel.
IsaacRtxRenderer.render (isaaclab_physx/renderers/isaac_rtx_renderer.py:473) launches
reshape_tiled_image with dim=(view_count, cfg.height, cfg.width) from the config, over whatever
annotator.get_data() returns. Before any frame has been rendered that buffer is empty, so the
kernel writes out of bounds:

• on cuda: CUDA error 700: an illegal memory access was encountered in wp_cuda_launch_kernel,
  once per camera
• on cpu: the same kernel never returns

Confirmed with CUDA_LAUNCH_BLOCKING=1. Without it the error surfaces later and misleadingly, at a
wp_memcpy_d2h of the env mask in camera.py:550, which sent me looking in the wrong place for a
long time.

In my case this was triggered by reading camera.data in the env's __init__ to record nominal
camera poses. That is arguably my bug, but the failure mode is very hard to diagnose, and non-XR runs
happen to get a viewport frame in first so it never appears without --xr.

Suggestion: validate the annotator buffer size against the expected dimensions and raise a clear
error, or skip the reshape, rather than launching the kernel.

Checklist

• I have checked that there is no similar issue in the repo (required)
• I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

• A DirectRLEnv with CameraCfg(spawn=None) cameras renders the scene correctly in the
  headset and in its RTX camera outputs under --xr --enable_cameras, or the unsupported
  combination is documented and reported with a clear error.
• Isaac-Stack-Cube-Galbot-Left-Arm-Gripper-Visuomotor-v0 tracks controller motion usably as
  shipped, or the docs state what needs calibrating and how.
• --xr no longer silently selects CPU when cameras are enabled, or it warns, so camera tasks do
  not hang without diagnostics.
• Reading camera data before the first render raises a clear error instead of launching an
  out-of-bounds Warp kernel.