Is your feature request related to a problem? Please describe.
Unitree has reported ~1 second of HMD streaming delay running XR teleoperation on https://github.com/unitreerobotics/xr_teleoperate, and has asked whether NVIDIA can help optimize it. At that level closed-loop teleoperation breaks down — the operator over-corrects because visual feedback lags their motion. Two problems compound: a hand-rolled XR streaming path (capture → encode → transport → decode → present, plus the pose round trip) has many places to accumulate a second of delay, and it reports no per-stage timing, so the figure can't be attributed to any of them. Optimizing in place means building the measurement infrastructure first, then tuning a bespoke pipeline against it.
Describe the solution you'd like
Migrate the XR transport to Isaac Teleop, which uses CloudXR as its streaming layer. This is the supported path, and it addresses both problems:
-
The pipeline is purpose-built for low-latency XR. Isaac Teleop shares one OpenXR session between rendering and the teleop device trackers, so pose and video ride a single CloudXR connection rather than two independently-tuned paths.
-
Per-stage latency telemetry already ships. The out-of-band teleop control hub streams live CloudXR metrics — render.pose_to_render_time, latency.pose_upload_ms, latency.pose_to_frame_received_ms, latency.xr_pose_age_ms, frame_pipeline.compositor_skipped_percent, streaming.framerate, network.latency_score. Names are CloudXR.js MetricsName values, so they map onto the SDK documentation. A latency figure becomes attributable to a stage instead of an opaque total. (That page's sample payload shows render.pose_to_render_time of 18.5 ms at 72 fps — illustrative of the telemetry format rather than a benchmark, but the right order of magnitude to compare against.)
Tracker samples also carry sample_time_local_common_clock and available_time_local_common_clock in a shared clock domain, so pipeline latency and cross-device sync can be measured from recordings rather than only live.
Describe alternatives you've considered
- Profile and optimize
xr_teleoperate's existing streaming path in place. Requires building the per-stage measurement infrastructure first, then maintaining a bespoke low-latency XR transport indefinitely — duplicating what CloudXR already provides.
- External high-speed camera measurement. Ground truth for total motion-to-photon, but no stage attribution and doesn't scale to routine regression checking.
- Encoder/decoder statistics alone. Miss queuing and compositor delays, which is often exactly where the latency accumulates.
Additional context
XR-headset teleoperation over CloudXR is supported today (see CloudXR teleoperation in Isaac Lab); remote teleoperation with immersive camera streaming to XR headsets is listed as an upcoming use case in the README, so a fully remote deployment should be planned against that roadmap item.
One thing worth pinning down early, whichever path is taken: what the ~1 s figure actually measures — motion-to-photon, glass-to-glass video only, or the pose round trip. Those differ substantially, and the definition has to be fixed before any before/after comparison means anything.
Is your feature request related to a problem? Please describe.
Unitree has reported ~1 second of HMD streaming delay running XR teleoperation on https://github.com/unitreerobotics/xr_teleoperate, and has asked whether NVIDIA can help optimize it. At that level closed-loop teleoperation breaks down — the operator over-corrects because visual feedback lags their motion. Two problems compound: a hand-rolled XR streaming path (capture → encode → transport → decode → present, plus the pose round trip) has many places to accumulate a second of delay, and it reports no per-stage timing, so the figure can't be attributed to any of them. Optimizing in place means building the measurement infrastructure first, then tuning a bespoke pipeline against it.
Describe the solution you'd like
Migrate the XR transport to Isaac Teleop, which uses CloudXR as its streaming layer. This is the supported path, and it addresses both problems:
The pipeline is purpose-built for low-latency XR. Isaac Teleop shares one OpenXR session between rendering and the teleop device trackers, so pose and video ride a single CloudXR connection rather than two independently-tuned paths.
Per-stage latency telemetry already ships. The out-of-band teleop control hub streams live CloudXR metrics —
render.pose_to_render_time,latency.pose_upload_ms,latency.pose_to_frame_received_ms,latency.xr_pose_age_ms,frame_pipeline.compositor_skipped_percent,streaming.framerate,network.latency_score. Names are CloudXR.jsMetricsNamevalues, so they map onto the SDK documentation. A latency figure becomes attributable to a stage instead of an opaque total. (That page's sample payload showsrender.pose_to_render_timeof 18.5 ms at 72 fps — illustrative of the telemetry format rather than a benchmark, but the right order of magnitude to compare against.)Tracker samples also carry
sample_time_local_common_clockandavailable_time_local_common_clockin a shared clock domain, so pipeline latency and cross-device sync can be measured from recordings rather than only live.Describe alternatives you've considered
xr_teleoperate's existing streaming path in place. Requires building the per-stage measurement infrastructure first, then maintaining a bespoke low-latency XR transport indefinitely — duplicating what CloudXR already provides.Additional context
XR-headset teleoperation over CloudXR is supported today (see CloudXR teleoperation in Isaac Lab); remote teleoperation with immersive camera streaming to XR headsets is listed as an upcoming use case in the README, so a fully remote deployment should be planned against that roadmap item.
One thing worth pinning down early, whichever path is taken: what the ~1 s figure actually measures — motion-to-photon, glass-to-glass video only, or the pose round trip. Those differ substantially, and the definition has to be fixed before any before/after comparison means anything.