[wip] mujoco xr as a quick robot visualizer - #869
Draft
jiwenc-nv wants to merge 3 commits into
Draft
Conversation
Render a MuJoCo scene stereoscopically into a Televiz XR session, with the operator's controllers drawn as markers. Single process, single thread, one OpenXR session: VizSession(kXr) owns the graphics-bound session and hands its handles to TeleopSession, so device trackers attach to the same session. The C++ renderer links no viz target. It borrows viz's VkDevice through the integer handles session_bindings.cpp already exposes "for callers wiring Televiz into a foreign Vulkan / OpenXR app", exports its per-eye colour and depth images as CUDA-linear buffers, and feeds ProjectionLayer.submit() over __cuda_array_interface__. Coupling is by protocol, not linkage, so the module needs only Vulkan, CUDA, MuJoCo and pybind11, and src/viz is unchanged. Controller poses are drawn as markers only -- no IK, no clutch, no rate limiting. A frame-convention bug and a control bug produce the identical symptom, and separating them is what makes the first one debuggable. The mujoco wheel supplies both the headers and libmujoco, so one library serves C++ and Python; the extension carries no RPATH and __init__.py imports mujoco first, making a version mismatch a clean ImportError rather than two copies loaded silently. Also adds glslang-tools, pkg-config and patchelf to the build prerequisites -- pkg-config is a hard configure failure under BUILD_VIZ=ON via GLFW, and patchelf was already documented but missing from the devcontainer. Verified: 313 ctest entries with the 4 pre-existing [window] failures (Tegra/Xvfb, unrelated) and 8 [xr] skips; 36 pytest cases; clang-format and pre-commit clean. The XR path itself -- the frame loop, session sharing via oxr_handles, depth-layer acceptance and controllers on a shared session -- is not verified anywhere: it needs a headset and a CloudXR runtime. The frame loop below the XR layer is exercised by --mode offscreen and by the Vulkan-to-CUDA-to-submit test, neither of which needs one. Planned and reviewed across three rounds by a six-persona panel. Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>
`uv pip install .` at the repository root never delivered this example: the
root pyproject's install.components filters out everything that is not
isaacteleop_wheel or isaacteleop_binaries, and neither of the example's
install rules named a component. Folding it in was the wrong fix -- _mujoco_xr
links libmujoco, so the isaacteleop wheel's contents would have depended on
whether the build host happened to have mujoco installed, and a MuJoCo-linked
.so would have shipped to everyone.
So the example becomes its own wheel, and the wheel is now the only way to run
it:
uv pip install ./examples/mujoco_xr
python -m mujoco_xr --mode offscreen
mujoco is a declared dependency rather than a build-host accident. This is
only possible because the module links no viz target -- just Vulkan, CUDA and
libmujoco.
The scene XML moves into the package (python/mujoco_xr/assets/) so it ships as
package data and one .parent resolves it both in the wheel and in the source
tree the tests import. All install() rules are gone, and the rig invokes
{python} -m mujoco_xr, which also gains a real preflight check: {python}
expands to an absolute sys.executable, so _check_commands_exist validates it
where the old bare `uv` token was skipped. It does NOT gain the
find_runtime_footguns() warning -- that requires `-m isaacteleop` or a .py
token -- so --no-launch-cloudxr-runtime remains load-bearing and unguarded.
The CMakeLists is dual-mode: add_subdirectory'd from the root, or top-level
under scikit-build-core. The in-tree path stays because ctest runs against it,
so the extension compiles twice. That is deliberate; collapsing it needs
isaacteleop on an index so the test environment can resolve it.
Note sdist.exclude, not wheel.exclude, keeps the in-place .so out of the
wheel. scikit-build-core reads .gitignore relative to the project root, so the
repo-wide *.so never applied, and wheel.exclude is applied again at zip time
where it would have deleted the real extension instead.
Verified: 313 ctest entries with the 4 pre-existing [window] failures and 8
[xr] skips, 36 pytest cases, clang-format, ruff and pre-commit clean. A cold
install into a fresh venv yields exactly one .so and runs to a clean exit.
Reviewed by tech-lead, bar-raiser and YAGNI.
Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>
A bare top-level `mujoco_xr` in site-packages is a namespace claim staked by
an example. Move it under a PEP 420 namespace so the import path says where it
came from, and so future examples have somewhere to live:
dist: isaacteleop-examples-mujoco-xr (was mujoco-xr-example)
import: isaacteleop_examples.mujoco_xr (was mujoco_xr)
run: python -m isaacteleop_examples.mujoco_xr
`isaacteleop_examples/` deliberately has no __init__.py. That is what lets a
second example distribution join the same namespace later; a regular package
would own it exclusively.
Only one other example carries an isaacteleop- dist prefix
(isaacteleop-haptic-feedback-examples), so this diverges from the bare names
the other eleven use. It is deliberate: this is the only example that installs
an importable package, so it is the only one whose name can collide with
anything.
The C++ side is untouched -- namespace mujoco_xr, PYBIND11_MODULE(_mujoco_xr),
the <mujoco_xr/shaders/...> include prefix, the mujoco_xr_py target, the ctest
label and the [mujoco_xr] log prefix all stay. The extension is reached by a
relative import, so the namespace does not appear in C++ at all.
sdist.exclude moved with the package. It is coupled to wheel.packages by path,
and leaving it behind would have silently re-admitted the stale in-place .so
that the previous commit's note exists to prevent -- verified cross-ABI, since
a same-ABI count cannot distinguish the two.
One behaviour change fell out of the rename rather than being designed in:
find_runtime_footguns() tests for the substring `-m isaacteleop`, which
`-m isaacteleop_examples.mujoco_xr` now contains and `-m mujoco_xr` did not.
The rig's warning about a dropped --no-launch-cloudxr-runtime is therefore
live where it previously could not fire. Measured, and the comment now says so.
Verified: 313 ctest entries with the 4 pre-existing [window] failures and 8
[xr] skips, 36 pytest cases, clang-format, ruff and pre-commit clean. A cold
install into a fresh venv ships exactly one .so, no namespace __init__.py, and
runs to a clean exit.
Signed-off-by: Jiwen Cai <jiwenc@nvidia.com>
Contributor
|
Important Review skippedIgnore keyword(s) in the title. ⛔ Ignored keywords (2)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
jiwenc-nv
marked this pull request as draft
August 2, 2026 01:01
Contributor
|
📝 Docs preview is not auto-deployed for fork PRs. A maintainer with write access to |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #(issue)
Type of change
Testing
Checklist
SKIP=check-copyright-year pre-commit run --all-filesgit commit -s) per the DCO