Merge pull request #309 from RobotControlStack/juelg/hardware-fixes

fix: franka hardware testing

Author: juelg

.dockerignore

@@ -1 +1,10 @@
 **/build/
+real*
+test*
+sim*
+transfer_cube
+single_pick*
+ball_maze
+candy
+*.parquet
+*venv*

Makefile

@@ -2,7 +2,7 @@ PYSRC = python
 CPPSRC = src
 COMPILE_MODE = Release
 LINT_EXCLUDE_RUFF = --exclude examples/teleop/SimPublisher
-LINT_EXCLUDE_MYPY = 'build|examples/teleop/SimPublisher'
+LINT_EXCLUDE_MYPY = 'build|examples/teleop/SimPublisher|examples/inference/franka.py'
 
 # CPP
 cppcheckformat:

README.md

@@ -26,7 +26,7 @@ Traditional robotics middleware (like ROS/ROS2) and complex motion planning pipe
 * **Zero ROS Overhead:** No complex message-passing, middleware, or network configuration required. Run natively in Python with a lightweight C++ backend.
 * **Frictionless Sim-to-Real:** Train your Reinforcement Learning or VLA policies in our MuJoCo Gymnasium wrapper, and deploy the *exact same code* directly to physical hardware.
 * **Synchronous Execution:** Optimized specifically for the highly parallelized, synchronous data collection required by modern ML workflows.
-* **Ready-to-Use Apps:** Ships with pre-built applications for data collection via teleoperation and remote model inference via [vlagents](https://github.com/RobotControlStack/vlagents).
+* **Ready-to-Use Apps:** Ships with pre-built applications for data collection via teleoperation and remote model inference via [vlagents](https://github.com/RobotControlStack/vlagents). See [examples/teleop/README.md](examples/teleop/README.md) and [examples/inference/README.md](examples/inference/README.md).
 
 ## 🧩 Wrapper-Based Architecture
 
@@ -132,7 +132,7 @@ pip install rcs-core
 
 ### From Source
 
-Make sure that common build tools (i.e., `build-essential`) and a C++ compiler like `gcc` or `clang` are installed on your system/conda/docker.
+Make sure that common build tools (i.e., `build-essential`), python headers and a C++ compiler like `gcc` or `clang` are installed on your system/conda/docker.
 
 *RCS works best in Python 3.11, and all extensions have been tested to work in 3.11.*
 
@@ -196,6 +196,7 @@ For full documentation, including advanced installation, modular usage, and API
 Useful quick-reference pages:
 - **[RCS Conventions](https://robotcontrolstack.org/user_guide/conventions)** for quaternion order, frames, Euler angles, and gripper semantics
 - **[Sim Scene Configuration](https://robotcontrolstack.org/user_guide/scene_configuration)** for `SimEnvCreatorConfig`, scene frames, and example setup patterns
+- **[Apps](https://robotcontrolstack.org/apps/index)** for the teleoperation and inference example entry points
 - **[libfranka Version Info](https://robotcontrolstack.org/extensions/libfranka_versions)** for the currently pinned `rcs_fr3` and `rcs_panda` `libfranka` versions and local-install guidance
 
 ## 🤝 Contribution

docker/Dockerfile

@@ -20,6 +20,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
     libglfw3-dev \
     libpoco-dev \
     ninja-build \
+    liburdfdom-dev \
     && rm -rf /var/lib/apt/lists/*
 
 RUN curl -LsSf https://astral.sh/uv/install.sh | sh
@@ -53,7 +54,11 @@ RUN chmod +x /usr/local/bin/link-editable-source \
     && uv pip install --no-build-isolation /opt/rcs-src/extensions/rcs_fr3 \
     && uv pip install /opt/rcs-src/extensions/rcs_realsense \
     && uv pip install /opt/rcs-src/extensions/rcs_robotiq2f85 \
-    && uv pip install /opt/rcs-src/extensions/rcs_zed
+    && uv pip install /opt/rcs-src/extensions/rcs_zed \
+    && uv pip install /opt/rcs-src/examples/teleop/SimPublisher \
+    && uv pip install /opt/rcs-src/2f85-python-driver \
+    && uv pip install /opt/rcs-src/vlagents \
+    && uv pip install /opt/rcs-src/rcs_duobench
 
 WORKDIR /workspace/robot-control-stack
 

docker/README.md

@@ -23,6 +23,7 @@ Notes:
 - `~/zed_models` is mounted into `/usr/local/zed/resources` to match the direct `docker run` setup.
 - `/dev/dri` is masked inside the container so host Mesa/AMD render nodes do not override the NVIDIA runtime devices.
 - NVIDIA PRIME/GLX environment variables are exported to bias OpenGL/EGL selection toward the NVIDIA stack when using X11 forwarding.
-- Python source changes are picked up from the mounted repo, including `extensions/rcs_zed`.
+- The simulator still bootstraps EGL for offscreen MuJoCo camera rendering; set `RCS_MUJOCO_DISABLE_EGL=1` only if you intentionally want to disable that path.
+- Python source changes are picked up from the mounted repo.
 - If you change C++ code in `rcs` or `rcs_fr3`, rebuild the image.
 - For non-GPU hosts, comment out the GPU-related lines in `docker/compose/dev.yml`.

docker/link-editable-source.sh

@@ -10,26 +10,21 @@ fi
 
 SITE_PACKAGES="$(python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])')"
 
-link_mixed_package() {
+link_compiled_package() {
     src_dir="$1"
     dst_dir="$2"
-    keep_dir_name="${3:-}"
 
     if [ ! -d "$src_dir" ] || [ ! -d "$dst_dir" ]; then
         return
     fi
 
-    # Replace only the Python sources from the mounted repo and keep compiled
-    # artifacts that were installed into site-packages during image build.
-    for path in "$src_dir"/* "$src_dir"/.[!.]* "$src_dir"/..?*; do
-        [ -e "$path" ] || continue
-        name="$(basename "$path")"
-        if [ -n "$keep_dir_name" ] && [ "$name" = "$keep_dir_name" ]; then
-            continue
-        fi
-        rm -rf "$dst_dir/$name"
-        cp -as "$path" "$dst_dir/$name"
-    done
+    tmp_keep="$(mktemp -d)"
+    find "$dst_dir" -maxdepth 1 \( -name '_core*.so' -o -name 'lib*.so*' \) -exec mv {} "$tmp_keep/" \;
+    rm -rf "$dst_dir"
+    mkdir -p "$dst_dir"
+    cp -as "$src_dir/." "$dst_dir/"
+    find "$tmp_keep" -maxdepth 1 -type f -exec mv {} "$dst_dir/" \;
+    rmdir "$tmp_keep"
 }
 
 link_pure_python_package() {
@@ -44,8 +39,9 @@ link_pure_python_package() {
     ln -s "$src_dir" "$dst_dir"
 }
 
-link_mixed_package "$REPO_ROOT/python/rcs" "$SITE_PACKAGES/rcs" "_core"
-link_mixed_package "$REPO_ROOT/extensions/rcs_fr3/src/rcs_fr3" "$SITE_PACKAGES/rcs_fr3" "_core"
+link_compiled_package "$REPO_ROOT/python/rcs" "$SITE_PACKAGES/rcs"
+link_compiled_package "$REPO_ROOT/extensions/rcs_fr3/src/rcs_fr3" "$SITE_PACKAGES/rcs_fr3"
 link_pure_python_package "$REPO_ROOT/extensions/rcs_realsense/src/rcs_realsense" "$SITE_PACKAGES/rcs_realsense"
 link_pure_python_package "$REPO_ROOT/extensions/rcs_robotiq2f85/src/rcs_robotiq2f85" "$SITE_PACKAGES/rcs_robotiq2f85"
 link_pure_python_package "$REPO_ROOT/extensions/rcs_zed/src/rcs_zed" "$SITE_PACKAGES/rcs_zed"
+link_pure_python_package "$REPO_ROOT/vlagents" "$SITE_PACKAGES/vlagents"

docs/apps/index.md

@@ -0,0 +1,22 @@
+# Apps
+
+RCS ships with ready-to-use applications for common operator workflows such as robot teleoperation and remote policy inference.
+
+## Teleoperation
+
+Use the Franka teleoperation app when you want to collect demonstrations or directly control a robot from an operator interface.
+
+- Example guide: [examples/teleop/README.md](../../examples/teleop/README.md)
+- Main script: [examples/teleop/franka.py](../../examples/teleop/franka.py)
+
+The current example focuses on Franka teleoperation with Meta Quest 3 and GELLO-based setups.
+
+## Inference
+
+Use the Franka inference app when you want to run a remote policy server, for example through `vlagents`, against an RCS hardware environment.
+
+- Example guide: [examples/inference/README.md](../../examples/inference/README.md)
+- Main script: [examples/inference/franka.py](../../examples/inference/franka.py)
+- Example config: [examples/inference/franka.json](../../examples/inference/franka.json)
+
+The inference example expects a compatible policy server to already be running and explains the runtime keyboard commands and config fields in its README.

docs/index.md

@@ -31,6 +31,13 @@ getting_started/index
 user_guide/index
 ```
 
+```{toctree}
+:maxdepth: 2
+:caption: Apps
+
+apps/index
+```
+
 ```{toctree}
 :maxdepth: 2
 :caption: API

examples/inference/README.md

@@ -0,0 +1,79 @@
+# Franka Inference Example
+
+This folder contains a hardware-oriented inference example for running a `vlagents` policy server against the Franka duo setup in [franka.py](franka.py) with configuration from [franka.json](franka.json).
+
+## Policy Server
+
+Before starting `franka.py`, make sure a `vlagents` policy server is already running. The example connects to a remote agent with:
+
+- `vlagents_host`
+- `vlagents_port`
+- `vlagents_model`
+
+The policy server setup and supported launch commands are documented in:
+
+- [RobotControlStack/vlagents](https://github.com/RobotControlStack/vlagents)
+- [vlagents/README.md](../../vlagents/README.md)
+
+Typical server startup looks like:
+
+```shell
+python -m vlagents start-server lerobot --port 20000 --host 0.0.0.0 --kwargs '{"policy_name": "act", "checkpoint_path": "<path to pretrained_model>", "n_action_steps": 1}'
+```
+
+For other policies such as `pi05` or `xvla`, use the matching startup command from the `vlagents` README and make sure the values in `franka.json` point at that server.
+
+## Config File
+
+[franka.json](franka.json) is an example config, not a universal default. You should review and usually change these values before running inference:
+
+- `vlagents_host`: Hostname or IP address where the policy server is running.
+- `vlagents_port`: Port exposed by the policy server.
+- `vlagents_model`: Agent id passed to `vlagents`, for example `lerobot`.
+- `instruction`: Natural-language task instruction sent to the policy on reset.
+- `robot_keys`: Robot ordering used to pack observations and unpack actions. The script assumes one 8-value action block per robot in this order: `7` joint values plus `1` gripper value.
+- `jpeg_encoding`: Whether observations are sent to the policy server using JPEG-compressed images.
+- `on_same_machine`: Set this according to whether the policy server runs on the same machine as the control process.
+- `fps`: Control loop target frequency used by the local rate limiter.
+- `record_path`: Output directory used when recording episodes.
+- `n_action_steps`: If `null`, the script requests one action per control step. If set to an integer greater than `0`, the script buffers that many actions from each policy response chunk.
+- `max_rel_mov_joints`: Maximum allowed relative joint movement per step when running in joint control mode.
+- `max_rel_mov_cart`: Maximum allowed relative Cartesian translation and rotation per step when running in Cartesian modes.
+
+The current `franka.py` example also contains hardware-specific constants for robot IPs, camera serials, gripper serials, and frame mappings. Those live in the script itself, so update [franka.py](franka.py) if your hardware setup differs.
+
+## Runtime Keys
+
+When [franka.py](franka.py) is running, it waits for keyboard input on stdin. The active commands are:
+
+- `e`: Start an episode without recording.
+- `r`: Start an episode and begin recording to `record_path`.
+- `s`: Mark the current episode as successful and reset the environment.
+- `q`: Stop the current episode and reset the environment.
+- `o`: Reload `franka.json`, reconnect the `vlagents` client, reset the environment, and clear any buffered actions.
+- `x`: Exit the program.
+
+## Observation And Action Mapping
+
+The script translates RCS observations to the `vlagents` `Obs` format as follows:
+
+- Every camera frame in `obs["frames"]` is converted to RGB and resized to `224x224`.
+- State is built by iterating through `robot_keys` in order and concatenating each robot's `joints` and `gripper` values.
+
+Action decoding is also order-dependent:
+
+- For each robot in `robot_keys`, the script reads `8` values from the policy action vector.
+- Values `0:7` become the robot joint command.
+- Value `7:8` becomes the robot gripper command.
+
+That means `robot_keys` must match the policy's expected robot ordering exactly.
+
+## Running
+
+After the policy server is up and `franka.json` is configured, run:
+
+```shell
+python examples/inference/franka.py
+```
+
+If the policy server is unreachable, the script will keep retrying connection until it becomes available or you exit.

examples/inference/franka.json

@@ -0,0 +1,20 @@
+{
+  "vlagents_host": "localhost",
+  "vlagents_port": 20000,
+  "vlagents_model": "lerobot",
+  "instruction": "use the left arm to place the white cube in the white bowl; use the right arm to place the black cube in the black bowl",
+  "robot_keys": [
+    "left",
+    "right"
+  ],
+  "jpeg_encoding": true,
+  "on_same_machine": false,
+  "fps": 30,
+  "record_path": "inference_recordings_bin_sort_duobench_xvla_bin_sort_real_2026-05-20_23-25-47_040000",
+  "n_action_steps": 30,
+  "max_rel_mov_joints": 0.08726646259971647,
+  "max_rel_mov_cart": [
+    0.5,
+    1.5707963267948966
+  ]
+}