Add LeRobot LIBERO policy rollout#2667
Draft
TomCC7 wants to merge 12 commits into
Draft
Conversation
❌ 18 Tests Failed:
View the top 3 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
TomCC7
force-pushed
the
cc/frontier_/policy-eval
branch
from
6567ead to
84d6ab2
Compare
TomCC7
commented
Jul 1, 2026
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class RobotLearningSample: |
Member
Author
There was a problem hiding this comment.
This is a bad name. if it's an input sample it should be named better. a bunch of ids that's not really used within module is added. also episode_id is a benchmark specific thing, should not be here at all. And, what's the performance of using lcm to transmit a large dictionary of images? is there a way to do it without serialization?
Member
Author
There was a problem hiding this comment.
Partially addressed in 072b32f: renamed RobotLearningSample to RobotPolicyObservation, removed benchmark-specific ids and top-level task from the policy observation, and keep language in metadata/observations for the contract. I left the image dictionary transport/performance question for a separate design pass rather than mixing transport redesign into this cleanup.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
DimOS had LIBERO runtime plumbing for scripted motor-surface demos, but no policy rollout path that exercised a real LeRobot policy through DimOS-owned inference, contract conversion, native runtime actions, and benchmark evaluation artifacts.
The official LeRobot LIBERO checkpoints use a native relative end-effector delta + gripper action surface, not the existing joint-position motor frame path. That left the policy benchmark blocked on overloaded motor semantics, missing rollout ownership boundaries, and manual environment setup.
Solution
StepRequestaction unions.RobotPolicyModulePolicyBackendLeRobotBackendVlaJepaLiberoRobotContractBenchmarkPolicyEvalRunnerlibero_objectmatrixsuccess_rate > 0.50gateVerification
uv run ruff check scripts/benchmarks/demo_lerobot_libero_policy_rollout.py dimos/robot_learning/policy_rollout packages/dimos-libero-pro-sidecar/src/dimos_libero_pro_sidecar/server.py packages/dimos-runtime-protocol/src/dimos_runtime_protocol packages/dimos-runtime-protocol/tests/test_models.py dimos/benchmark/runtime/test_libero_pro_sidecar_profile.pyuv run pytest packages/dimos-runtime-protocol/tests/test_models.py dimos/benchmark/runtime/test_libero_pro_sidecar_profile.py dimos/benchmark/runtime/test_sidecar_import_boundaries.py dimos/robot_learning/policy_rollout/test_robot_policy_module.py dimos/robot_learning/policy_rollout/test_evaluation.py dimos/robot_learning/policy_rollout/test_lerobot_backend.py dimos/robot_learning/policy_rollout/test_vla_jepa_libero_contract.py -qopenspec validate --allsuccess_rate=0.900, gate threshold> 0.50.