feat(action): add RoboCasa mobile-manipulation action-policy dataset and config - #184
feat(action): add RoboCasa mobile-manipulation action-policy dataset and config#184hyzhou-nv wants to merge 2 commits into
Conversation
|
LGTM |
6db2be7 to
2ec6d37
Compare
Adds RoboCasa to the action-policy path: dataset loader, dataset factory,
Nano post-training config, embodiment registration, closed-loop evaluator,
and the LeRobot v2.1 -> v3.0 dataset converter.
RoboCasa is a mobile-manipulation benchmark, so unlike DROID and LIBERO the
base is not fixed. The loader exposes a 15-D raw action contract that passes
the native base command through unchanged:
[base_motion(4), control_mode(1), eef_pos(3), eef_rot6d(6), gripper(1)]
Observations are the agentview_left | eye_in_hand composite at full
resolution, with EEF proprioception prepended as a clean condition token.
data/generator/action/datasets/ RoboCasaLeRobotDataset and its factory
configs/.../action_policy_robocasa_nano.py
simulation/robocasa/ closed-loop evaluator, next to LIBERO's
scripts/convert_robocasa_to_lerobot_v30.sh
The action server gains support for the `use_state=True` contract the recipe
trains with: the current end-effector pose is prepended to the action sequence
as a clean conditioning frame, making it `chunk_length + 1` long. Without it a
`use_state` checkpoint was served an all-zero `chunk_length` action tensor and
the request's `"state"` field was ignored. Measured on target/atomic at iter
16000, 18 tasks x 20 rollouts, same weights either way: 226/360 without the
state token against 287/360 with it. Requests that omit "state" are unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2ec6d37 to
875e363
Compare
fwd4
left a comment
fwd4
left a comment
There was a problem hiding this comment.
Re-review (requesting changes):
The new RoboCasa path is much cleaner, but I found these runtime-contract issues:
-
[P1] Action-width mismatch in the documented evaluation path. The default recipe uses
use_base_action=Trueandbase_encoding="raw", so the action/state-token width is 15. Whenraw_action_dimis not supplied and stats do not provide it, the server falls back to 7. The documented exported-checkpoint invocation supplies neither, whileclosed_loop_eval.pysends a 15-D padded state token; the first/predictrequest will fail with a width error. Please infer the effective width from the experiment/checkpoint, or require/document--raw-action-dim 15, and make/inforeport the effective value. -
[P1/P2] The domain registry contradicts this recipe.
EMBODIMENT_TO_RAW_ACTION_DIM["robocasa"]is still 10, while this PR’s default is 15 and the loader supports 10/15/20 depending on configuration. Generic registry-driven validation/processing now has an ambiguous or incorrect contract. Please make the width explicit per dataset/config, or align the registry semantics with the supported variants. -
[P2] Broken horizon fallback.
official_horizon()references_OFFICIAL_HORIZON_FALLBACK, but that symbol is not defined in the file. If the RoboCasa import fails, the supposed fallback raisesNameErrorinstead of returning a horizon or a clear error. -
[P2] Silent shard loss. The loader takes only
matches[0]when more than one dated<task>/*/lerobotshard exists, so additional exports are silently ignored. Merge all intended shards or reject ambiguous input.
Please add a small smoke test for the documented server/evaluator handshake with the 15-D state token; this new server/eval path currently has no automated coverage.
…point Address review on NVIDIA#184 and NVIDIA/cosmos#318. The policy server fell back to a width of 7 when --raw-action-dim was omitted and no action stats supplied one. RoboCasa never normalizes through action stats, so the documented evaluation path always took that fallback: the evaluator sent a 15-D state token and every /predict was rejected. Derive the width from the experiment config instead (use_base_action / base_encoding -> 10/15/20, the same mapping as RoboCasaLeRobotDataset.action_dim), keep --raw-action-dim as an override, fail startup when none of the three sources resolves, and report the effective width and its source from /info. Also: * robocasa now sizes its raw action per dataset, so drop it from EMBODIMENT_TO_RAW_ACTION_DIM and list it in _PER_DATASET_ACTION_WIDTH alongside libero and hand_pose, which are per-dataset for the same reason. The registry no longer claims a single 10-D contract the loader contradicts. * official_horizon() referenced an undefined _OFFICIAL_HORIZON_FALLBACK, so the import-failure path raised NameError. Warn and return None; the caller already falls back to --max-steps. Mirroring the horizons here would just go stale, as v1.0.1 rebasing every one of them by 1.5x shows. * the loader kept only the first dated export per task, silently training on less data than was staged. Register all of them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…point Address review on NVIDIA#184 and NVIDIA/cosmos#318. The policy server fell back to a width of 7 when --raw-action-dim was omitted and no action stats supplied one. RoboCasa never normalizes through action stats, so the documented evaluation path always took that fallback: the evaluator sent a 15-D state token and every /predict was rejected. Derive the width from the experiment config instead (use_base_action / base_encoding -> 10/15/20, the same mapping as RoboCasaLeRobotDataset.action_dim), keep --raw-action-dim as an override, fail startup when none of the three sources resolves, and report the effective width and its source from /info. Also: * robocasa now sizes its raw action per dataset, so drop it from EMBODIMENT_TO_RAW_ACTION_DIM and list it in _PER_DATASET_ACTION_WIDTH alongside libero and hand_pose, which are per-dataset for the same reason. The registry no longer claims a single 10-D contract the loader contradicts. * official_horizon() referenced an undefined _OFFICIAL_HORIZON_FALLBACK, so the import-failure path raised NameError. Warn and return None; the caller already falls back to --max-steps. Mirroring the horizons here would just go stale, as v1.0.1 rebasing every one of them by 1.5x shows. * the loader kept only the first dated export per task, silently training on less data than was staged. Register all of them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
8a5be2e to
c9e74fa
Compare
2 participants
Adds RoboCasa support to the action-policy SFT path: a LeRobot dataset loader, the dataset factory, the Nano post-training config, and the embodiment registrations.
RoboCasa is a mobile-manipulation benchmark, so the base is not fixed. The loader exposes a 15-D raw action contract that passes the native base_motion through unchanged:
Observations are the agentview_left | eye_in_hand composite at full resolution, with EEF proprioception as a prepended clean condition token.
All changes are additive; no existing line is modified.