Add optional rerun debug view of placement validation by xyao-nv · Pull Request #984 · isaac-sim/IsaacLab-Arena

xyao-nv · 2026-07-31T06:03:00Z

Summary

Rerun debug view of placement validation

Detailed description

Adds an opt-in Rerun view that draws every candidate layout using bbox
When in curobo image and reachability validation is enabled, the cuRobo reachability check draws the robot base, the grasps it solved, reachable/unreachable per target, and IK error.
Turned on from config, not a CLI flag: placement_validators.debug_visualize (and/ordebug_visualize_rrd_path for headless runs) in an env graph YAML, or ObjectPlacerParams(debug_visualize=True).
Off by default and off in every shipped env. Include instructions on how to enable it.

Example viz

In base docker (during solver validation):
https://github.com/user-attachments/assets/d73882ee-ab13-4309-aa7b-f19de10ad13c

In curobo docker (where reachability is enabled)
https://github.com/user-attachments/assets/55d99e37-6eff-4558-8444-e0a54f251f04

Placement solving is sim-free, so its candidate layouts have been hard to inspect without starting Isaac Sim. Opting in via ObjectPlacerParams now streams every candidate to a Rerun viewer (or an .rrd, for headless runs). - Add PlacementRerunVisualizer: one process-wide view that draws each candidate's object boxes and the checks that accepted or rejected it, one frame per candidate on a `candidate` timeline. - Turn it on with ObjectPlacerParams.debug_visualize / debug_visualize_rrd_path; ObjectPlacer builds the view before its validators so a check can add its own layer to it. - Let the cuRobo reachability check layer on what only it knows: the robot base frame, the top-down grasps it solved, per-target reachable/unreachable verdicts, and its IK error scalars. - Keep frames aligned across checks: expensive checks only see the candidates that passed the cheap ones, so the placer tells the view which candidates the running check was handed. Signed-off-by: Xinjie Yao <xyao@nvidia.com>

- Add debug_visualize / debug_visualize_rrd_path to PlacementValidatorSpec so an env's YAML can ask for the Rerun view without editing Python. - Forward both through build_checks_for_placer_params into ObjectPlacerParams. - Turn the view on in the butter_raisin_box scene as a worked example. - Cover the default-off and forwarding paths in the graph-spec tests. Signed-off-by: Xinjie Yao <xyao@nvidia.com>

- Spawn the Rerun viewer under `setpriv --pdeathsig TERM` instead of `rr.spawn()`, so the kernel closes the window when the run exits. - Isaac Sim's `SimulationApp.close()` ends in `os._exit()`, so no `atexit` hook can do this. - Stops a stale viewer from holding port 9876, which silently made the next run's spawn a no-op. - Wait for the viewer to serve before logging, replacing the readiness wait `rr.spawn()` did. Signed-off-by: Xinjie Yao <xyao@nvidia.com>

- Take `debug_visualize: true` out of the shipped butter/raisin scene, where it spawned a viewer window on every build, and move the worked example to a test-data graph YAML. - Guard that with a test asserting no versioned env under `isaaclab_arena_environments` asks for it. - Track which candidates each check was run on, so the view no longer draws an expensive check as rejecting a layout it skipped; the summary count now derives from the same record. - Warn when the viewer port is already served, and confirm the spawned viewer is still alive once it answers, so a run cannot silently log into somebody else's window. - Fall back to killing a viewer that ignores SIGTERM instead of raising out of close(). Signed-off-by: Xinjie Yao <xyao@nvidia.com>

- Acceptance now follows the placer, which gates layouts on required_checks alone. - A failure the placer does not gate on reads as accepted, naming the advisory check. - Pull the wording out into summarize_candidate_verdict() so the rule is unit-testable. Signed-off-by: Xinjie Yao <xyao@nvidia.com>

- Show the env graph YAML and Python routes in the visualizer module, pointing at the worked example. - Say the same from the reachability check, whose layer only appears once that view is on. - Drop ACCEPTED_COLOR / REJECTED_COLOR, which nothing has ever drawn with. Signed-off-by: Xinjie Yao <xyao@nvidia.com>

alexmillane

Thanks for adding this functionality! Looks pretty good.

See suggestions below!

alexmillane · 2026-07-31T10:00:51Z

        required_checks=set(required_checks) if required_checks is not None else None,
        solver_params=RelationSolverParams(verbose=False, save_position_history=False),
+        debug_visualize=placement_validators is not None and placement_validators.debug_visualize,
+        debug_visualize_rrd_path=(


is there some way to avoid the opaque abbreviation rrd?

Ah I see below that rrd is actually a file type. Suggestion still to do debug_visualize_output_path

alexmillane · 2026-07-31T11:10:32Z

+    debug_visualize_rrd_path: str | None = Field(
+        default=None,
+        description=(
+            "Path to record the debug visualization to as a Rerun .rrd file, for headless runs. Enables "
+            "the visualization on its own; combine with debug_visualize to both record and watch live."
+        ),
+    )


Cool feature!

alexmillane · 2026-07-31T11:11:38Z

        self.params = params or ObjectPlacerParams()
        self._solver = RelationSolver(params=self.params.solver_params)
+        # Populated before the validators are built so a check can add its own layer to the same view.
+        self.params.debug_visualizer = get_or_create_placement_visualizer(self.params)


Suggestion that we only keep params in the params struct? I.e. no objects that implement run-time functionality.

alexmillane · 2026-07-31T11:12:54Z

+        # Per-check candidate indices that check was actually run on; expensive checks skip candidates.
+        evaluated_slots_by_check: dict[str, list[int]] = {}
        layout_pass_verdicts_by_check: dict[str, list[bool]] = {}
+        # Layouts are drawn before the checks run so a check's own layer lands on top of its candidate.


"so a check's own layer lands on top of its candidate."

Not sure I understand that. What's a check's "layer"?

alexmillane · 2026-07-31T11:16:16Z

+    def _log_candidate_layouts(
+        self,
+        positions: list[dict[PlaceableAsset, tuple[float, float, float]]],
+        orientations: list[dict[PlaceableAsset, float]],
+        bboxes: list[dict[PlaceableAsset, AxisAlignedBoundingBox]],
+    ) -> list[int] | None:
+        """Draw every candidate of this batch in the Rerun debug view, returning their timeline indices.
+
+        None when the debug view is off, which is the default.
+        """
+        visualizer = self.params.debug_visualizer
+        if visualizer is None:
+            return None
+        candidate_indices = visualizer.next_batch_indices(len(positions))
+        for slot, candidate_index in enumerate(candidate_indices):
+            anchors = set(get_anchor_objects(list(positions[slot])))
+            visualizer.log_layout(candidate_index, positions[slot], orientations[slot], bboxes[slot], anchors)
+        return candidate_indices


Can we move this into the visualizer? Am I correct that this is a function that wraps visualizer.log_layout to accept a list of candidates effectively.

This file is already quite long and complex. If we can isolate the visualization stuff in the visualizer, that will help keep things as small as possible.

alexmillane · 2026-07-31T11:17:48Z

+    debug_visualize_rrd_path: str | None = None
+    """Path to record the debug visualization to as a Rerun ``.rrd`` file, for headless runs.
+
+    Enables the visualization on its own; combine with ``debug_visualize`` to both record and watch live."""


Enables the visualization on its own? Does this mean that this also starts the live view?

Consider rewording.

alexmillane · 2026-07-31T11:18:06Z

+"""Rerun debug view of build-time placement validation, sim-free (no SimApp).
+
+Rerun's viewer is a separate process fed by the logging SDK, so nothing here touches Isaac Sim -- the
+window comes up while layouts are being solved, before any simulation exists.
+
+Turn it on from an env graph YAML (worked example:
+``isaaclab_arena/tests/test_data/placement_debug_view_env_graph.yaml``)::
+
+    placement_validators:
+      debug_visualize: true                          # spawn a viewer window; needs a reachable display
+      debug_visualize_rrd_path: /tmp/placement.rrd   # and/or record, for headless runs
+
+or in Python with ``ObjectPlacerParams(debug_visualize=True)``. Either field alone enables the view.
+Shipped envs leave it off, since it spawns a window on every build; enable it while debugging a scene
+whose layouts look wrong, then take it back out.
+
+Every candidate layout is one frame of the ``candidate`` timeline, so scrubbing it shows what was
+solved and which checks rejected it. Checks that know more about a candidate than its boxes add their
+own layer under ``world/robot`` (see the cuRobo reachability check).
+
+The spawned window belongs to the run: it comes up during placement, stays up for the rest of the
+run, and dies with the process that spawned it. Record to an ``.rrd`` to inspect layouts afterwards.
+"""


Suggestion to shorten.

alexmillane · 2026-07-31T11:19:41Z

        self._base_quat_xyzw = base_pose.rotation_xyzw
        # Guards the zero-target warning so it fires once per validator, not once per candidate layout.
        self._warned_no_targets = False
+        self._visualizer = params.debug_visualizer


Suggestion to keep functional objects out of parameter structs.

alexmillane · 2026-07-31T11:23:42Z

        self,
        positions: dict[ObjectBase, tuple[float, float, float]],
        orientations: dict[ObjectBase, float],
+        batch_slot: int,


What is a batch_slot.

This is diffcult to detemine what it is from the variable name.

I would guess that slot is coming from rerun? This is just the index of the solution among the batch right? Suggestion to use batch_idx, solution_idx.

The slot terminology says what this variable is used for inside the function, not what it actually is. In general, prefer describing what something is, rather than what you intend to use it for.

alexmillane · 2026-07-31T11:25:09Z

+"""The reachability check's layer of the placement Rerun debug view, sim-free (no SimApp).
+
+Core placement already draws each candidate layout's boxes (see
+``isaaclab_arena.relations.placement_visualizer``); this adds what only the IK check knows -- where the
+robot stands, the top-down grasps it solved, and whether each one was reachable. Everything is logged
+against the same candidate frame, so the two layers compose.
+"""


Suggestion to limit docstrings to what this function/module does. This docstring makes the classic agent mistake of describing what is happing elsewhere in the program.

xyao-nv added 7 commits July 29, 2026 17:20

enbale ik

bbfac37

xyao-nv changed the title ~~Xyao/feature/placement rerun debug view~~ Add optional rerun debug view of placement validation Jul 31, 2026

alexmillane reviewed Jul 31, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add optional rerun debug view of placement validation - #984

Add optional rerun debug view of placement validation#984
xyao-nv wants to merge 7 commits into
mainfrom
xyao/feature/placement-rerun-debug-view

xyao-nv commented Jul 31, 2026 •

edited

Loading

Uh oh!

alexmillane left a comment

Uh oh!

alexmillane Jul 31, 2026

Uh oh!

alexmillane Jul 31, 2026

Uh oh!

alexmillane Jul 31, 2026

Uh oh!

alexmillane Jul 31, 2026

Uh oh!

alexmillane Jul 31, 2026

Uh oh!

alexmillane Jul 31, 2026

Uh oh!

alexmillane Jul 31, 2026

Uh oh!

alexmillane Jul 31, 2026

Uh oh!

alexmillane Jul 31, 2026

Uh oh!

alexmillane Jul 31, 2026

Uh oh!

alexmillane Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

xyao-nv commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Detailed description

Example viz

Uh oh!

alexmillane left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

xyao-nv commented Jul 31, 2026 •

edited

Loading