Skip to content

Fix replay crash on trailing idle action in IK tasks - #6810

Open
rwiltz wants to merge 3 commits into
isaac-sim:developfrom
rwiltz:rwiltz/fix_so101_replay
Open

Fix replay crash on trailing idle action in IK tasks#6810
rwiltz wants to merge 3 commits into
isaac-sim:developfrom
rwiltz:rwiltz/fix_so101_replay

Conversation

@rwiltz

@rwiltz rwiltz commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Description

Replaying demonstrations for an absolute task-space (IK) task crashed after all
episodes had replayed successfully, with a torch.linalg.solve ... input matrix is singular error from the differential IK controller instead of exiting cleanly.

Root cause is a trailing step past the end of the recorded data. In
replay_episodes_loop, env.step(actions) sat outside the has_next_action
check, so once every environment had exhausted its episodes the loop applied the
untouched idle_action one last time before terminating. No task defines
idle_action, so it falls back to torch.zeros(env.action_space.shape) — and for
an absolute-pose action ([pos_xyz, quat_xyzw, gripper]) a zeros action carries a
zero-norm quaternion.

DifferentialIKController.set_command renormalized it as quat / norm(quat),
i.e. 0/0, producing a NaN target orientation. The NaN propagated into the joint
position targets, diverged the articulation, and only surfaced on the next
decimation sub-step as an unrelated "singular matrix" failure — which is why the
traceback points at the solver rather than at the defect.

Fixes:

  • scripts/tools/replay_demos.py: stop before stepping once every environment is
    exhausted, so the replay terminates without applying the idle action or running
    another IK solve.
  • DifferentialIKController.set_command: a degenerate (zero-norm) commanded
    quaternion now holds the current end-effector orientation (identity when none was
    supplied) instead of emitting NaN. Applied per-environment via a branchless
    torch.where, so there is no added host sync on the training hot path.
  • adaptive_dls: a non-finite Jacobian is reported by its actual cause instead of
    the opaque LAPACK singular-matrix / convergence failure. The check runs only on
    the failure path, so the happy path is unchanged.

The controller change is not redundant with the script change: with --num_envs > 1,
an environment that finishes early keeps receiving the zero-quaternion idle action
every step while the others replay, which would diverge that environment and take
down the whole batch.

Reproduced and verified with IsaacContrib-Stack-Cube-SO101-IK-Abs-v0. On the
unfixed code the QA command produces a byte-for-byte identical traceback; with the
fix it prints Finished replaying 1 episode. and exits 0.

Note (out of scope): idle_action defaulting to zeros is unsound for any
absolute-pose task, since a zero quaternion is never a valid orientation. Seeding
the quaternion to identity would be a broader change affecting every task using the
replay script, so it is left for a separate PR.

Fixes # (issue)

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump extension.toml — CI handles that)
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

replay_demos.py stepped the environment one extra time after every
environment had exhausted its episodes: the step call sat outside the
has_next_action check, so the loop applied the untouched idle action
before terminating. No task defines idle_action, so it falls back to
zeros -- and for absolute task-space tasks a zeros action carries a
zero-norm quaternion.

DifferentialIKController.set_command renormalized that command as
quat / norm(quat), producing NaN. The NaN reached the joint position
targets, diverged the articulation, and surfaced on the next decimation
sub-step as an unrelated "torch.linalg.solve: the input matrix is
singular" error -- after all demonstrations had replayed successfully.

Stop the replay loop before the trailing step, and harden the
controller so a degenerate command holds the current end-effector
orientation instead of emitting NaN. The controller change is not
redundant: with --num_envs > 1 an environment that finishes early keeps
receiving the zero-quaternion idle action while the others replay, so
it would diverge that environment and take down the whole batch.

Also report a non-finite Jacobian in adaptive_dls by its actual cause
rather than as an opaque LAPACK singular-matrix failure, which is what
made this crash point away from the real defect.

Reported by QA replaying IsaacContrib-Stack-Cube-SO101-IK-Abs-v0.
@rwiltz
rwiltz requested a review from a team July 30, 2026 20:38
@github-actions github-actions Bot added bug Something isn't working isaac-lab Related to Isaac Lab team labels Jul 30, 2026
Comment thread source/isaaclab/isaaclab/controllers/differential_ik.py Outdated
@greptile-apps

greptile-apps Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR makes demonstration replay stop before its trailing idle step and hardens differential IK against degenerate quaternion commands and non-finite Jacobians.

  • Holds the current end-effector orientation when an absolute pose contains a non-normalizable quaternion.
  • Adds explicit adaptive-DLS diagnostics for non-finite Jacobians.
  • Adds regression coverage and a changelog entry.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
scripts/tools/replay_demos.py Stops replay before stepping after every environment has exhausted its recorded actions.
source/isaaclab/isaaclab/controllers/differential_ik.py Adds per-environment quaternion fallback handling and explicit non-finite Jacobian diagnostics.
source/isaaclab/test/controllers/test_differential_ik_features.py Covers degenerate and tiny quaternion behavior plus adaptive-DLS Jacobian validation.

Reviews (2): Last reviewed commit: "Narrow the degenerate quaternion test to..." | Re-trigger Greptile

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac Lab Review Bot

The replay loop now avoids the trailing idle step, and the IK controller prevents zero-quaternion commands from producing NaNs. Two API details need correction before merge: the tolerance changes behavior for tiny nonzero quaternions, and the new absolute-pose fallback semantics are absent from the public contract.

  • Design and architecture: The replay termination change preserves episode bookkeeping while removing the erroneous final step. The per-environment orientation fallback is appropriately localized, but its threshold should not broaden the fix beyond genuinely degenerate commands without an intentional compatibility decision.
  • API: set_command previously normalized every nonzero quaternion, whereas quaternions below 1e-6 now select a fallback orientation. Additionally, ee_quat now influences degenerate absolute-pose commands despite the docstring describing it as relevant only to position and relative-pose modes. Narrow the degeneracy condition or document the compatibility change, and document the new fallback behavior.
  • Implementation: The adaptive-DLS exception handling preserves the original exception for finite Jacobians and provides a clearer error for non-finite inputs. Tests cover zero-quaternion identity, current-orientation, and per-environment fallback paths, but do not cover compatibility for tiny nonzero quaternions. The required Isaac Lab changelog fragment is present.

Minor fixes needed. Posted 2 actionable findings inline.

Automated review; human maintainers own approval decisions.

Comment thread source/isaaclab/isaaclab/controllers/differential_ik.py Outdated
Comment thread source/isaaclab/isaaclab/controllers/differential_ik.py
rwiltz added 2 commits July 30, 2026 16:47
The non-finite Jacobian diagnostic was reached only when svdvals or
solve raised LinAlgError, so it depended on backend error-reporting
behavior. Backends that propagate NaN instead of raising bypassed it
and let non-finite joint position targets reach the articulation. An
Inf-valued Jacobian took that path even on CPU.

Check finiteness up front instead, which covers both behaviors and
drops the try/except. The check costs one device sync (~32us, flat in
batch size); adaptive_dls is used only by the SO-101 teleop and replay
task at frame rate, where that is immaterial.

Extend the regression test over NaN, +Inf and -Inf, and add a case
asserting a well-conditioned Jacobian is unaffected by the guard.
The 1e-6 norm cutoff was broader than the crash it fixes. A quaternion
such as [0, 0, 0, 1e-7] normalizes cleanly to identity and did so
before this branch, but the cutoff silently replaced it with the
fallback orientation -- a behavior change beyond the zero-norm fix.

Decide degeneracy by whether the normalization produced a finite
result instead. This drops the magic constant and is exact at both
ends: a zero quaternion (0/0) and a norm that underflows to zero
(Inf) are still caught, while anything that normalizes keeps its
previous meaning. A plain norm > 0 test would not cover underflow.

Document the fallback in set_command, including ee_quat's role for
absolute pose commands, which the docstring previously described as
relevant only to position and relative-pose modes.
@rwiltz

rwiltz commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

@greptile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant