Newton: rigid-body root scale is dropped from the Fabric world matrix, so scaled assets render at the wrong size

[lgulich]

Summary

On the Newton backend, a rigid body whose root prim carries a non-unit xformOp:scale is rendered at its unscaled size. Physics is correct — collision geometry, mass and poses all honour the scale — but the transform written to Fabric for RTX is a pure rigid transform, so the scale is dropped from the world matrix.

An asset authored in centimetres and scaled to metres (scale 0.01) renders 100× too large. On PhysX the same asset renders correctly, so this is Newton-specific.

The visible effect is a giant object filling the frame and occluding the rest of the scene. Anything consuming rendered output — camera observations, depth, tiled rendering, recorded video, perception pipelines — receives that, while the simulation state stays correct.

Affected code

source/isaaclab_newton/isaaclab_newton/physics/newton_manager.py#L61-L75:

@wp.kernel(enable_backward=False)
def _set_fabric_transforms(
    fabric_transforms: wp.fabricarray(dtype=wp.mat44d),
    newton_indices: wp.fabricarray(dtype=wp.uint32),
    newton_body_q: wp.array(ndim=1, dtype=wp.transformf),
):
    """Write Newton body transforms to Fabric world matrices."""
    i = int(wp.tid())
    idx = int(newton_indices[i])
    transform = newton_body_q[idx]
    fabric_transforms[i] = wp.transpose(wp.mat44d(wp.transform_to_matrix(transform)))

newton_body_q holds wp.transformf — position and quaternion only. wp.transform_to_matrix therefore produces a matrix whose upper-left 3×3 is a pure rotation, and writing it into omni:fabric:worldMatrix overwrites whatever scale the prim had.

Newton has no reason to carry scale in body_q (a rigid body's scale does not evolve), so the omission is reasonable per se — the problem is that the kernel writes a world matrix, which is where scale lives, rather than only the pose components.

Notably, the correct scale is still available in Fabric alongside it, as _worldScale. Only the matrix is wrong.

Reproduction

Self-contained: authors a 1 m cube whose rigid-body root is scaled to 0.01, spawns it, steps, and reads back the transform RTX consumes. No external assets.

python repro_fabric_scale.py --physics newton   # WRONG, 100x too large
python repro_fabric_scale.py --physics physx    # OK

import argparse, tempfile
from pathlib import Path

parser = argparse.ArgumentParser()
parser.add_argument("--physics", choices=["newton", "physx"], default="newton")
args = parser.parse_args()

from isaaclab.app import AppLauncher

# enable_cameras matters: without rendering the prim is never registered for
# Fabric transform writeback, and the bug cannot appear.
simulation_app = AppLauncher({"headless": True, "enable_cameras": True}).app

import numpy as np
from pxr import Gf, Usd, UsdGeom, UsdPhysics

import isaaclab.sim as sim_utils
from isaaclab.assets import RigidObject, RigidObjectCfg

SCALE = 0.01
PRIM = "/World/Bin"


def author_asset(path):
    """A 1 m cube under a rigid-body root scaled to 0.01 -> a 0.01 m cube."""
    stage = Usd.Stage.CreateNew(path)
    UsdGeom.SetStageUpAxis(stage, UsdGeom.Tokens.z)
    root = UsdGeom.Xform.Define(stage, "/Bin")
    stage.SetDefaultPrim(root.GetPrim())
    UsdPhysics.RigidBodyAPI.Apply(root.GetPrim())
    UsdPhysics.MassAPI.Apply(root.GetPrim()).CreateMassAttr(1.0)
    root.AddScaleOp().Set(Gf.Vec3f(SCALE, SCALE, SCALE))
    cube = UsdGeom.Cube.Define(stage, "/Bin/Collider")
    cube.CreateSizeAttr(1.0)
    UsdPhysics.CollisionAPI.Apply(cube.GetPrim())
    stage.Save()


with tempfile.TemporaryDirectory() as tmp:
    asset = str(Path(tmp) / "scaled_body.usda")
    author_asset(asset)

    physics = None
    if args.physics == "newton":
        from isaaclab_newton.physics.mjwarp_manager_cfg import MJWarpSolverCfg
        from isaaclab_newton.physics.newton_manager_cfg import NewtonCfg
        physics = NewtonCfg(solver_cfg=MJWarpSolverCfg())

    sim = sim_utils.SimulationContext(sim_utils.SimulationCfg(dt=0.005, physics=physics))
    obj = RigidObject(RigidObjectCfg(
        prim_path=PRIM,
        spawn=sim_utils.UsdFileCfg(usd_path=asset),
        init_state=RigidObjectCfg.InitialStateCfg(pos=(0.0, 0.0, 1.0)),
    ))
    sim.reset()
    for _ in range(20):
        sim.step()
        obj.update(sim.get_physics_dt())

    from usdrt import Usd as RtUsd
    import omni.usd

    rt = RtUsd.Stage.Attach(omni.usd.get_context().get_stage_id())
    prim = rt.GetPrimAtPath(PRIM)
    names = {str(a.GetName()) for a in prim.GetAttributes()}

    print(f"\nbackend: {args.physics}   authored root scale: {SCALE}\n")
    if "_worldScale" in names:
        print(f"  _worldScale               = {prim.GetAttribute('_worldScale').Get()}")
    if "omni:fabric:worldMatrix" in names:
        m = np.asarray(prim.GetAttribute("omni:fabric:worldMatrix").Get(), dtype=float).reshape(4, 4)
        print(f"  worldMatrix implied scale = "
              f"{tuple(round(float(np.linalg.norm(m[i, :3])), 6) for i in range(3))}")
    if "_worldExtent" in names:
        ext = prim.GetAttribute("_worldExtent").Get()
        lo = np.asarray([ext.GetMin()[i] for i in range(3)], dtype=float)
        hi = np.asarray([ext.GetMax()[i] for i in range(3)], dtype=float)
        size = hi - lo
        print(f"  rendered size             = {tuple(round(float(v), 4) for v in size)} m")
        print(f"\n  VERDICT: {'OK' if abs(size[0] - SCALE) < 1e-4 else f'WRONG, {size[0] / SCALE:.0f}x too large'}")

simulation_app.close()

Actual

backend: newton   authored root scale: 0.01

  _worldScale               = (0.01, 0.01, 0.01)
  worldMatrix implied scale = (1.0, 1.0, 1.0)
  rendered size             = (1.0, 1.0, 1.0) m

  VERDICT: WRONG, 100x too large

backend: physx   authored root scale: 0.01

  worldMatrix implied scale = (0.01, 0.01, 0.01)
  rendered size             = (0.01, 0.01, 0.01) m

  VERDICT: OK

Expected

The Newton run should match PhysX: worldMatrix implied scale (0.01, 0.01, 0.01) and rendered size 0.01 m.

Why this is easy to miss

• Physics is unaffected. Collision extents, mass and body poses are all correct, so anything asserting on simulation state passes. Only the render transform is wrong.
• It requires rendering. With enable_cameras=False the prim is never registered for Fabric transform writeback — it has no omni:fabric:worldMatrix attribute at all — and the defect cannot appear. A headless physics-only test will not catch it.
• _worldScale still reads correctly, so inspecting that attribute suggests everything is fine.
• Nothing is logged.

Observed originally on an asset authored in millimetres with root scale 0.007: intended 0.42 x 0.28 x 0.105 m, rendered 60 x 40 x 15 m (143x). It filled the camera frame and occluded the rest of the scene, which presented as a perception failure rather than a transform bug.

Suggested fix

Compose the existing scale into the matrix instead of writing a pure rigid transform. The value is already in Fabric as _worldScale, so the kernel can read it as a second fabricarray input and scale the rotation basis before the write:

    transform = newton_body_q[idx]
    m = wp.mat44d(wp.transform_to_matrix(transform))
    s = fabric_world_scale[i]          # existing per-prim _worldScale
    # scale the rotation basis columns; leave translation intact
    ...
    fabric_transforms[i] = wp.transpose(m)

Alternatively, write the pose components (_worldPosition / _worldOrientation) and let Fabric compose the world matrix, so the authored scale is never overwritten.

Either way the invariant is: a rigid body's authored scale must survive the physics-to-render sync, because the physics engine has no scale to contribute.

Environment

• Isaac Lab release/3.0.0-beta2 (af1bab4); same code in the isaaclab 3.0.0b2 wheel
• newton 1.4.0, warp-lang 1.15.0, isaacsim 6.0.0.1
• Linux x86_64, Python 3.12, RTX 6000 Ada