Add Sensor PPISP controller support for camera post-processing (#5947)

[moennen]

Adds PPISP PPISPAuto controller support to IsaacLab camera post-processing.

This change parses embedded PPISP controller weights from USD/SPG-authored controller CUDA sources, stores them in PpispCfg, and evaluates the controller in Warp before applying PPISP. The controller predicts per-camera exposure and color-latent parameters, while existing static PPISP config continues to provide responsivity, vignetting, and CRF parameters.

The implementation keeps the original pure-Warp controller path as a correctness reference and adds a native wp.func_native CUDA-snippet backend for high-env-count tiled camera workloads. The default controller backend uses an automatic heuristic: Warp for single/large images, native CUDA snippets for many small camera tiles.

Also adds tests for:

• USD PPISPAuto controller parsing

• controller weight layout and NumPy equivalence

• native-vs-Warp controller equivalence

• PPISP camera sensor coverage across Isaac RTX, OVRTX, and Newton paths

  • opt-in PPISP pipeline performance sweep

  Dependencies:

  • Existing Warp dependency
  • CUDA required for the native controller backend and performance test

  Fixes #

  Type of change

  • New feature (non-breaking change which adds functionality)

  Screenshots

Not applicable. This is camera post-processing and renderer/backend integration behavior.

Checklist

• I have read and understood the contribution guidelines
• I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have updated the changelog and the corresponding version in the extension's config/extension.toml file
• I have added my name to the CONTRIBUTORS.md or my name already exists there

Description

Important

Confirm the pull request base before submitting. Target develop for all
contributions. The release/3.0.0-beta2 branch is a frozen stable landing
snapshot and is not used for ongoing maintenance.

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context.
List any dependencies that are required for this change.

Fixes # (issue)

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (existing functionality will not work without user modification)
• Documentation update

Screenshots

Please attach before and after screenshots of the change if applicable.

Checklist

• I have read and understood the contribution guidelines
• I have run the pre-commit checks with ./isaaclab.sh --format
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump extension.toml — CI handles that)
• I have added my name to the CONTRIBUTORS.md or my name already exists there

[greptile-apps]

Greptile Summary

Adds camera-authored PPISP controller support across the camera renderer backends.

• Parses static PPISP parameters and embedded controller weights from USD camera attributes.
• Evaluates per-camera exposure and color parameters through Warp-native CUDA kernels before the PPISP image pass.
• Integrates the controller pipeline with Isaac RTX, OVRTX, and Newton renderers.
• Adds controller parsing, numerical-equivalence, renderer-integration, and performance tests.

Confidence Score: 4/5

The shader-discovery compatibility regression should be fixed before merging because existing PPISP-authored USD assets can silently render without their calibrated post-processing.

The new resolver replaces the existing RenderProduct/PPISP-shader path with camera-attribute-only discovery, so previously supported static assets resolve to no ISP configuration rather than continuing through the unchanged static pipeline; the sentinel documentation also remains aligned with the removed lookup behavior.

Files Needing Attention: source/isaaclab_ppisp/isaaclab_ppisp/cfg.py

Important Files Changed

Filename	Overview
source/isaaclab_ppisp/isaaclab_ppisp/cfg.py	Adds camera-attribute and controller-weight parsing, but drops compatibility with shader-authored PPISP assets.
source/isaaclab_ppisp/isaaclab_ppisp/kernels.py	Adds native CUDA controller feature extraction, MLP inference, and controller-parameter image processing.
source/isaaclab_ppisp/isaaclab_ppisp/pipeline.py	Adds cached controller weights and scratch buffers and dispatches controller inference before PPISP.
source/isaaclab_physx/isaaclab_physx/renderers/isaac_rtx_renderer.py	Resolves camera-authored PPISP configuration and constructs the updated pipeline.
source/isaaclab_ov/isaaclab_ov/renderers/ovrtx_renderer.py	Updates OVRTX camera preparation for camera-attribute discovery and exposure overrides.
source/isaaclab_newton/isaaclab_newton/renderers/newton_warp_renderer.py	Enables PPISP resolution even when Newton has no matched camera prim path.

Sequence Diagram

sequenceDiagram
    participant Camera as Camera sensor
    participant Resolver as PPISP config resolver
    participant Renderer as Renderer backend
    participant Controller as PPISP controller
    participant ImagePass as PPISP image pass
    Camera->>Resolver: Resolve isp_cfg from USD camera
    Resolver-->>Renderer: Static inputs + optional weights
    Renderer->>Controller: HDR image and prior exposure
    Controller-->>Renderer: Per-camera exposure and color latents
    Renderer->>ImagePass: HDR, predicted parameters, static coefficients
    ImagePass-->>Camera: LDR RGB/RGBA

Loading

_{Reviews (1): Last reviewed commit: "Add Sensor PPISP controller support for ..." | Re-trigger Greptile}

[isaaclab-review-bot]

Isaac Lab Review Bot

The PPISP controller architecture is coherent, but the PR removes or renames multiple public shader-based APIs without the required deprecation period. It also directly edits the generated changelog and leaves the public pipeline documentation inconsistent with the new per-camera controller behavior.

• Design and architecture: The two-pass design—predicting per-camera exposure and color latents before applying shared responsivity, vignetting, and CRF parameters—is internally consistent. Controller weight offsets are validated against the expected layout, and scratch buffers are cached by device and image shape. However, the corresponding public class documentation still incorrectly states that per-view ISP is unsupported.
• API: The migration from shader-based discovery to camera-authored attributes is implemented as a breaking replacement rather than an additive transition. PPISP_SHADER_NAME and ppisp_cfg_from_usd_shader are removed, PpispCfg.shader_prim_path is renamed, ppisp_cfg_from_usd_stage changes path semantics, and PpispPipeline drops its stage argument. These public surfaces require deprecated compatibility shims and migration guidance before removal under repository policy.
• Implementation: The renderer resolution paths, controller buffer caching, offset checks, and static/controller equivalence tests are consistent with the implementation. Before merge, the direct edit to source/isaaclab_ppisp/docs/CHANGELOG.rst should be reverted in favor of the changelog fragment, and the PpispPipeline class docstring should distinguish shared static coefficients from per-camera controller outputs.

Significant concerns. Posted 3 actionable findings inline.

Automated review; human maintainers own approval decisions.