[DEMO] tomllib import env regression (perf-gate HARD_FAILURE)

[Neil4561]

Demo PR (not for merge)

HARD_FAILURE case for the performance smoke-test demo.

Change: simulates the environment-regression class where benchmark startup breaks before any FPS data is produced.

Reference case: this is modeled after commit a337c0b, which added import tomllib to the install CLI. tomllib is only available in the Python standard library starting in Python 3.11, so a Python 3.10 CI image would fail during import before any benchmark could run.

Explanation: this mirrors the real environment-regression failure mode we discussed. The benchmark crashes during import/startup, before it can initialize the environment or enter the simulation loop. Since there is no usable FPS series, the gate should report HARD_FAILURE for each affected cell instead of treating it like a normal slowdown.

The current perf image uses Python 3.11, so this demo reproduces the same class of failure deterministically with an explicit import failure guard. The point of the demo is the verdict behavior: startup/import failures are separated from performance regressions.

Matrix: Isaac-Cartpole-Direct + Isaac-Factory-GearMesh-Direct-v0 (physx) + Isaac-Velocity-Flat-G1-v0.

Trigger via mirror to pull-request/<N> or workflow_dispatch.

[github-actions]

Performance Smoke Results

Baseline read SHA: e87de2efea41

Run context

• GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition
• Runtime: N/A
• Threshold policy: not applicable because no usable FPS was produced for baseline comparison.

How to read this table

• PASS: no meaningful slowdown was detected.
• WARN: suspicious or uncertain result, such as insufficient baselines, retry-only success, or a slowdown in the WARN band.
• BLOCK: a blocking-level regression signal was detected. In this POC, BLOCK is advisory (it flags a regression) and only fails the check when the gate is explicitly configured as blocking.
• HARD_FAILURE: the benchmark did not produce usable FPS data, for example an import/init/runtime failure or config mismatch.
• Delta: change in FPS versus the baseline. + means faster than baseline (speedup); - means slower than baseline (slowdown).
• Threshold policy: explains which comparison rule produced the verdict. rolling_window means each row uses its own compatible baseline median/MAD window, not one shared FPS cutoff.
• Noise: baseline MAD as a percent of baseline FPS. Higher noise means the task/backend naturally varies more from run to run.
• Phase: the stage a HARD_FAILURE happened in (e.g. import, init, runtime); blank when the benchmark ran to completion.
• Retried: yes if the cell only passed after an automatic re-run.

Task	Backend	Verdict	FPS	Baseline	Delta (+ faster / - slower)	Noise	Samples	Phase	Notes	Retried
Isaac-Cartpole-Direct	newton	HARD_FAILURE	N/A	N/A	N/A	N/A	0	import		yes
Isaac-Cartpole-Direct	physx	HARD_FAILURE	N/A	N/A	N/A	N/A	0	import		yes
Isaac-Factory-GearMesh-Direct-v0	physx	HARD_FAILURE	N/A	N/A	N/A	N/A	0	import		yes
Isaac-Velocity-Flat-G1-v0	newton	HARD_FAILURE	N/A	N/A	N/A	N/A	0	import		yes
Isaac-Velocity-Flat-G1-v0	physx	HARD_FAILURE	N/A	N/A	N/A	N/A	0	import		yes

Updated for 01b24bf · run