Collect successful OSMO runs after individual failures

[cvolkcvolk]

A parallel experiment starts N independent runner tasks and finishes with one collector.

Architecture

runner 0 ----\
runner 1 -----+--> collector --> aggregated index.html
   ... -------+
runner N ----/

Each runner invokes the Arena experiment normally. When that process finishes, the runner writes experiment_runner_result.json with either completed or failed plus the original process exit code. These values use the existing RunStatus enum.

If Arena exits with an error, the wrapper records the failure and then exits successfully from the perspective of OSMO. This separation is deliberate: OSMO can schedule the downstream collector, while the application failure remains available in the result file.

Collector behavior

The collector supports exactly two runner results:

Runner A: completed, exit 0  --> copy all artifacts
Runner B: failed, exit 1     --> copy only the result JSON
                                |
                                +--> build aggregated index.html

• For completed runs, it copies the complete run output into the aggregated experiment.
• For failed runs, it preserves the result file but excludes partial episode results and videos.
• If every run failed, it still produces an empty index.html report.

For example, one completed run and one failed run produce:

experiment-output/
|-- run-a/
|   |-- experiment_runner_result.json
|   |-- episode_results_rebuild0.jsonl
|   `-- videos...
|-- run-b/
|   `-- experiment_runner_result.json
`-- index.html

The report generator only reads episode-result JSONL files and videos. The failed run therefore remains auditable through its result file but does not appear in the report.

A missing, malformed, unknown, or inconsistent result violates the runner/collector contract and fails collection. A completed result without its expected run directory also fails. These cases indicate a bug, incompatible version, or storage problem and should not silently produce an incomplete report.

Why

Previously, one failed runner could give the collector FAILED_UPSTREAM. The collector would not run, even when the other runners had produced valid results. This change lets one submission still produce a report containing every successful run.

Smoke experiment

osmo_failure_collection_smoke_experiment.yaml provides an end-to-end check with four one-episode Pi0 runs:

• Two valid runs should complete and appear in the report.
• One run uses an unknown pickup asset and should fail during environment construction.
• One run uses an unknown policy adapter and should fail after creating partial output.

The collector should finish, retain failure result files, discard partial failed-run artifacts, and build a report containing only the two valid runs.

Known limitation

This handles application failures after the runner entry script has started. OSMO-level failures such as an image download error, eviction, node loss, task cancellation, or failure to publish the result file can still prevent the collector from running.

Validation

• 29 focused OSMO workflow and collector tests pass.
• Collector cases cover mixed success and failure, all failed, partial failed artifacts, and contract violations.
• The smoke experiment composes and renders as four runner groups plus one collector in dry-run mode.
• All repository pre-commit checks pass.