Sleep workload never checks whether the command actually succeeded; always reports PASSED

[shreyaskommuri]

Sleep workload never checks whether the command actually succeeded; always reports PASSED

Describe the Bug

SleepTestDefinition never overrides was_run_successful(), so it inherits
the base TestDefinition default, which unconditionally reports success:

# src/cloudai/models/workload.py
def was_run_successful(self, tr: TestRun) -> JobStatusResult:
    return JobStatusResult(is_successful=True)

Every other workload I checked overrides this with a real check —
fio.py, vllm.py, nccl.py, nixl_ep.py, dynamo_mocker.py,
nemo_run.py, and sglang.py all implement was_run_successful().
src/cloudai/workloads/sleep/sleep.py does not. As a result, a Sleep test
is reported PASSED regardless of whether the underlying command actually
ran, produced output, or exited non-zero.

This came up while validating CloudAI through a small wrapper project,
CloudAI Autotune.
Autotune calls the current CloudAI CLI as its execution layer; I was running
a real (non-dry-run) cloudai run through Autotune to check whether a
genuinely-failing command would be surfaced correctly, and it wasn't.

Because was_run_successful lives on the workload's TestDefinition
rather than on any system-specific class, this isn't a standalone-only
issue — it affects Sleep identically on every system it supports (Slurm,
Kubernetes, Standalone, per the support matrix in the README).

Steps to Reproduce

Version/context where observed: local CloudAI checkout based on
origin/main (commit 28c317c9).

Create a Sleep test config with a command that will genuinely fail (an
invalid seconds value produces sleep -1, which fails immediately with
sleep: illegal option -- 1 on this system):

# test/sleep_bad.toml
name = "sleep_bad"
description = "sleep test"
test_template_name = "Sleep"

[cmd_args]
seconds = -1

# test_scenario/sleep_bad_scenario.toml
name = "sleep-bad-scenario"

[[Tests]]
id = "Tests.sleepbad"
test_name = "sleep_bad"
time_limit = "00:01:00"

Run it for real (not dry-run) through Autotune, which just shells out to the
cloudai CLI:

autotune run test_scenario/sleep_bad_scenario.toml \
  --cloudai-bin <path-to-cloudai> \
  --system-config <standalone-system-config> \
  --tests-dir test

Or directly against cloudai run:

cloudai run \
  --test-scenario test_scenario/sleep_bad_scenario.toml \
  --system-config <standalone-system-config> \
  --tests-dir test \
  --output-dir <output-dir>

Output:

[INFO] Executing command for test Tests.sleepbad: sleep -1
[INFO] Job completed: Tests.sleepbad (iteration 1 of 1)
[INFO] Scenario results
╔════════════════╤════════╤════════════════════════════════════════════════════╗
║ Case           │ Status │ Details                                            ║
╟────────────────┼────────┼────────────────────────────────────────────────────╢
║ Tests.sleepbad │ PASSED │ <output-dir>/sleep-bad-scenario_.../Tests.sleepbad/0║
╚════════════════╧════════╧════════════════════════════════════════════════════╝
[INFO] All jobs are complete.

Process exits 0. No stdout.txt/stderr.txt were even captured for the
test run (only test-run.toml exists in the output directory), confirming
the underlying sleep -1 never ran to a real completion — yet the scenario
still reports PASSED.

Running the identical command directly confirms it genuinely fails:

$ sleep -1; echo "exit code: $?"
sleep: illegal option -- 1
usage: sleep number[unit] [...]
exit code: 1

Expected Behavior

A Sleep test whose underlying command fails to execute (non-zero exit,
missing output, killed process, etc.) should be reported FAILED, not
PASSED.

Sleep is CloudAI's own documented lightweight sanity-check workload —
already described in issue #920/PR #921 as "the lightweight sanity-check
workload people can use to validate standalone CloudAI wiring without GPUs
or heavyweight serving dependencies." A workload whose entire purpose is
validating that CloudAI's wiring works should be able to detect that the
wiring is broken; right now it structurally cannot, on any system, for any
reason.

One possible fix is to implement was_run_successful() on
SleepTestDefinition, checking the exit status of the executed command (or,
at minimum, verifying stdout.txt/stderr.txt were produced with a
zero exit code), matching the pattern every other workload already follows.
A regression test could construct a Sleep test with a deliberately-failing
command and assert the resulting status is FAILED, not PASSED.

Screenshots

Not applicable. The relevant evidence is the captured stdout/log text above.

Additional Context

AI was used for context and guidance while investigating and drafting this
report.