Skip to content

Keep completed results when --fail-fast aborts a run - #4744

Open
ivmat wants to merge 1 commit into
model-checking:mainfrom
ivmat:fix-fail-fast-collect
Open

Keep completed results when --fail-fast aborts a run#4744
ivmat wants to merge 1 commit into
model-checking:mainfrom
ivmat:fix-fail-fast-collect

Conversation

@ivmat

@ivmat ivmat commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

check_all_harnesses collected results with collect::<Result<Vec<_>>>(), which short-circuits on the first error. A --fail-fast abort was such an error, so every result that had already completed was dropped. The summary then contradicted the per-harness output above it, and --export-json under-reported the run the same way.

Completed results now accumulate in a shared vector as harnesses finish. The abort signal carries no payload; the failing harness records its result like any other. Results are re-sorted into harness order after the parallel loop, since completion order is nondeterministic.

The parallel fail-fast UI test pinned the old behavior: "1 failures, 1 total" for ten failing harnesses under --jobs 4. With completed results retained, those counts depend on thread scheduling, so the test becomes a script-based test asserting the stable properties: the run aborts early (fewer than ten run) and every counted harness is a failure. The sequential UI test is unchanged: it aborts on its first harness, so its pinned summary stays correct. A new sequential script-based test proves retention deterministically ("1 successfully verified, 1 failures, 2 total"); it fails on main.

Testing: cargo test -p kani-driver, rustfmt, and clippy are clean; the two new script-based tests and the existing stop_at_single_fail UI test pass.

Resolves #4729

@ivmat
ivmat requested a review from a team as a code owner August 18, 2026 05:52
@feliperodri feliperodri added the [I] Refactoring / Clean Up Refactoring or cleaning up of existing code label Aug 18, 2026
@feliperodri
feliperodri requested a balanced review from Copilot August 18, 2026 14:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Retains completed harness results when --fail-fast aborts verification, keeping summaries and JSON exports accurate.

Changes:

  • Accumulates parallel results safely and restores harness ordering.
  • Replaces schedule-dependent UI expectations with script-based checks.
  • Adds a deterministic sequential regression test.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
kani-driver/src/harness_runner.rs Retains and sorts completed results.
tests/ui/multiple-harnesses/stop_at_single_fail/fail_fast_test_parallel.expected Removes obsolete fixed summary.
tests/script-based-pre/fail_fast_parallel/fixture.rs Updates parallel fixture documentation.
tests/script-based-pre/fail_fast_parallel/early_abort.sh Checks parallel fail-fast behavior.
tests/script-based-pre/fail_fast_parallel/early_abort.expected Adds expected script result.
tests/script-based-pre/fail_fast_parallel/config.yml Configures parallel regression test.
tests/script-based-pre/fail_fast_keeps_completed/keeps_completed.sh Checks deterministic result retention.
tests/script-based-pre/fail_fast_keeps_completed/keeps_completed.expected Adds expected script result.
tests/script-based-pre/fail_fast_keeps_completed/fixture.rs Adds passing and failing harnesses.
tests/script-based-pre/fail_fast_keeps_completed/config.yml Configures sequential regression test.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread kani-driver/src/harness_runner.rs
Comment thread tests/script-based-pre/fail_fast_parallel/early_abort.sh
`check_all_harnesses` collected with `collect::<Result<Vec<_>>>()`,
which short-circuits on the first error. A `--fail-fast` abort was such
an error, so every harness that had already completed was dropped. The
summary then contradicted the per-harness output above it, and the
`--export-json` file under-reported the run the same way.

Accumulate completed results in a shared vector instead. The abort
signal carries no payload; the failing harness records its result like
any other. Results are re-sorted into harness order after the parallel
loop, since completion order is nondeterministic.

The parallel fail-fast UI test pinned the dropped-results behavior
("1 failures, 1 total" with ten failing harnesses under `--jobs 4`).
With completed results retained, its counts depend on thread
scheduling, so it becomes a script-based test asserting the stable
properties: the run aborts early, every counted harness is a failure,
and the summary total equals the number of verdicts printed above it.
The sequential UI test is unchanged: it aborts on its first harness,
so its pinned summary stays correct. A new sequential script-based
test covers result retention deterministically, in both the summary
and the --export-json file.

Resolves model-checking#4729
@ivmat
ivmat force-pushed the fix-fail-fast-collect branch from 7ef0626 to 3ed8af4 Compare August 18, 2026 18:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

[I] Refactoring / Clean Up Refactoring or cleaning up of existing code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

--fail-fast discards results for harnesses that already completed

3 participants