Keep completed results when --fail-fast aborts a run - #4744
Open
ivmat wants to merge 1 commit into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Retains completed harness results when --fail-fast aborts verification, keeping summaries and JSON exports accurate.
Changes:
- Accumulates parallel results safely and restores harness ordering.
- Replaces schedule-dependent UI expectations with script-based checks.
- Adds a deterministic sequential regression test.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
kani-driver/src/harness_runner.rs |
Retains and sorts completed results. |
tests/ui/multiple-harnesses/stop_at_single_fail/fail_fast_test_parallel.expected |
Removes obsolete fixed summary. |
tests/script-based-pre/fail_fast_parallel/fixture.rs |
Updates parallel fixture documentation. |
tests/script-based-pre/fail_fast_parallel/early_abort.sh |
Checks parallel fail-fast behavior. |
tests/script-based-pre/fail_fast_parallel/early_abort.expected |
Adds expected script result. |
tests/script-based-pre/fail_fast_parallel/config.yml |
Configures parallel regression test. |
tests/script-based-pre/fail_fast_keeps_completed/keeps_completed.sh |
Checks deterministic result retention. |
tests/script-based-pre/fail_fast_keeps_completed/keeps_completed.expected |
Adds expected script result. |
tests/script-based-pre/fail_fast_keeps_completed/fixture.rs |
Adds passing and failing harnesses. |
tests/script-based-pre/fail_fast_keeps_completed/config.yml |
Configures sequential regression test. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
`check_all_harnesses` collected with `collect::<Result<Vec<_>>>()`,
which short-circuits on the first error. A `--fail-fast` abort was such
an error, so every harness that had already completed was dropped. The
summary then contradicted the per-harness output above it, and the
`--export-json` file under-reported the run the same way.
Accumulate completed results in a shared vector instead. The abort
signal carries no payload; the failing harness records its result like
any other. Results are re-sorted into harness order after the parallel
loop, since completion order is nondeterministic.
The parallel fail-fast UI test pinned the dropped-results behavior
("1 failures, 1 total" with ten failing harnesses under `--jobs 4`).
With completed results retained, its counts depend on thread
scheduling, so it becomes a script-based test asserting the stable
properties: the run aborts early, every counted harness is a failure,
and the summary total equals the number of verdicts printed above it.
The sequential UI test is unchanged: it aborts on its first harness,
so its pinned summary stays correct. A new sequential script-based
test covers result retention deterministically, in both the summary
and the --export-json file.
Resolves model-checking#4729
ivmat
force-pushed
the
fix-fail-fast-collect
branch
from
August 18, 2026 18:42
7ef0626 to
3ed8af4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
check_all_harnessescollected results withcollect::<Result<Vec<_>>>(), which short-circuits on the first error. A--fail-fastabort was such an error, so every result that had already completed was dropped. The summary then contradicted the per-harness output above it, and--export-jsonunder-reported the run the same way.Completed results now accumulate in a shared vector as harnesses finish. The abort signal carries no payload; the failing harness records its result like any other. Results are re-sorted into harness order after the parallel loop, since completion order is nondeterministic.
The parallel fail-fast UI test pinned the old behavior: "1 failures, 1 total" for ten failing harnesses under
--jobs 4. With completed results retained, those counts depend on thread scheduling, so the test becomes a script-based test asserting the stable properties: the run aborts early (fewer than ten run) and every counted harness is a failure. The sequential UI test is unchanged: it aborts on its first harness, so its pinned summary stays correct. A new sequential script-based test proves retention deterministically ("1 successfully verified, 1 failures, 2 total"); it fails onmain.Testing:
cargo test -p kani-driver,rustfmt, andclippyare clean; the two new script-based tests and the existingstop_at_single_failUI test pass.Resolves #4729