[CI] Parity: use authoritative flaky-test summaries - #3560
Open
ethanwee1 wants to merge 1 commit into
Open
Conversation
The log-based flaky detector inferred flakiness from the last individual "... PASSED" line seen before a "Test succeeded in new process" marker. That heuristic attaches the flaky flag to whatever test happened to pass most recently, which is often an unrelated test that passed normally much earlier in the shard -- not the test that actually crashed/failed and then passed on rerun. Concrete example (run 29255582284, mi350, inductor/test_compiled_autograd): test_wrap_recreates_contexts_for_repeated_runs passed normally at ~16% and was wrongly reported as flaky, while the test that truly failed-then-passed on rerun was FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_jacfwd. run_test.py already prints an authoritative per-shard summary naming the exact tests: The following tests failed and then succeeded when run in a new process ['test/inductor/test_compiled_autograd.py::...::test_jacfwd'] Parse that line directly instead of guessing. Verified against the real ROCm and CUDA logs from the run above: test_wrap is no longer flagged and test_jacfwd is correctly reported on both platforms.
This was referenced Aug 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Parse
run_test.py's authoritative failed-then-passed summary instead of attributing flakiness to the last unrelatedPASSEDline.Clean replacement for #3431 on current
develop, stacked on #3554. Current develop already contains SHA-prefixed log label handling, so the empty duplicate commit was intentionally omitted.Test plan
detect_log_failures.pyMade with Cursor
Alignment landing order
Depends on #3554 only for branch stacking. Land after the downloader resilience PRs; the flaky parser change itself is isolated.