Fix the split guard reporting a cached PASS in CI - #3
Merged
Conversation
main went red on the negative control, and it went red for the right reason: the control caught the guard reporting "ok (cached)" while never looking at the injected internal/dast import. A guard that can replay a stale PASS is exactly the failure mode the control exists to detect, so this is the mechanism working, not a flaky check. Cause: TestSplit's verdict comes from the output of an external `go list -deps` process, which Go's test-result cache does not track. With the build cache restored by setup-go's cache: true, `go test` can serve a previous PASS for a package whose import graph has since changed. Fixes, in increasing order of how much they buy: 1. -count=1 on all three TestSplit invocations, and a prominent warning in split_test.go's doc comment so the trap is not reintroduced by someone running it another way. This alone removes the dependence on cache behaviour. 2. The negative control now asserts the RIGHT failure instead of any failure. A nonzero exit proves only that something went wrong -- a compile error would also produce one and would let a broken guard pass for a working one. It now requires the output to name internal/dast/ciprobe, and explicitly fails if it ever sees "(cached)" again. Honest note on evidence: I could not reproduce the stale PASS locally. Locally the package hash changes when the file is added, so the cache correctly misses and the guard fails as intended both with and without -count=1. The stale result appeared only in CI against a restored cache. Rather than guess at the precise cache-key mechanism, -count=1 removes the question, and the "(cached)" assertion turns any recurrence into an explicit, self-describing failure instead of a silent green. Local evidence: with the violation injected, `go test -count=1 ./cmd/anvil -run TestSplit` FAILS naming internal/dast/probe; after revert it passes; gofmt clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mainwent red on the artifact-split negative control — and it went red for the right reason. The control caughtTestSplitreportingok (cached)while never looking at the injectedinternal/dastimport. A guard that can replay a stale PASS is precisely the failure mode the control exists to detect, so this is the mechanism working, not a flaky check.Cause
TestSplit's verdict comes from the output of an externalgo list -depsprocess, which Go's test-result cache does not track. With the build cache restored bysetup-go'scache: true,go testcan serve a previous PASS for a package whose import graph has since changed.Fix
1.
-count=1on all threeTestSplitinvocations, plus a prominent warning insplit_test.go's doc comment so the trap isn't reintroduced by someone invoking it another way. This alone removes the dependence on cache behaviour.2. The negative control now asserts the right failure, not just any failure. A nonzero exit proves only that something went wrong — a compile error would produce one too, and would let a broken guard masquerade as a working one. It now requires the output to name
internal/dast/ciprobe, and explicitly fails if it ever sees(cached)again:That turns any recurrence into a self-describing failure instead of a silent green.
Honest note on evidence
I could not reproduce the stale PASS locally. Locally the package hash changes when the file is added, so the cache correctly misses and the guard fails as intended with or without
-count=1. The stale result appeared only in CI against a restored cache.Rather than guess at the precise cache-key mechanism,
-count=1removes the question entirely, and the(cached)assertion makes any recurrence loud. I'd rather ship a fix that is robust to a mechanism I haven't fully characterised than a fix that depends on my having characterised it correctly.Testing
The real test is this PR's own CI run, which exercises the negative control against a warm cache — the exact condition that failed on
main.