Skip to content

Fix the split guard reporting a cached PASS in CI - #3

Merged
tom-snyder merged 1 commit into
mainfrom
fix/split-guard-test-cache
Aug 7, 2026
Merged

Fix the split guard reporting a cached PASS in CI#3
tom-snyder merged 1 commit into
mainfrom
fix/split-guard-test-cache

Conversation

@tom-snyder

Copy link
Copy Markdown
Member

Summary

main went red on the artifact-split negative control — and it went red for the right reason. The control caught TestSplit reporting ok (cached) while never looking at the injected internal/dast import. A guard that can replay a stale PASS is precisely the failure mode the control exists to detect, so this is the mechanism working, not a flaky check.

Cause

TestSplit's verdict comes from the output of an external go list -deps process, which Go's test-result cache does not track. With the build cache restored by setup-go's cache: true, go test can serve a previous PASS for a package whose import graph has since changed.

Fix

1. -count=1 on all three TestSplit invocations, plus a prominent warning in split_test.go's doc comment so the trap isn't reintroduced by someone invoking it another way. This alone removes the dependence on cache behaviour.

2. The negative control now asserts the right failure, not just any failure. A nonzero exit proves only that something went wrong — a compile error would produce one too, and would let a broken guard masquerade as a working one. It now requires the output to name internal/dast/ciprobe, and explicitly fails if it ever sees (cached) again:

::error::Go replayed a cached result instead of running the test. -count=1 is not taking effect.

That turns any recurrence into a self-describing failure instead of a silent green.

Honest note on evidence

I could not reproduce the stale PASS locally. Locally the package hash changes when the file is added, so the cache correctly misses and the guard fails as intended with or without -count=1. The stale result appeared only in CI against a restored cache.

Rather than guess at the precise cache-key mechanism, -count=1 removes the question entirely, and the (cached) assertion makes any recurrence loud. I'd rather ship a fix that is robust to a mechanism I haven't fully characterised than a fix that depends on my having characterised it correctly.

Testing

# violation injected
go test -count=1 ./cmd/anvil -run TestSplit   -> FAIL, names internal/dast/probe
# after revert
go test -count=1 ./cmd/anvil -run TestSplit   -> ok
gofmt -l .                                     -> clean

The real test is this PR's own CI run, which exercises the negative control against a warm cache — the exact condition that failed on main.

main went red on the negative control, and it went red for the right reason:
the control caught the guard reporting "ok (cached)" while never looking at the
injected internal/dast import. A guard that can replay a stale PASS is exactly
the failure mode the control exists to detect, so this is the mechanism working,
not a flaky check.

Cause: TestSplit's verdict comes from the output of an external `go list -deps`
process, which Go's test-result cache does not track. With the build cache
restored by setup-go's cache: true, `go test` can serve a previous PASS for a
package whose import graph has since changed.

Fixes, in increasing order of how much they buy:

1. -count=1 on all three TestSplit invocations, and a prominent warning in
   split_test.go's doc comment so the trap is not reintroduced by someone
   running it another way. This alone removes the dependence on cache
   behaviour.

2. The negative control now asserts the RIGHT failure instead of any failure.
   A nonzero exit proves only that something went wrong -- a compile error
   would also produce one and would let a broken guard pass for a working one.
   It now requires the output to name internal/dast/ciprobe, and explicitly
   fails if it ever sees "(cached)" again.

Honest note on evidence: I could not reproduce the stale PASS locally. Locally
the package hash changes when the file is added, so the cache correctly misses
and the guard fails as intended both with and without -count=1. The stale
result appeared only in CI against a restored cache. Rather than guess at the
precise cache-key mechanism, -count=1 removes the question, and the "(cached)"
assertion turns any recurrence into an explicit, self-describing failure
instead of a silent green.

Local evidence: with the violation injected, `go test -count=1 ./cmd/anvil
-run TestSplit` FAILS naming internal/dast/probe; after revert it passes;
gofmt clean.
@tom-snyder
tom-snyder merged commit 4008cc2 into main Aug 7, 2026
3 checks passed
@tom-snyder
tom-snyder deleted the fix/split-guard-test-cache branch August 7, 2026 04:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant