test: stop the e2e stack broker deadlocking on close - #707
Merged
Conversation
go/cmd/sim-ferroamp/e2e_test.go carries the same race, and the fix here is the same one. mochi-mqtt v2.7.9 Clients.GetByListener takes the read lock and then calls Clients.Len, which takes it again. sync.RWMutex is not reentrant, so a Clients.Delete landing between the two read locks blocks the second one while the writer waits for the first. Both goroutines stop for good. Server.Close walks GetByListener, and the broker runs Clients.Delete the moment a client drops. stack.Close calls reg.ShutdownAll, which disconnects the Ferroamp driver's MQTT client, and then closes the broker — exactly that race. Nothing bounds it but `make e2e`'s 180s timeout, so it would burn three minutes and fail an unrelated pull request. Close the broker through closeBroker, which waits for the client count to fall back to the inline client before closing. Both writers on Clients are then quiet: the disconnect path has already run its Delete, and the clearExpiredClients sweeper skips the inline client because its StopTime is zero. A 5-second deadline on Close is the backstop — should the broker still wedge, the test says so in seconds rather than burning the whole timeout. v2.7.9 is the newest release, so there is nothing to upgrade to. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Jul 30, 2026
Draft
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
go/test/e2e/stack_test.gocarries the same MQTT-broker shutdown race that#706 fixes in
go/cmd/sim-ferroamp/e2e_test.go, and this is the same fix.The race
mochi-mqtt v2.7.9
Clients.GetByListener(clients.go:91) takesRLockand thencalls
Clients.Len, which takesRLockagain.sync.RWMutexis not reentrant:a
Clients.Deletewrite lock landing between the two read locks blocks thesecond one, while the writer waits for the first. Both goroutines stop for good.
Server.ClosewalksGetByListener, and the broker runsClients.Deletethemoment a client drops.
stack.Closecallsreg.ShutdownAll(), which disconnectsthe Ferroamp driver's MQTT client, and then closes the broker — exactly that
race. Confirmed from a CI goroutine dump on the sibling test:
https://github.com/srcfl/ftw/actions/runs/30378575736 (attempt 1, job "core (Go)").
Nothing bounds it here but
make e2e's-timeout 180s, so a hit would burnthree minutes and fail an unrelated pull request.
The fix
Capture the broker's client count as a baseline right after it starts, and close
through
closeBroker, which waits (bounded, 2s) for the count to fall back tothat baseline before calling
Close.Closethen runs behind a 5s deadline thatfails the test with a clear message if it still wedges.
Both writers on
Clientsare quiet at baseline, which is what makes the waitsufficient rather than merely likely to help:
server.go:491(disconnect path) —Len()only drops onceDeletehasreturned, so baseline means no disconnect is in flight. Nothing reconnects
after
ShutdownAll.server.go:1711(clearExpiredClients, a 1s ticker ineventLoop) — skipsclients whose
StopTime()is zero. At baseline only the inline clientremains, and it is never stopped, so the sweeper issues no
Deleteat all.New()registers the inline client (server.go:200-202), so the baseline isalready settled where it is captured and does not race
Serve().v2.7.9 is the newest release, so there is no upgrade that fixes this.
Verification
make e2epasses; forced uncached run at 25.7s.make verifyclean.An instrumented run reported
baseline=1, already at baseline on entry toteardown, settling in 24µs — the guard costs nothing on the happy path. The
instrumentation is not part of this change.
Notes for review
go/cmd/sim-ferroamp/e2e_test.go, whichbelongs to open test: stop the sim-ferroamp broker deadlocking on close #706. The helper therefore exists in two copies, each
cross-referencing the other in its comment. There is no shared test-helper
package to host it, and creating one would mean editing the file test: stop the sim-ferroamp broker deadlocking on close #706 owns.
and used the
no-changesetlabel, applied here too.🤖 Generated with Claude Code