Skip to content

test: stop the e2e stack broker deadlocking on close - #707

Merged
frahlg merged 1 commit into
masterfrom
fix-e2e-stack-broker-close-deadlock
Jul 30, 2026
Merged

test: stop the e2e stack broker deadlocking on close#707
frahlg merged 1 commit into
masterfrom
fix-e2e-stack-broker-close-deadlock

Conversation

@frahlg

@frahlg frahlg commented Jul 28, 2026

Copy link
Copy Markdown
Member

go/test/e2e/stack_test.go carries the same MQTT-broker shutdown race that
#706 fixes in go/cmd/sim-ferroamp/e2e_test.go, and this is the same fix.

The race

mochi-mqtt v2.7.9 Clients.GetByListener (clients.go:91) takes RLock and then
calls Clients.Len, which takes RLock again. sync.RWMutex is not reentrant:
a Clients.Delete write lock landing between the two read locks blocks the
second one, while the writer waits for the first. Both goroutines stop for good.

Server.Close walks GetByListener, and the broker runs Clients.Delete the
moment a client drops. stack.Close calls reg.ShutdownAll(), which disconnects
the Ferroamp driver's MQTT client, and then closes the broker — exactly that
race. Confirmed from a CI goroutine dump on the sibling test:
https://github.com/srcfl/ftw/actions/runs/30378575736 (attempt 1, job "core (Go)").

Nothing bounds it here but make e2e's -timeout 180s, so a hit would burn
three minutes and fail an unrelated pull request.

The fix

Capture the broker's client count as a baseline right after it starts, and close
through closeBroker, which waits (bounded, 2s) for the count to fall back to
that baseline before calling Close. Close then runs behind a 5s deadline that
fails the test with a clear message if it still wedges.

Both writers on Clients are quiet at baseline, which is what makes the wait
sufficient rather than merely likely to help:

  • server.go:491 (disconnect path) — Len() only drops once Delete has
    returned, so baseline means no disconnect is in flight. Nothing reconnects
    after ShutdownAll.
  • server.go:1711 (clearExpiredClients, a 1s ticker in eventLoop) — skips
    clients whose StopTime() is zero. At baseline only the inline client
    remains, and it is never stopped, so the sweeper issues no Delete at all.

New() registers the inline client (server.go:200-202), so the baseline is
already settled where it is captured and does not race Serve().

v2.7.9 is the newest release, so there is no upgrade that fixes this.

Verification

make e2e passes; forced uncached run at 25.7s. make verify clean.

An instrumented run reported baseline=1, already at baseline on entry to
teardown, settling in 24µs — the guard costs nothing on the happy path. The
instrumentation is not part of this change.

Notes for review

🤖 Generated with Claude Code

go/cmd/sim-ferroamp/e2e_test.go carries the same race, and the fix here is
the same one. mochi-mqtt v2.7.9 Clients.GetByListener takes the read lock and
then calls Clients.Len, which takes it again. sync.RWMutex is not reentrant,
so a Clients.Delete landing between the two read locks blocks the second one
while the writer waits for the first. Both goroutines stop for good.

Server.Close walks GetByListener, and the broker runs Clients.Delete the
moment a client drops. stack.Close calls reg.ShutdownAll, which disconnects
the Ferroamp driver's MQTT client, and then closes the broker — exactly that
race. Nothing bounds it but `make e2e`'s 180s timeout, so it would burn three
minutes and fail an unrelated pull request.

Close the broker through closeBroker, which waits for the client count to
fall back to the inline client before closing. Both writers on Clients are
then quiet: the disconnect path has already run its Delete, and the
clearExpiredClients sweeper skips the inline client because its StopTime is
zero. A 5-second deadline on Close is the backstop — should the broker still
wedge, the test says so in seconds rather than burning the whole timeout.

v2.7.9 is the newest release, so there is nothing to upgrade to.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@frahlg frahlg added the no-changeset PR intentionally exempt from the changeset requirement (dev tooling / non-shipping) label Jul 28, 2026
@frahlg
frahlg merged commit c43d4af into master Jul 30, 2026
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-changeset PR intentionally exempt from the changeset requirement (dev tooling / non-shipping)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant