Fix 100% CPU spin on closed port forward socket - #54
Open
petrouv wants to merge 1 commit into
Open
Conversation
dataplicity 0.4.40 `portforward.Connection.run` handles a zero byte recv() with a `break` that only leaves the inner `for` loop over the poll results, not the outer `while`. EOF keeps the socket permanently readable, so poll() returns immediately and the thread calls recv() forever, burning a core. The loop's only per connection exit is `channel.is_closed`, which flips only when the m2m server answers `Channel.close()` with notify_close. While the tunnel is unhealthy - exactly when sockets get aborted - that answer never arrives. `close_event` cannot be used instead, it is shared by the whole port forwarding service. Reloading the config entry does not stop the orphaned thread either. recv() now raises a BaseException that escapes the `except Exception` around it, still runs the `finally` cleanup of run(), and is swallowed by a wrapper. Verified against dataplicity 0.4.40 with a peer that sends FIN and keeps the socket in CLOSE_WAIT: before: thread alive after 6s, 5.58s CPU burned after: thread exits at once, 0.00s CPU, channel.close() called
petrouv
force-pushed
the
fix/portforward-eof-busy-loop
branch
from
August 3, 2026 07:48
2a87b07 to
58b02d9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Symptom
On a Home Assistant Supervised instance, HA Core sat at 200%+ CPU permanently. One thread had burned 28809 seconds of CPU — almost 8 hours — over 11 hours of uptime, doing no useful work:
fd 25was127.0.0.1:42258 -> 127.0.0.1:8123in stateCLOSE-WAIT— a forwarded connection to the HA web server that the peer had closed.Cause
In
dataplicity==0.4.40,portforward.Connection.run:recv()returningb""means EOF, but thebreakonly leaves the innerforloop over the poll results. EOF keeps the socket permanently readable, sopoll()returns immediately,recv()returnsb""again, and the thread spins at full speed forever. Theexcept Exception: breakabove it has the same problem.Neither of the loop's exits can save it:
channel.is_closedflips only inChannel.on_close(), i.e. when the m2m server answersChannel.close()withnotify_close.Channel.close()is documented as "Call to request a close". While the tunnel is unhealthy — which is exactly when connections get aborted — that answer never arrives.close_eventis shared by the whole port forwarding service ("The one close event to rule them all"), so a single connection must not set it.The thread also survives
homeassistant.reload_config_entry: it kept spinning for 11 hours across a reload, and further spinning threads accumulated alongside it — three at once.Fix
recv()raises aBaseExceptionsubclass on EOF. It escapes theexcept Exceptionaround the call, still runs thefinallycleanup inrun()(channel.close()+_shutdown_read()), and is swallowed by a thinrunwrapper. This keeps the fix per connection and avoids duplicating the vendored loop.Verification
Reproduction against real
dataplicity==0.4.40: a peer accepts, sendsFIN, and holds the socket inCLOSE_WAIT, with a fake channel whoseis_closednever flips (unhealthy tunnel).channel.close()calledReproduction script
No regression with the patch applied: the tunnel keeps serving normally, HA answers over the wormhole URL in ~1.6 s, and no forwarder thread is left behind.