Skip to content

Fix 100% CPU spin on closed port forward socket - #54

Open
petrouv wants to merge 1 commit into
AlexxIT:masterfrom
petrouv:fix/portforward-eof-busy-loop
Open

Fix 100% CPU spin on closed port forward socket#54
petrouv wants to merge 1 commit into
AlexxIT:masterfrom
petrouv:fix/portforward-eof-busy-loop

Conversation

@petrouv

@petrouv petrouv commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Symptom

On a Home Assistant Supervised instance, HA Core sat at 200%+ CPU permanently. One thread had burned 28809 seconds of CPU — almost 8 hours — over 11 hours of uptime, doing no useful work:

strace -c -p <tid>
% time     seconds  usecs/call     calls    errors syscall
 81.88    4.391663          21    205323     81826 futex
 12.18    0.653231          27     23711           ppoll
  5.94    0.318318          26     11855           recvfrom

recvfrom(25, "", 1048576, 0, NULL, NULL) = 0     # 8818 times in 6 seconds

fd 25 was 127.0.0.1:42258 -> 127.0.0.1:8123 in state CLOSE-WAIT — a forwarded connection to the HA web server that the peer had closed.

Cause

In dataplicity==0.4.40, portforward.Connection.run:

while not self.close_event.is_set():
    poll_result = poll.poll(5 * 1000)
    if self.channel.is_closed:
        break
    for _file_descriptor, event_mask in poll_result:
        if event_mask & readable_events:
            data = self.socket.recv(CHUNK_SIZE)
            if data:
                self.channel.write(data)
            else:
                # No data means the socket has been closed
                break          # <-- leaves the `for`, not the `while`

recv() returning b"" means EOF, but the break only leaves the inner for loop over the poll results. EOF keeps the socket permanently readable, so poll() returns immediately, recv() returns b"" again, and the thread spins at full speed forever. The except Exception: break above it has the same problem.

Neither of the loop's exits can save it:

  • channel.is_closed flips only in Channel.on_close(), i.e. when the m2m server answers Channel.close() with notify_close. Channel.close() is documented as "Call to request a close". While the tunnel is unhealthy — which is exactly when connections get aborted — that answer never arrives.
  • close_event is shared by the whole port forwarding service ("The one close event to rule them all"), so a single connection must not set it.

The thread also survives homeassistant.reload_config_entry: it kept spinning for 11 hours across a reload, and further spinning threads accumulated alongside it — three at once.

Fix

recv() raises a BaseException subclass on EOF. It escapes the except Exception around the call, still runs the finally cleanup in run() (channel.close() + _shutdown_read()), and is swallowed by a thin run wrapper. This keeps the fix per connection and avoids duplicating the vendored loop.

Verification

Reproduction against real dataplicity==0.4.40: a peer accepts, sends FIN, and holds the socket in CLOSE_WAIT, with a fake channel whose is_closed never flips (unhealthy tunnel).

thread alive after 6s CPU burned channel.close() called
before yes 5.58 s no
after no 0.00 s yes
Reproduction script
import socket, threading, time
from dataplicity import portforward

class FakeChannel:
    def __init__(self):
        self.is_closed = False
        self.close_requested = False
    def set_callbacks(self, on_data=None, on_close=None, on_control=None):
        pass
    def write(self, data):
        pass
    def close(self):
        self.close_requested = True

srv = socket.socket()
srv.bind(("127.0.0.1", 0))
srv.listen(1)
port = srv.getsockname()[1]
keep = []

def accept():
    conn, _ = srv.accept()
    keep.append(conn)
    conn.shutdown(socket.SHUT_WR)

threading.Thread(target=accept, daemon=True).start()

ch = FakeChannel()
conn = portforward.Connection(threading.Event(), ch, ("127.0.0.1", port))
cpu0 = time.process_time()
conn.start()
conn.join(timeout=6)
print(conn.is_alive(), time.process_time() - cpu0, ch.close_requested)

No regression with the patch applied: the tunnel keeps serving normally, HA answers over the wormhole URL in ~1.6 s, and no forwarder thread is left behind.

dataplicity 0.4.40 `portforward.Connection.run` handles a zero byte recv()
with a `break` that only leaves the inner `for` loop over the poll results,
not the outer `while`. EOF keeps the socket permanently readable, so poll()
returns immediately and the thread calls recv() forever, burning a core.

The loop's only per connection exit is `channel.is_closed`, which flips only
when the m2m server answers `Channel.close()` with notify_close. While the
tunnel is unhealthy - exactly when sockets get aborted - that answer never
arrives. `close_event` cannot be used instead, it is shared by the whole port
forwarding service. Reloading the config entry does not stop the orphaned
thread either.

recv() now raises a BaseException that escapes the `except Exception` around
it, still runs the `finally` cleanup of run(), and is swallowed by a wrapper.

Verified against dataplicity 0.4.40 with a peer that sends FIN and keeps the
socket in CLOSE_WAIT:
  before: thread alive after 6s, 5.58s CPU burned
  after:  thread exits at once, 0.00s CPU, channel.close() called
@petrouv
petrouv force-pushed the fix/portforward-eof-busy-loop branch from 2a87b07 to 58b02d9 Compare August 3, 2026 07:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant