Skip to content

Fix exec hang and stop timeout#1926

Open
lakshitsoni26 wants to merge 3 commits into
apple:mainfrom
lakshitsoni26:fix-exec-hang-and-stop-timeout
Open

Fix exec hang and stop timeout#1926
lakshitsoni26 wants to merge 3 commits into
apple:mainfrom
lakshitsoni26:fix-exec-hang-and-stop-timeout

Conversation

@lakshitsoni26

Copy link
Copy Markdown

Fixes #1916

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update

Motivation and Context

This PR resolves a critical bug where container exec and container stop commands hang indefinitely when an SSH session disconnects abruptly (or the terminal window is closed without cleanly exiting).

The underlying issues were identified and addressed in two distinct commits:

  1. PTY Resource Leak / Orphaned Process: When an XPC client connection dropped unexpectedly (e.g., SIGHUP, broken pipe), the corresponding exec process inside the container remained orphaned. Subsequent exec attempts would try to acquire the same resources, hit a blocking read/write on a dead file descriptor, and hang indefinitely. This was fixed by registering an onDisconnect listener in ContainersHarness to forcefully kill the specific orphaned process upon a client disconnect.
  2. Ignored Stop Timeout: The container stop command completely ignored the stopTimeoutSeconds flag if an exec session was blocking the RuntimeService actor lock. This PR wraps the lock acquisition and the graceful stop sequence inside a withThrowingTaskGroup. This ensures that even if an exec session is infinitely holding the lock, the container is forcefully killed if it doesn't shut down within the designated 5-second timeout window.

Fixing these primary issues also inherently prevents the secondary symptom where container list incorrectly reports the container as running while container exec fails.

Testing

  • Tested locally
  • Added/updated tests
  • Added/updated docs

Local Test Verification:

# Simulated a SIGHUP abrupt disconnect to an active SSH/bash session, 
# then immediately started a new exec session. It connected instantly without hanging:
./.build/debug/container exec -it test0 bash
root@test0:/# echo SUCCESS_EXEC
SUCCESS_EXEC

# Verified the stop timeout enforcement triggers gracefully:
./.build/debug/container stop test0
test0
./.build/debug/container stop test0  0.02s user 0.01s system 0% cpu 5.875 total

# Verified correct state sync:
./.build/debug/container list --all
ID     IMAGE                            OS     ARCH   STATE    STARTED
test0  docker.io/library/ubuntu:latest  linux  arm64  stopped  2026-07-09T11:22:51Z

Ensure the exec session process is cleaned up when the client connection drops (e.g. broken pipe, SSH drop, SIGHUP), preventing zombie processes from hanging subsequent exec sessions.
Wrap the lock acquisition and graceful stop in a TaskGroup that enforces the timeout. This ensures that if the lock is held indefinitely by a hung exec session, the container is forcefully killed after the timeout.
stephenlclarke added a commit to stephenlclarke/container that referenced this pull request Jul 9, 2026
Port the useful exec-session cleanup shape from apple#1926 into the Stephen runtime lane without pushing to Apple.

The containerCreateProcess XPC route now keeps its XPCServerSession, and the harness registers disconnect cleanup for client-attached ad hoc exec processes. Detached exec processes keep their existing Docker-compatible lifetime and are not killed when the CLI connection closes.

This improves container-compose exec and lifecycle reliability by preventing orphaned attached exec sessions from blocking later exec or stop operations.

Release-Highlight: Improves container compose exec reliability by killing attached exec processes when the client disconnects, preventing orphaned sessions from blocking later exec or stop operations while preserving detached exec; ports the useful cleanup from apple#1926 for apple#1916.
When the guest agent disconnects unexpectedly (e.g. because of the PTY closing on the host), the host tries to forward SIGKILL to the agent but hits a `clientIsStopped` exception from GRPC, leaving waiters hung. This catches that exception and cleanly releases waiters.

Credit: Bug found by @stephenlclarke.
@lakshitsoni26

Copy link
Copy Markdown
Author

Fix clientIsStopped error when sending signal to dead agent

When the guest agent disconnects unexpectedly (e.g. because of the PTY closing on the host), the host tries to forward SIGKILL to the agent but hits a clientIsStopped exception from GRPC, leaving waiters hung. This catches that exception and cleanly releases waiters.

Credit: Bug found by @stephenlclarke.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: container exec/stop hang indefinitely on SSH reconnect, requiring full macOS restart to recover

1 participant