Skip to content

Stop ssh consuming the test driver's stdin - #107

Merged
kwsantiago merged 1 commit into
mainfrom
fix-ssh-stdin-hang
Aug 2, 2026
Merged

Stop ssh consuming the test driver's stdin#107
kwsantiago merged 1 commit into
mainfrom
fix-ssh-stdin-hang

Conversation

@kwsantiago

@kwsantiago kwsantiago commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds -n to the five ssh invocations in the VM tests. Fixes check (adminaccess-bringup) hanging in CI until the 30-minute job timeout.

Root cause

machine.succeed() runs its command through a shell on the VM's backdoor serial channel. ssh without -n reads that channel's stdin, so it swallows the driver's own protocol stream. The command completes in the guest while the driver waits forever for a completion marker that ssh already consumed.

The CI log shows exactly that. box.succeed(f"{ssh} keepadmin@{lan} sudo -n true") starts at 00:58:11, and the guest reports:

sudo[1281]: keepadmin : *** ; USER=root ; COMMAND=/run/current-system/sw/bin/true
sudo[1281]: pam_unix(sudo:session): session opened for user root ... session closed
sshd-session[1280]: Received disconnect ... disconnected by user

Session opened and closed at t+18.9s. The driver then printed nothing for 29 minutes, until the runner cancelled the job. The guest did the work; the driver never heard about it.

It is a race, which is why it passes locally and hangs in CI.

Scope

Five invocations across three files had the same shape, not just the one that was failing:

file sites
tests/adminaccess-bringup.nix 1, the one hanging
tests/mesh-admin-ssh.nix 3
tests/yubikey-ssh.nix 1

The other four pass today on the same coin-flip. Fixed all five rather than the symptom. None of them feed stdin: every command is true or sudo -n true, so -n changes nothing about intent.

Why this went unnoticed

Across the last 25 CI runs, no execution of check (adminaccess-bringup) exceeded 3 minutes except the two that exposed this. Every other run was the fast lane, where the VM tests do not execute, and the job reports success in about 65s because it is a no-op there. The admin-access path, SSH and YubiKey, has had the appearance of CI coverage without the substance.

Test plan

Run locally with KVM, on this branch:

  • adminaccess-bringup: PASS (the test that hangs in CI)
  • yubikey-ssh: PASS
  • mesh-admin-ssh: PASS, and 3/3 on forced --rebuild repeats
  • Baseline mesh-admin-ssh on b766ecd without -n: PASS, confirming the comparison is meaningful
  • All three files parse; nix fmt reports 0 changed
  • Full VM matrix in CI, labeled full-ci

One honest note on the measurement: the very first mesh-admin-ssh run on this branch failed, which I initially reported as -n breaking it. It did not reproduce in four subsequent runs, and that first run executed immediately after another VM test in the same invocation. The failure is not attributable to this change, but mesh-admin-ssh may have a pre-existing flake under load, which is worth watching separately.

Unblocks #106, whose only remaining failure is this hang.

Summary by CodeRabbit

  • Bug Fixes
    • Improved SSH-based test reliability by preventing SSH from consuming the test harness input stream.
    • Prevented potential hangs during VM and serial-channel test completion.

@coderabbitai

coderabbitai Bot commented Aug 2, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 49d4d487-aa5b-4a51-a54f-730432e6b6b7

📥 Commits

Reviewing files that changed from the base of the PR and between b766ecd and 1a0d9e5.

📒 Files selected for processing (3)
  • tests/adminaccess-bringup.nix
  • tests/mesh-admin-ssh.nix
  • tests/yubikey-ssh.nix

Walkthrough

The SSH commands in three Nix VM tests now use ssh -n. This prevents SSH from reading the VM driver’s stdin and consuming its serial protocol stream.

Changes

SSH stdin handling

Layer / File(s) Summary
Update SSH test commands
tests/adminaccess-bringup.nix, tests/mesh-admin-ssh.nix, tests/yubikey-ssh.nix
SSH commands now use -n. Mesh test comments document the serial-channel stdin issue. Existing connection options and assertions remain unchanged.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Possibly related PRs

Poem

A rabbit sees SSH hop,
With stdin closed, tests no longer stop.
The serial stream stays clear and bright,
Commands finish cleanly every night.
-n makes the harness take flight!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: preventing SSH from consuming the test driver's stdin.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix-ssh-stdin-hang

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@kwsantiago
kwsantiago merged commit 76db18a into main Aug 2, 2026
37 checks passed
@kwsantiago
kwsantiago deleted the fix-ssh-stdin-hang branch August 2, 2026 14:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant