Skip to content

Ignore nested node_modules and .git in the workspace root watch - #2041

Open
SawyerHood wants to merge 3 commits into
mainfrom
bb/fix-1779-recursive-workspace-watch-can-oom-the-h-thr_2dyz3mvb6z
Open

Ignore nested node_modules and .git in the workspace root watch#2041
SawyerHood wants to merge 3 commits into
mainfrom
bb/fix-1779-recursive-workspace-watch-can-oom-the-h-thr_2dyz3mvb6z

Conversation

@SawyerHood

Copy link
Copy Markdown
Collaborator

What was wrong

When a UI surface subscribes to environment-detail, the host daemon watches the environment root recursively with @parcel/watcher. The workspace-root subscription's ignore list was built only from the root's own top-level !! dir/ entries of git status --ignored=matching --untracked-files=normal, plus the plain path .git. Plain entries resolve to <root>/<entry>, so they never match nested directories. A non-git root got no ignore list at all. For an "umbrella" root with untracked nested checkouts, parcel crawled every nested node_modules, .git, and cache directory, added one inotify watch per directory, and held a DirTree entry for every file. At the reporter's scale that is ~1M watches and tens of GB RSS, the kernel OOM-kills the child, and ParcelWatcherProxy replays the same subscription on respawn. See #1779 and the investigation report: https://get-bb.github.io/reports/issues/1779.html

What changed

  • packages/host-watcher/src/workspace-status-watcher.ts: add WORKSPACE_ROOT_ALWAYS_IGNORED_GLOBS (*/**/.git, **/node_modules, **/.cache, **/__pycache__) and apply them to every workspace-root subscribe: the Git-derived list, the git-status-failed fallback, and the non-git branch. Parcel matches glob entries against the root-relative path and skips matching subtrees during its crawl. */**/.git keeps <root>/.git watchable so the non-git branch still promotes the workspace after git init.
  • No wire change. The ignore list stays inside the host daemon, so HOST_DAEMON_PROTOCOL_VERSION is unchanged.
  • Not in this PR: the report's optional layer 2 (a directory/inotify budget with a terminal watch error). That changes the host RPC error shape and is a separate change.

Note: a workspace that tracks files inside a nested node_modules no longer produces content-change events for those files. The watch only drives "something changed, re-read status", so this is acceptable.

How you verified

  • New packages/host-watcher/test/workspace-root-ignores.test.ts (Linux only): builds an umbrella root with 4 nested repos and 2,400 nested node_modules directories, drives the real watchWorkspaceStatus() with the real parcel watcher, and counts inotify watches in /proc/self/fdinfo. Before the fix: 2,486 watches for the git root and every nested dir for the non-git root (both cases fail). After the fix: fewer than 20 watches in both cases.
  • Updated three exact-array ignore assertions in test/watch-status.test.ts; the git init promotion test still passes.
  • pnpm exec turbo run test typecheck --filter=@bb/host-watcher --force: 6 files, 47 tests passed; typecheck passed.

Fixes #1779

AGENT GENERATED: by Claude Opus 5

The workspace-root parcel subscription only excluded the root's own
top-level git-ignored directories plus the path ".git", so an umbrella
root with untracked nested checkouts (or a non-git root) got one inotify
watch per nested directory and could OOM the host. Add recursive glob
ignores to every workspace-root subscribe.

Fixes #1779

Co-Authored-By: Claude <noreply@anthropic.com>
@SawyerHood
SawyerHood marked this pull request as ready for review August 20, 2026 17:23
@bb-slop-cop

bb-slop-cop Bot commented Aug 20, 2026

Copy link
Copy Markdown

🚨 SLOP COP 🚨 · review

I am SlopCop. I am reviewing this pull request for security, code quality, architecture, performance, and test coverage.

// still skip nested repositories and heavy directories.
this.startWatchSubscription({
kind: "workspace-root",
options: { ignore: createPlainWorkspaceRootIgnores() },

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚨 slopcop/review — Different same-root subscriptions make this fix depend on start order.

Parcel keeps one directory tree for each root, even when subscriptions use different ignore sets. If a plugin watch starts first, this workspace watch reuses the full tree and can still create every Linux watch. If this watch starts first, a later plugin watch cannot see existing excluded paths. Isolate subscriptions with different ignore sets. Add a Linux test for both start orders.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I measured both start orders on Linux with two real parcel subscriptions on the same umbrella root (one plain, one with the workspace globs):

plain-first: afterPlain=204 watches, afterBoth=204   (plain subscription saw nested events)
ours-first:  afterOurs=3 watches,    afterBoth=3     (plain subscription did not see nested events)

So the claim "this workspace watch reuses the full tree and can still create every Linux watch" is not what happens: InotifyBackend::watchDir reuses the same inotify fd, so the workspace watch adds zero watches when a full-tree subscription already exists. The watch count is set by the least restrictive subscription on that root, which in that scenario is the other subscription's ignore list, not this one.

The reverse order is real, but it is parcel's per-root DirTree::getCached cache and it is not introduced by this PR: the existing .git and Git-derived ignores (.turbo, coverage, …) already prune the shared tree the same way for any later same-root subscription. Parcel keeps the tree in a C++ singleton keyed by the resolved root, so separate subscriptions in one backend cannot be isolated from the JS side; that would need a parcel change or separate watcher processes. I am not changing it in this PR, which is scoped to the #1779 OOM. If we want to track the shared-tree limitation for same-root plugin watches, that belongs in its own issue.

// after `git init`.
const WORKSPACE_ROOT_ALWAYS_IGNORED_GLOBS = [
"*/**/.git",
"**/node_modules",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚨 slopcop/review — These patterns do not ignore child events on macOS or Windows.

Parcel checks each event path on these systems. **/node_modules matches the directory, but it does not match node_modules/pkg/file.js. A macOS test returned the package path and file path after an ignored-file update. **/node_modules/** removed those child events. Add child patterns for each directory. Add a test that verifies events on supported systems.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ebcce62. The globs are now */**/.git/**, **/node_modules/**, **/.cache/**, **/__pycache__/**. picomatch lets a trailing /** match zero segments, so one pattern covers the directory (Linux crawl skip) and its children (macOS/Windows per-event check). I verified the regexes against .git/HEAD, a/.git/HEAD, a/node_modules/x/y.js and src/index.ts, and added a real-watcher test (does not report changes inside nested node_modules or nested .git) that runs on every platform and fails on main.

@bb-slop-cop bb-slop-cop Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚨 SLOP COP 🚨 · review

Plain-English summary: This change keeps the workspace watcher out of nested dependency, Git, and cache directories. It lowers Linux watch counts and host memory use.

I found two material problems.

  1. Parcel shares one directory tree for each root, without regard to each ignore set. A same-root plugin watch makes the result depend on start order. One order keeps the Linux OOM risk. The other order hides files from the plugin watch.
  2. The directory globs do not match child event paths on macOS or Windows. A macOS test confirmed that an ignored file update still returned package and file events. Child globs removed those events.

Please isolate same-root subscriptions with different ignore sets. Add a Linux test for both start orders. Also add child globs and a portable event test.

I found no new security defect. The Git path glob risk and the absent hard resource limit predate this change.

Project-wide search found no duplicate workspace-root ignore policy. Similar skip lists serve different features, so a shared helper would create a false dependency. The host watcher owns this policy. This change needs no daemon protocol version update.

The focused host-watcher tests passed with 45 tests and two macOS skips. The type check passed. The Linux package job passed with the new inotify tests. A direct macOS Parcel test confirmed the child-event problem. This host-only change has no practical browser test path.

SawyerHood and others added 2 commits August 20, 2026 18:17
Parcel tests each event path against the ignore globs on macOS and
Windows, so `**/node_modules` alone let events from files inside nested
node_modules through. Use `<dir>/**` globs, which picomatch matches
against both the directory and its children, and add a portable
real-watcher test that writes inside nested node_modules and .git.

Co-Authored-By: Claude <noreply@anthropic.com>
Build the nested tree in parallel, give the tests a longer timeout, and
bind the real parcel subscribe once at module load so a spy left behind
by a failed test cannot recurse into itself.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Recursive workspace watch can OOM the host for a large environment root

1 participant