Ignore nested node_modules and .git in the workspace root watch - #2041
Ignore nested node_modules and .git in the workspace root watch#2041SawyerHood wants to merge 3 commits into
Conversation
The workspace-root parcel subscription only excluded the root's own top-level git-ignored directories plus the path ".git", so an umbrella root with untracked nested checkouts (or a non-git root) got one inotify watch per nested directory and could OOM the host. Add recursive glob ignores to every workspace-root subscribe. Fixes #1779 Co-Authored-By: Claude <noreply@anthropic.com>
|
🚨 SLOP COP 🚨 · I am SlopCop. I am reviewing this pull request for security, code quality, architecture, performance, and test coverage. |
| // still skip nested repositories and heavy directories. | ||
| this.startWatchSubscription({ | ||
| kind: "workspace-root", | ||
| options: { ignore: createPlainWorkspaceRootIgnores() }, |
There was a problem hiding this comment.
🚨 slopcop/review — Different same-root subscriptions make this fix depend on start order.
Parcel keeps one directory tree for each root, even when subscriptions use different ignore sets. If a plugin watch starts first, this workspace watch reuses the full tree and can still create every Linux watch. If this watch starts first, a later plugin watch cannot see existing excluded paths. Isolate subscriptions with different ignore sets. Add a Linux test for both start orders.
There was a problem hiding this comment.
I measured both start orders on Linux with two real parcel subscriptions on the same umbrella root (one plain, one with the workspace globs):
plain-first: afterPlain=204 watches, afterBoth=204 (plain subscription saw nested events)
ours-first: afterOurs=3 watches, afterBoth=3 (plain subscription did not see nested events)
So the claim "this workspace watch reuses the full tree and can still create every Linux watch" is not what happens: InotifyBackend::watchDir reuses the same inotify fd, so the workspace watch adds zero watches when a full-tree subscription already exists. The watch count is set by the least restrictive subscription on that root, which in that scenario is the other subscription's ignore list, not this one.
The reverse order is real, but it is parcel's per-root DirTree::getCached cache and it is not introduced by this PR: the existing .git and Git-derived ignores (.turbo, coverage, …) already prune the shared tree the same way for any later same-root subscription. Parcel keeps the tree in a C++ singleton keyed by the resolved root, so separate subscriptions in one backend cannot be isolated from the JS side; that would need a parcel change or separate watcher processes. I am not changing it in this PR, which is scoped to the #1779 OOM. If we want to track the shared-tree limitation for same-root plugin watches, that belongs in its own issue.
| // after `git init`. | ||
| const WORKSPACE_ROOT_ALWAYS_IGNORED_GLOBS = [ | ||
| "*/**/.git", | ||
| "**/node_modules", |
There was a problem hiding this comment.
🚨 slopcop/review — These patterns do not ignore child events on macOS or Windows.
Parcel checks each event path on these systems. **/node_modules matches the directory, but it does not match node_modules/pkg/file.js. A macOS test returned the package path and file path after an ignored-file update. **/node_modules/** removed those child events. Add child patterns for each directory. Add a test that verifies events on supported systems.
There was a problem hiding this comment.
Fixed in ebcce62. The globs are now */**/.git/**, **/node_modules/**, **/.cache/**, **/__pycache__/**. picomatch lets a trailing /** match zero segments, so one pattern covers the directory (Linux crawl skip) and its children (macOS/Windows per-event check). I verified the regexes against .git/HEAD, a/.git/HEAD, a/node_modules/x/y.js and src/index.ts, and added a real-watcher test (does not report changes inside nested node_modules or nested .git) that runs on every platform and fails on main.
There was a problem hiding this comment.
🚨 SLOP COP 🚨 · review
Plain-English summary: This change keeps the workspace watcher out of nested dependency, Git, and cache directories. It lowers Linux watch counts and host memory use.
I found two material problems.
- Parcel shares one directory tree for each root, without regard to each ignore set. A same-root plugin watch makes the result depend on start order. One order keeps the Linux OOM risk. The other order hides files from the plugin watch.
- The directory globs do not match child event paths on macOS or Windows. A macOS test confirmed that an ignored file update still returned package and file events. Child globs removed those events.
Please isolate same-root subscriptions with different ignore sets. Add a Linux test for both start orders. Also add child globs and a portable event test.
I found no new security defect. The Git path glob risk and the absent hard resource limit predate this change.
Project-wide search found no duplicate workspace-root ignore policy. Similar skip lists serve different features, so a shared helper would create a false dependency. The host watcher owns this policy. This change needs no daemon protocol version update.
The focused host-watcher tests passed with 45 tests and two macOS skips. The type check passed. The Linux package job passed with the new inotify tests. A direct macOS Parcel test confirmed the child-event problem. This host-only change has no practical browser test path.
Parcel tests each event path against the ignore globs on macOS and Windows, so `**/node_modules` alone let events from files inside nested node_modules through. Use `<dir>/**` globs, which picomatch matches against both the directory and its children, and add a portable real-watcher test that writes inside nested node_modules and .git. Co-Authored-By: Claude <noreply@anthropic.com>
Build the nested tree in parallel, give the tests a longer timeout, and bind the real parcel subscribe once at module load so a spy left behind by a failed test cannot recurse into itself. Co-Authored-By: Claude <noreply@anthropic.com>
What was wrong
When a UI surface subscribes to environment-detail, the host daemon watches the environment root recursively with
@parcel/watcher. The workspace-root subscription's ignore list was built only from the root's own top-level!! dir/entries ofgit status --ignored=matching --untracked-files=normal, plus the plain path.git. Plain entries resolve to<root>/<entry>, so they never match nested directories. A non-git root got no ignore list at all. For an "umbrella" root with untracked nested checkouts, parcel crawled every nestednode_modules,.git, and cache directory, added one inotify watch per directory, and held aDirTreeentry for every file. At the reporter's scale that is ~1M watches and tens of GB RSS, the kernel OOM-kills the child, andParcelWatcherProxyreplays the same subscription on respawn. See #1779 and the investigation report: https://get-bb.github.io/reports/issues/1779.htmlWhat changed
packages/host-watcher/src/workspace-status-watcher.ts: addWORKSPACE_ROOT_ALWAYS_IGNORED_GLOBS(*/**/.git,**/node_modules,**/.cache,**/__pycache__) and apply them to every workspace-root subscribe: the Git-derived list, the git-status-failed fallback, and the non-git branch. Parcel matches glob entries against the root-relative path and skips matching subtrees during its crawl.*/**/.gitkeeps<root>/.gitwatchable so the non-git branch still promotes the workspace aftergit init.HOST_DAEMON_PROTOCOL_VERSIONis unchanged.Note: a workspace that tracks files inside a nested
node_modulesno longer produces content-change events for those files. The watch only drives "something changed, re-read status", so this is acceptable.How you verified
packages/host-watcher/test/workspace-root-ignores.test.ts(Linux only): builds an umbrella root with 4 nested repos and 2,400 nestednode_modulesdirectories, drives the realwatchWorkspaceStatus()with the real parcel watcher, and counts inotify watches in/proc/self/fdinfo. Before the fix: 2,486 watches for the git root and every nested dir for the non-git root (both cases fail). After the fix: fewer than 20 watches in both cases.test/watch-status.test.ts; thegit initpromotion test still passes.pnpm exec turbo run test typecheck --filter=@bb/host-watcher --force: 6 files, 47 tests passed; typecheck passed.Fixes #1779