Skip to content

Target Node 24, remove all runtime dependencies, 2.0.0 - #28

Open
expo-tuft[bot] wants to merge 4 commits into
mainfrom
tuft/1786478817.411279/hyperinstall-a1e96f51
Open

Target Node 24, remove all runtime dependencies, 2.0.0#28
expo-tuft[bot] wants to merge 4 commits into
mainfrom
tuft/1786478817.411279/hyperinstall-a1e96f51

Conversation

@expo-tuft

@expo-tuft expo-tuft Bot commented Aug 11, 2026

Copy link
Copy Markdown

Hyperinstall's dependency list had aged into a steady stream of Dependabot PRs (yargs@6, npm-package-arg@4, rimraf@2, fstream-npm@1, instapromise@latest, co) while the program itself is a few hundred lines that Node can now run unassisted. This targets Node 24 and drops every runtime dependency plus the Babel build step.

package.json

  • engines.node: >=16 <=16>=24
  • "type": "module" — the source is plain ESM that Node runs directly, so there's no lib/ build and npm publish ships src/
  • dependencies: 10 → 0
  • devDependencies: 6 → 3 (eslint, @eslint/js, prettier)
  • yarn.lock: 3,213 → 583 lines
  • Version bumped to 2.0.0, since ESM-only and Node >=24 are breaking
  • Fixed the stale exponent/hyperinstall repository/bugs URLs
removed replaced with
instapromise node:fs/promises
co async/await
rimraf@2 fs.rm({ recursive, force })
lodash language built-ins + util.isDeepStrictEqual
@exponent/promise-props, await-lock a few lines of local code (src/Lock.js)
yargs@6, minimist (unused) util.parseArgs + hand-written usage
npm-package-arg@4 src/localDeps.js — answers the one question hyperinstall asked it: is this specifier a local path?
fstream-npm@1 (deprecated) src/checksum.js — see below
@babel/cli, @babel/core, @babel/preset-env native ESM, no build step
eslint-config-universe@10 (pulls in the TypeScript and React plugin trees) eslint 9 flat config with js.configs.recommended

Hashing local dependencies

fstream-npm was used for one thing: list a local file: dependency's files so their contents can be checksummed and a change can trigger a reinstall. Its modern equivalent, npm-packlist, is driven from an @npmcli/arborist tree; it can be called with a stand-in for that tree, but it installs 10 packages and its ignore-walk descends the whole working tree — 30s to list react-native's 4,508 publishable files, versus 68ms for git ls-files (measured in a comment below).

src/checksum.js instead selects files the way a person thinks about a package:

  • git ls-files --cached --others --exclude-standard — the files Git knows about, ignored ones excluded. Hashed by content, so flipping between branches and back doesn't look like a change.
  • Files the files field of package.json publishes but Git ignores — how a package ships build output that isn't checked in — hashed by size + mtime, which catches a rebuild without reading gigabytes of artifacts.
  • No Git work tree, or the package itself is ignored → every file is read, skipping node_modules and VCS metadata. Nothing there distinguishes source from build output.

Reads and stats are bounded to 8 at a time. File names are hashed alongside their digests, so renames and deletions register as changes.

Cold-cache checksum of one local dependency, versus hashing every file in it:

package files walk everything this PR
spawn-async 12 4ms 20ms
expo-modules-core 1,031 128ms 73ms
expo-router 2,906 333ms 145ms
react-native 58,686 20,159ms 936ms

Known limits, all shared with the fstream-npm version: symlinked files aren't followed, file modes aren't hashed, submodule contents aren't descended into, and local dependencies of local dependencies aren't followed transitively.

Tests and CI

There were no tests. Added 34 covering the install decision logic (first install, no-op re-run, dependency/lockfile/cache-breaker changes, missing node_modules, local file: dep changes, --force/--clean, state pruning), the checksum file selection in and out of a Git work tree, local-specifier detection, and the CLI end to end. They use node:test, so no test-framework dependency. CI runs lint, format check, and tests on Node 24 and 26.

Beyond the suite, verified against a real project: init, first install (yarn for the package with a yarn.lock, npm for the other), a 50ms no-op re-run, a gitignored log in a local dependency causing no reinstall while a change to its published build/ output and to its src/ each caused exactly one, the generated npm-hyperinstall script, and clean.

Other behavior notes

  • The generated npm-hyperinstall script now quotes "$@" and $ROOT, and is chmoded 0755 so a restrictive umask can't leave it non-executable.
  • hyperinstall clean no longer fails when the state file or script is already gone.
  • Checksums changed shape, so packages with local dependencies get one reinstall on the first run after upgrading.
  • README: documents --clean (previously described as part of --force) and the Node 24 prerequisite.

expo-tuft Bot and others added 3 commits August 11, 2026 13:15
Hyperinstall's dependencies had aged into a stream of Dependabot PRs
while the program itself is a few hundred lines that Node can now run on
its own. This drops every runtime dependency and the Babel build step.

- engines: Node >=24; the source is plain ESM that runs directly, so
  there is no lib/ build and "npm publish" ships src/
- instapromise -> node:fs/promises
- co -> async/await
- rimraf -> fs.rm({ recursive, force })
- lodash -> language built-ins and util.isDeepStrictEqual
- @exponent/promise-props, await-lock -> a few lines of local code
- yargs, minimist (unused) -> util.parseArgs with hand-written usage
- npm-package-arg -> src/localDeps.js, which answers the one question
  hyperinstall asked it: is this specifier a local path?
- fstream-npm (deprecated) -> src/checksum.js, which hashes a local
  dependency's files directly, skipping node_modules and VCS metadata
- @babel/*, eslint-config-universe -> eslint 9 flat config + prettier 3

Dev dependencies are now eslint, @eslint/js, and prettier.

Also adds a test suite (node:test, no test framework dependency) and a
CI workflow running lint, format, and tests on Node 24 and 26.

Behavior notes:
- Local "file:" dependency checksums now cover every file in the
  dependency except node_modules and VCS metadata, instead of the exact
  set of files npm publish would include. This is a superset, so it
  never misses a change but may trigger one extra install.
- The generated npm-hyperinstall script now quotes "$@" and $ROOT, and
  is chmod'ed 0o755 so the process umask cannot leave it unexecutable.
- hyperinstall clean no longer fails when the state file or the script
  is already gone.

Co-authored-by: ide <379606+ide@users.noreply.github.com>
ESM-only and Node >=24 are breaking changes.

Co-authored-by: ide <379606+ide@users.noreply.github.com>
The first pass at replacing fstream-npm hashed every file under a local
dependency except node_modules and VCS metadata. That is a superset of
what npm publishes, so it never misses a change, but it also hashes
build output, caches, and logs. Measured against a react-native
checkout, one local dependency of ~59k files and 14 GB took 20s to
checksum -- hyperinstall's entire point is that an up-to-date run costs
a fraction of a second.

Files are now selected the way a person thinks about a package:

- "git ls-files --cached --others --exclude-standard" lists the files
  Git knows about, skipping anything ignored. These are hashed by
  content, so switching branches back and forth doesn't look like a
  change.
- Files that the "files" field of package.json publishes but Git ignores
  -- how packages ship build output that isn't checked in -- are hashed
  by size and modification time, which detects a rebuild without reading
  gigabytes of artifacts.
- Outside a Git work tree, or when the package itself is ignored, every
  file is read as before, since there is nothing to tell source apart
  from build output.

Reads and stats are bounded to 8 at a time, and entries that Git lists
but that can't be read (deleted-but-tracked files, submodules) are
skipped; because file names are hashed alongside their digests, their
disappearance is itself a change.

Checksum times for local dependencies, cold cache:

  spawn-async         12 files     4ms ->   20ms  (git adds a ~14ms spawn)
  expo-modules-core  1031 files  128ms ->   73ms
  expo-router        2906 files  333ms ->  145ms
  react-native      58686 files 20159ms -> 936ms

Co-authored-by: ide <379606+ide@users.noreply.github.com>
@expo-tuft

expo-tuft Bot commented Aug 11, 2026

Copy link
Copy Markdown
Author

Bumped to 2.0.0, and reworked how local file: dependencies are hashed after measuring the first approach — it was worse than I gave it credit for in the description above.

The problem. Replacing fstream-npm with "hash every file except node_modules and VCS metadata" is safe in the sense that it never misses a change, but it hashes build output, caches, and logs. Against a local react-native checkout that's 58,686 files and 14 GB: 20.2 seconds to checksum one dependency. Hyperinstall exists to make an up-to-date run cost ~0.2s.

It also means routine, meaningless churn (a rebuild, a stray npm-debug.log) triggers a full reinstall of every package that depends on it.

The fix. Select files the way a person thinks about a package:

  • git ls-files --cached --others --exclude-standard — the files Git knows about, ignored ones excluded. Hashed by content, so flipping between branches and back doesn't look like a change.
  • Files the files field of package.json publishes but Git ignores — how a package ships build output that isn't checked in — hashed by size + mtime. That still catches a rebuild without reading gigabytes of artifacts. (For react-native's files list, this is the difference between 936ms and 11.8s.)
  • No Git work tree, or the package itself is ignored → read everything, as before. Nothing there distinguishes source from build output, and an empty git ls-files result falls back rather than silently hashing nothing.

Reads and stats are bounded to 8 at a time. Entries Git lists but that can't be read (deleted-but-tracked files, submodule gitlinks) are skipped; since file names are hashed alongside their digests, a file disappearing is itself a change.

Cold-cache checksum of one local dependency:

package files before after
spawn-async 12 4ms 20ms
expo-modules-core 1,031 128ms 73ms
expo-router 2,906 333ms 145ms
react-native 58,686 20,159ms 936ms

The 16ms floor is the git ls-files spawn, paid once per local dependency and in parallel across packages.

Known limits, all of which the fstream-npm version shared or handled the same way: symlinked files aren't followed, file modes aren't hashed, and submodule contents aren't descended into. Local dependencies of local dependencies still aren't followed transitively.

5 more tests cover the Git paths: ignored files excluded, tracked and untracked changes detected, published-but-ignored build output detected (including a same-mtime-different-size case via utimes), and the ignored-package fallback. 34 tests total, green on Node 24 and 26.

Also verified end to end on a real project: a gitignored *.log in a dependency causes no reinstall, while a change to its published build/ output, a new file in build/, and a change to its src/ each cause exactly one — with a no-op run at 50ms.

@expo-tuft expo-tuft Bot changed the title Target Node 24 and remove all runtime dependencies Target Node 24, remove all runtime dependencies, 2.0.0 Aug 11, 2026
@expo-tuft

expo-tuft Bot commented Aug 11, 2026

Copy link
Copy Markdown
Author

One clarification on the table in my previous comment: "before" there is the intermediate walk-everything commit in this PR, not fstream-npm. To check that the rework is parity with what's on main rather than just a fix for my own regression, I ran the original algorithm (fstream-npm + per-file sha1 + hash of the sorted hashes) against the same packages:

package fstream-npm (main) walk everything this PR
spawn-async 5ms / 3 files 4ms / 12 files 20ms / 12 files
expo-modules-core 208ms / 899 files 128ms / 1,031 files 73ms / 1,031 files
expo-router 347ms / 2,186 files 333ms / 2,906 files 145ms / 2,906 files
react-native 1,193ms / 4,508 files 20,159ms / 58,686 files 936ms / 4,920 files

So fstream-npm was never the slow part — its publish-rule filtering was doing real work, which is what the walk-everything commit threw away. Git's ignore rules land in the same neighborhood (4,920 files vs 4,508 for react-native) at comparable or better cost. Tiny packages pay ~15ms more for the git ls-files spawn.

The rework also fixes a blind spot in the original that isn't about speed. It hashed the per-file checksums sorted, with the file names discarded (sortBy(fileChecksums) over an object yields its values), so any change that preserved the multiset of file contents was invisible. Verified against both implementations on a fixture:

rename a.js -> c.js          old changed: false   new changed: true
swap contents of two files   old changed: false   new changed: true

A rename in a local dependency therefore never triggered a reinstall. This PR hashes name\0digest pairs, so it does.

Two smaller things the original did that the rework drops: every file in the package was read concurrently with no bound (EMFILE risk on large packages — the new code caps at 8), and the end handler was an async callback whose rejection escaped the surrounding promise, so a file disappearing mid-walk surfaced as an unhandled rejection rather than a normal failure.

@expo-tuft

expo-tuft Bot commented Aug 11, 2026

Copy link
Copy Markdown
Author

Following up on why this doesn't use npm's current implementation, since "just use what npm uses" is the obvious question.

npm replaced fstream-npm with npm-packlist (npm 11.19 bundles v10.0.4; v11 is current). It's driven from an @npmcli/arborist tree — libnpmpack loads the tree and hands it to pacote, which calls packlist(tree). The constructor signature is still (tree, opts).

You can sidestep Arborist by passing a stand-in — packlist({ path, package, edgesOut: new Map() }) works and produces the same list fstream-npm did (4,508 files for react-native, 2,184 vs 2,186 for expo-router). So the "it needs Arborist" objection is softer than I put it in the description: npm-packlist alone installs 10 packages, not the 114 (23 MB) that a real Arborist brings.

The blocker is speed. npm-packlist uses ignore-walk, which descends the entire working tree and applies ignore rules per directory rather than pruning ignored directories up front:

package npm-packlist (list only) fstream-npm (list + hash) this PR
spawn-async 5ms 5ms 20ms
expo-router 419ms 347ms 145ms
react-native 30,335ms 1,193ms 936ms

30 seconds to enumerate 4,508 files, repeatable across runs, before hashing a single byte — because that package's working tree holds ~59k files of build output that ignore-walk walks into and then discards. That's the same pathology as the walk-everything commit, just inside a dependency.

So: 10 more packages to install and 30x slower on the case that matters, to compute a file list that git ls-files produces in 68ms. Sticking with Git.

Covers the fallback when the git binary is missing entirely, not just
when the package is outside a work tree: changes are still detected, and
ignored files count toward the checksum rather than being missed.

Co-authored-by: ide <379606+ide@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants