feat(cli): use upstream @dfinity/pic, deprecate pocket-ic below 9.0.0 - #642
Conversation
`pocket-ic` 9.0.0 and newer now runs through upstream `@dfinity/pic` 0.23.0 instead of the `pic-js-mops` fork. That fork has no public source repository — it is published from a local checkout of `dfinity/pic-js` with four patches, so the only way to audit it is diffing the published tarball, and one person can publish a fix. Two of the patches are now upstream (dfinity/pic-js#276 for an explicit binary path, #278 for a server `--ttl`), the third needs no upstream change (`serverProcess` is TS-private but present at runtime, so the client seam casts), and the fourth is handled here. `@dfinity/pic` stays a devDependency, pre-bundled into `dist/vendor/`: its postinstall downloads a ~94 MB pocket-ic binary and throws when it cannot, and the published CLI is the unbundled `dist/` tree with the full dependency list, so a `dependencies` entry would make `npm i -g ic-mops` pull that binary and fail outright with no network. mops manages the binary itself via `[toolchain] pocket-ic`. Removable later: if pic makes the download lazy, this becomes an ordinary dependency again. Pins below 9.0.0 keep working through the unchanged `pic-ic` client but now warn, and go away in v3 — the same deprecate-then-drop path used for the dfx replica and vessel. `addCycles` gets a wrapper rather than a cast, because it takes a `number` on `pic-ic` and a `bigint` upstream and an intersection type would compile while feeding the wrong one at runtime. Verified with a packed CLI installed into a throwaway prefix: with `pocket-ic = "12.0.0"` both `mops test --mode replica` and `mops bench` pass, spawning the toolchain-managed binary with `--ttl`, and the install downloads no binary; with `pocket-ic = "4.0.0"` the replica test still passes and warns. `@icp-sdk/core` moves to 5.4.0 as a single deduped copy (pic wants ^5, and 6.x would nest a second copy and give two `Principal`/`IDL` identities).
Splits `NEXT-MAJOR.md` into a committed v3 scope and a deferred section so nothing is lost, and records the reasoning inline: - Keep today's version model. Bare `1.2.3` stays exact, conflicts stay max-wins, freshness keeps coming from caret-bounded `mops update`. Cargo's model needs `=` pins as an escape hatch, and `=` in published packages needs backend validator changes plus a window where older CLIs cannot parse what newer ones publish, so that track moves to a future major gated on moc `--override`. v3 keeps the pure wins: a real semver comparator and an always-on cross-major conflict warning. - `--locked` instead of a `mops ci` command, and drop `--lock`: `check` becomes `--locked`, `ignore` has no successor, `update` becomes self-healing on a corrupt lock. - Lockfile v4 gets its own read-early/write-later track, decoupled from the CLI major. - dfx: remove every implicit fallback, keep `--replica dfx` explicit until our own dev loop is on icp-cli. - The `@dfinity/pic` switch ships in 2.x; only dropping `pic-ic` and enforcing a supported server range stay in v3. TODO.md picks up the `files: ["*"]` packaging bloat found along the way and drops the stale `mops ci` phrasing.
9f1308c to
2e8aff0
Compare
Cursor AI review👍 APPROVE — looks safe to merge
VerdictDecision: APPROVE Generated for commit a281d55 |
automation-sa-sre
left a comment
There was a problem hiding this comment.
Automated approval: the AI review verdict for 2e8aff0 is APPROVE. See the "Cursor AI review" comment for details.
Moving mops to `@icp-sdk/core` 5.4.0 broke every replica test that runs on the `dfx` or `dfx-pocket-ic` replica. 5.x calls `/api/v4/.../call`, `/api/v3/.../query` and `/api/v3/.../read_state`; dfx 0.29.1's HTTP gateway serves only the `v2` endpoints. It answers `/api/v4` with the gateway's own 400 "canister id not resolved" HTML page rather than a 404, so the agent's v4-to-v2 fallback — which keys on 404 — never fires, and `query`/`read_state` have no fallback at all. The failure surfaced as an 8 KB HTML error page dumped into the test output. The bump was only ever needed because `@dfinity/pic` depends on `@icp-sdk/core` ^5, and `vendor:pic` left that dependency external. It is a plain dependency, not a peer, so the vendored bundle can carry its own copy instead. Nothing crosses the two copies: `idlFactory` receives pic's `IDL` as an argument, and the `canisterId` pic returns only goes back into pic or through `.toText()`. This also realigns the CLI with `frontend/`, which was left on 4.0.2. Dropping `--external:@icp-sdk/core` grows `dist/vendor/pic.mjs` from 600 KB to 1.1 MB — one file in the tarball, against the ~94 MB postinstall download that vendoring avoids. Verified with the dist build installed into a throwaway prefix: the root `mops test` sweep passes all 18 files (both `test/storage-actor.test.mo` and `cli/tests/pocket-ic/test/hello.test.mo` failed before), `pocket-ic = "12.0.0"` passes on `@dfinity/pic` for both `mops test` and `mops bench`, `pocket-ic = "4.0.0"` still passes on `pic-ic` and still warns, and unpinned `mops bench` still runs on the dfx replica with its deprecation warning.
`toolchain use pocket-ic` still referred to a "bundled pic-js client", which is now upstream `@dfinity/pic` vendored into `dist/`, and the skill never stated the supported version floor. Adds the `9.0.0` floor and the deprecation of lower pins. Unpinned behavior is unchanged — replica tests and benchmarks still fall back to the deprecated `dfx` replica with a warning — so the guidance at the top and under replica tests stays as it is.
automation-sa-sre
left a comment
There was a problem hiding this comment.
Automated approval: the AI review verdict for 72a0660 is APPROVE. See the "Cursor AI review" comment for details.
The spike concluded that 4.0.2 -> 5.4.0 was clean. It is not: 5.x drops the IC HTTP API v2 endpoints, which are all the dfx replicas serve, so unpinned replica tests and benchmarks break. CI caught it. mops keeps 4.0.2 and bundles pic's own 5.x, and the bump belongs with the dfx-replica removal. Follow-ups filed as #651, #652, #655, #657.
automation-sa-sre
left a comment
There was a problem hiding this comment.
Automated approval: the AI review verdict for a281d55 is APPROVE. See the "Cursor AI review" comment for details.
`decompress` is a critical, unfixable advisory in the CLI's dependency tree, and the plan doc said dropping vessel would take it with it. Neither half of that was right, so both are corrected here along with the parts of the plan that #642 made stale. ## `decompress` has to go, and vessel won't take it `cli/package.json` pins `decompress` 4.2.1. Both advisories against it cover `<=4.2.1`, so the newest release is still affected and there is nothing to upgrade to: ``` $ npm audit critical decompress Decompress: Archive extraction can create files and links outside of the target directory decompress: Arbitrary File Write via Archive Extraction (Zip Slip) fixAvailable: false ``` [GHSA-mp2f-45pm-3cg9](GHSA-mp2f-45pm-3cg9), [GHSA-h39j-r5qq-r9mm](GHSA-h39j-r5qq-r9mm). Removing the dependency is the only remedy, and there are two call sites — the toolchain extractor (`toolchain-utils.ts:64`, for moc/wasmtime/pocket-ic/lintoko archives) and `installFromGithub` (`vessel.ts:167`, for `repo = "..."` deps). The second one is the exposed surface: those archives come from arbitrary user-specified repos, which is precisely the zip-slip threat model. Filed in `TODO.md` rather than here, because replacing an extractor breaks nothing and should not wait for a major. ## `installFromGithub` must move, not be deleted The vessel item listed `installFromGithub` among the things v3 deletes. It lives in `cli/vessel.ts` for historical reasons only — `cli/commands/install/install-dep.ts:23` and `cli/commands/add.ts:104` call it for ordinary GitHub deps, which mops supports and which the lockfile-v4 track plans to improve with commit-SHA pinning. Deleting it would break git deps; it moves into `cli/commands/install/` instead. This is also why dropping vessel does not resolve the advisory on its own: that `decompress` call site outlives vessel. ## pic migration is history now #642 shipped the `@dfinity/pic` switch, so the long spike narrative is replaced by current state and the three follow-up issues it produced ([#651](#651) vendor bundling, [#652](#652) `@icp-sdk/core` 5.x, [#655](#655) log hook). Only removing the legacy `pic-ic` client is still v3 work, tracked in [#653](#653). Provenance notes are kept in condensed form, since the reason the fork existed is worth not relearning. ## Docs only No code changes. Nothing here is an argument for cutting v3 sooner — the decompress work deliberately lands outside it.
Closes #561.
pocket-ic9.0.0 and newer now runs through upstream@dfinity/pic0.23.0. Nothing changes for anyone on a supported pin: the same toolchain-managed binary starts with the same--ttl, and installing the CLI still downloads no extra binary. Pins below 9.0.0 keep working but warn, and stop working in v3.Why replace the fork
pic-js-mopshas no public source repository. It is published to npm from a local checkout ofdfinity/pic-jscarrying four patches, so the only way to audit it is diffing the published tarball against upstream, and exactly one person can publish a fix.Two of those patches are now upstream and shipped together in 0.23.0: dfinity/pic-js#276 for an explicit binary path and dfinity/pic-js#278 for a server
--ttl. The third needs no upstream change —serverProcessis TS-private but present at runtime, so the client seam casts through it for canister-log streaming. The fourth was a stripped postinstall, handled below.Why it is a devDependency, not a dependency
@dfinity/picdownloads a ~94 MB pocket-ic binary in a postinstall that throws when it cannot. The publishedic-mopspackage is the unbundleddist/tree carrying the full dependency list — the bun bundle is a separate artifact that onlycli.mops.oneinstalls — so adependenciesentry reaches every user. Installing into a throwaway prefix with egress blocked:mops manages that binary itself via
[toolchain] pocket-ic, so the download was never wanted. Three pieces replace it:vendor/pic-entry.mjsre-exportsPocketIcandPocketIcServerexplicitly, because pic ships CJS and a plainexport *through esbuild yields no named exports at all;vendor:picbundles that todist/vendor/pic.mjs(1.1 MB); andfix-distrepoints the dynamic import, throwing if it is ever renamed.import("pic-ic")is deliberately left alone.Removable later — if pic makes the download lazy, as suggested during review of #276, all three pieces and their smoke test go away.
Deprecating pins below 9.0.0
Before, a
pocket-ic = "4.0.0"pin ran with no notice. After:The warning fires where the legacy client is selected rather than per command, so it cannot drift from the selection logic and prints once per run.
One subtlety worth calling out:
addCyclestakes anumberonpic-icand abigintupstream. The intersection cast that reconciled the two old clients would still compile here and then hand anumberto the modern client at runtime, so the seam exports a wrapper that branches on the same pin that picked the client.Migration
Bump the pin:
mops toolchain use pocket-ic 12.0.0.The affected set is narrower than the warning implies.
pic-ic@0.5.4only speaks the 4.0.0 server protocol, so 5.x–8.x pins already fail withBinTimeoutErroronmaintoday — only 4.0.0 and 9.x+ ever worked, matching the range documented in #412. In practice this affects 4.0.0 pins.@icp-sdk/corestays on 4.0.2pic depends on
@icp-sdk/core^5, but as a plain dependency, not a peer — so it does not dictate mops's version. mops stays on 4.0.2 and the vendored bundle carries pic's own5.xinside it.That pin is load-bearing, not incidental.
@icp-sdk/core5.x drops the IC HTTP APIv2endpoints, andv2is the only version thedfxanddfx-pocket-icreplicas serve — dfx 0.29.1 answers/api/v4/…/callwith a 400 and an HTML error page, and 404sv3query/read_state. The SDK's fallback keys on 404 forv4only, so it never fires, andquery/read_statehave no fallback at all. Bumping mops to 5.x therefore breaks every replica test and benchmark in a project with nopocket-icpin, which is how the unpinned path is still wired. That is also what made CI red on an earlier revision of this branch.Two copies are safe here because nothing crosses the seam:
idlFactoryis the caller-injected({ IDL }) => …form, so pic supplies its ownIDL, and thecanisterIdpic returns only ever goes back into pic or through.toText(). Worth keeping in mind for future edits — handing a mops-constructedPrincipalto pic would be the way to break it. Keeping 4.0.2 also realigns the CLI withfrontend/, and leaves the SDK bump with the dfx-replica removal in v3, where it belongs.What is unchanged
Supported pins behave identically — same binary path, same
--ttl, same output. Unpinned projects still fall back to the deprecated dfx replica with its existing warning.pic-icitself is untouched. The bundledcli.mops.onedistribution needed no changes, since it already inlines pic fromnode_modules.Verified against a packed CLI in a throwaway prefix rather than a dev checkout, across all three replica paths: the root unpinned
mops testsweep passes all 18 files on the dfx replica;pocket-ic = "12.0.0"passesmops test --mode replicaandmops benchspawning<cache>/pocket-ic/12.0.0/pocket-ic --port-file … --ttl 60with no warning and no binary downloaded at install;pocket-ic = "4.0.0"passes the replica test with the warning above, spawning throughpic-ic's older--pidstyle. Legacymops benchis confirmed only as far as a successful client-to-server round trip — a 2024-era replica rejects modern moc output, and older moc cannot parsepersistent actor.Follow-ups worth flagging
npm installincli/still pulls the 94 MB binary, because devDependency install scripts run for local dev. The lazy-download change upstream fixes that too.@icp-sdk/corecosts ~500 KB indist/vendor/pic.mjs. It collapses back to one copy when v3 drops the dfx replica and mops can move to 5.x itself.pic-ic, theAnyPocketIc*union types it forces, and turns this warning into a hard error alongside an enforced supported server range.files: ["*"]ships both distributions inside the npm tarball, roughly doubling it. Tracked inTODO.md.Also here: v3 plan rescope
NEXT-MAJOR.mdis now split into a committed v3 scope and a deferred section, with the reasoning recorded inline:1.2.3remains exact, conflicts stay max-wins, freshness keeps coming from caret-boundedmops update. Cargo's model needs=pins as an escape hatch, and=in published packages requires backend validator changes plus a window where older CLIs cannot parse what newer ones publish — caret without the escape hatch is the worst of both. That track moves to a future major, gated on moc--overrideso multi-version coexistence lands with it. v3 keeps the pure wins: a real semver comparator instead ofparseInt, and a cross-major conflict warning that always fires.--locked, not amops cicommand. npm's separate command is the ecosystem outlier; cargo, pnpm and yarn all use a flag, and a flag composes with the commands that resolve implicitly.--lockthen disappears entirely:checkbecomes--locked,ignorehas no successor, andupdatebecomes self-healing on a corrupt lock.--replica dfxstays until our own dev loop is on icp-cli.