From 8a1293e8e7bee3f9555de3b50d0da93df3116368 Mon Sep 17 00:00:00 2001 From: NiveditJain Date: Wed, 19 Aug 2026 14:41:40 +0530 Subject: [PATCH 1/2] Retry bun run build on the release path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `bun --bun next build` is a demonstrated flake and it proved it in the worst place: publishing v1.0.1, bun 1.3.14 took a SIGSEGV during the TypeScript phase ("oh no: Bun has crashed. This indicates a bug in Bun, not your code"), exited 132, and took release-assets, publish, verify-install and announce down with it. Nothing about the code being released was wrong — the identical command had passed on the identical commit in ci.yml minutes earlier. ci.yml's build job has wrapped this in 3 attempts since it was written. publish.yml's two `bun run build` steps had none, so the path where a spurious failure costs the most was the one without a net. The publish job's build matters more than cli-tarball's: by then the release assets are already attached, so a crash there leaves a GitHub Release advertising daemon binaries whose npm package never shipped. release-pipeline.test.ts now fails on a bare `run: bun run build` anywhere in publish.yml, so this cannot silently reopen. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KbT6esM8A8mkSymH13keLt --- .github/workflows/publish.yml | 28 +++++++++++++++++++++++++-- CHANGELOG.md | 2 ++ __tests__/ci/release-pipeline.test.ts | 26 +++++++++++++++++++++++++ 3 files changed, 54 insertions(+), 2 deletions(-) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 1de5dfe5b..d61200262 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -355,8 +355,24 @@ jobs: PUBLISH_VERSION: ${{ needs.preflight.outputs.publish_version }} run: npm version "$PUBLISH_VERSION" --no-git-tag-version + # Retried, because `bun --bun next build` is a demonstrated flake and this + # is a release. On 2026-08-19 it took a SIGSEGV inside bun 1.3.14 during + # the TypeScript phase of v1.0.1 — "oh no: Bun has crashed. This indicates + # a bug in Bun, not your code" — exited 132, and took the whole release + # down with it: release-assets, publish, verify-install and announce all + # skipped on a crash that had nothing to do with the code being shipped. + # The identical command had passed on the identical commit in ci.yml + # minutes earlier, which is the definition of a retryable failure. + # + # ci.yml's `build` job has wrapped this in 3 attempts since it was + # written. The release path — the one where a spurious failure costs the + # most — was the half without it. - name: Build - run: bun run build + uses: nick-fields/retry@v4 + with: + max_attempts: 3 + timeout_minutes: 10 + command: bun run build - name: Pack the CLI tarball run: | @@ -590,8 +606,16 @@ jobs: # Order is unchanged — this still runs after `npm version` and after the # daemon artifacts land in RUNNER_TEMP (never the checkout, so Next's file # tracing cannot sweep them into `.next/standalone`). + # Retried for the same reason as cli-tarball's Build, and with more at + # stake: by this point the release assets are already attached, so a + # spurious bun crash here leaves a release advertising binaries whose + # package never shipped. - name: Build the tarball contents - run: bun run build + uses: nick-fields/retry@v4 + with: + max_attempts: 3 + timeout_minutes: 10 + command: bun run build - name: Publish env: diff --git a/CHANGELOG.md b/CHANGELOG.md index 7424e9e16..c7c8e7ca6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,8 @@ ### Fixes +- **Retry `bun run build` on the release path, which is the half that never had a net.** `bun --bun next build` is a demonstrated flake, and on 2026-08-19 it proved it in the worst place: publishing v1.0.1, bun 1.3.14 took a SIGSEGV during the TypeScript phase — `oh no: Bun has crashed. This indicates a bug in Bun, not your code` — exited 132, and took `release-assets`, `publish`, `verify-install` and `announce` down with it, all skipped. Nothing about the code being released was wrong; the identical command had passed on the identical commit in `ci.yml` minutes earlier, which is the definition of a retryable failure. `ci.yml`'s `build` job has wrapped this in three attempts since it was written, and publish.yml's two `bun run build` steps — `cli-tarball`'s and `publish`'s — had none, so the path where a spurious failure costs the most was the one without protection. The second one matters more than the first: by the time `publish` builds, the release assets are already attached, so a crash there leaves a GitHub Release advertising daemon binaries whose npm package never shipped. Both are retried now, and `release-pipeline.test.ts` asserts it rather than trusting anyone to remember — a bare `run: bun run build` anywhere in publish.yml fails the suite. (#727) + - **Stop CI paying for work it throws away, and stop a stalled apt mirror holding a release for six hours.** Four costs, found by measuring a green run rather than a red one. **`bun install` was running a full Next.js production build**: `package.json`'s `prepare` is `bun run build`, which bun fires as an install lifecycle hook, so six of `ci.yml`'s eight jobs spent ~28s (14s compile + 13s TypeScript) building an application they never read — and the `build` job did it twice, since its own Build step then re-ran the same thing warm in 7s. `rust-quality` has passed `--ignore-scripts` since it landed and installs in **one second**, which is the control that proves the rest; `translate-docs.yml` already guards the same way with a comment naming this exact hazard. Every install now does. **The cargo cache cost more to move than the work it replaced**: one `cargo-Linux-*` entry had reached **5,727 MB** — 57% of the repo's entire 10 GiB quota in a single key, which is the LRU-eviction pressure the previous fix here was about and did not remove — and restoring it took **127 seconds** against the 74s `cargo test` it existed to avoid. The cause is `path: target` taken literally: it archives every intermediate the workspace ever produced, including this workspace's own crates, which recompile in seconds and are the artifacts most likely to be stale. `Swatinem/rust-cache` caches the dependency artifacts and prunes the rest. **`rust-quality` ran in full on every pull request**, including the many that touch no Rust; its `Detect crates` gate was written for a stage-1 empty workspace and, with all three crates present, had been answering `true` unconditionally for months. It now also diffs the merge commit against its first parent, so it still reports a status — no `needs:` edge, nothing serialised behind it — while finishing in seconds on a TypeScript-only branch. `docs` gained the same gate, and its `mintlify` install is pinned to `4.2.680` to match `translate-docs.yml`, where floating meant an upstream release could redden a branch that changed nothing. **And 190 of 208 unit test files were building a jsdom they never touched**: the config set `environment: "jsdom"` globally for the sake of 16 React files and two more that already opt in per-file, and the `test` matrix runs the suite three times, so it was paid three times per run. Split into `node`/`dom` projects on the file extension, jsdom construction drops from **40.96s to 13.67s** measured locally, and a new `.test.tsx` still gets a DOM without anyone remembering to ask. The release hang is the same story in one step: **nothing in this repo had a job timeout except `integration-suite.yml`**, so when the linux-x64 daemon leg hit a stalled Azure mirror on 2026-08-19 it sat in `apt-get update` through three runner re-dispatches with v1.0.1 blocked behind it, while the arm64 leg ran the identical step in seconds. Every job across five workflows now declares one; the apt step is retried and given real acquire timeouts (its defaults are long enough to be no timeout at all against a stall) and loses its `-qq`, which was suppressing the one thing worth having in the log — which mirror stalled. `build-daemon.yml` also gains a concurrency group, scoped to `pull_request` so the `workflow_call` legs that *are* a release's binaries are never cancelled. `musl-tools` itself stays: `-p failproofaid` reaches `rusqlite` with `bundled` and `ring` through rustls, so the `cc` crate needs `musl-gcc` on both musl legs. **The bound is `sudo timeout`, not `nick-fields/retry`**, and that distinction is the whole fix rather than a style choice — the retry action was tried first and CI rejected it: it bounds a step by killing the process tree **as the runner user**, and apt runs as root, so the four-minute timeout fired exactly as designed and the action then died with `kill EPERM` instead of retrying, converting a recoverable stall into a failed leg. `timeout` inside the `sudo` makes the killer root too. The same run also showed the stall is real and not a one-off: every `azure.archive.ubuntu.com` line came back `Ign`, apt fell back to `archive.ubuntu.com`, fetched the InRelease files and then sat for three and a half minutes emitting nothing — so the step now tries `apt-get install` **before** `apt-get update` at all, since the refresh is the part that stalls and the runner image's package lists usually make it unnecessary. Dropping the `prepare` hook does cost the `test` job one thing it was silently getting: the custom-policy loader tests resolve `import ... from 'failproofai'` through `findDistIndex()` and need a real `dist/index.js`, so the job now builds that one bundle explicitly — three milliseconds against the ~28s it replaces. Worth recording how that surfaced, because the check that should have caught it did not: running the **whole suite** with `dist/` moved aside passes, since an earlier test writes the file before the loader tests read it, and only running them alone fails. Test-order luck read as a clean bill of health. None of these regressions turns CI red on its own either, which is why `release-pipeline.test.ts` now asserts all four — the timeouts, the `--ignore-scripts`, the absence of a bare `target` cache path, and an apt step that is bounded by a killer running as the same user apt does. Net: **~4.0 min wall and ~15.7 runner-minutes per pull request down to ~1.7 min and ~9**, with ~5 GiB of cache quota returned. (#726) - Let the box pick the translation model per tier, and stop a re-install double-scheduling the box. `getModelForTier` now reads `TRANSLATE_MODEL_TIER1` / `TRANSLATE_MODEL_TIER23`, so the seven languages most readers actually arrive in can keep a strong model while the long tail runs on something cheap — the CLI's `--model` flag flattens every tier to one model, which is the opposite of what the tier split exists for. Any id the gateway serves over the Anthropic `/v1/messages` shape works, since that is the API the translator speaks (verified: `deepseek-v4-pro` and `deepseek-v4-flash` both answer there). Separately, `install.sh` now strips the pre-marker cron form as well as its own marker: a box set up before the marker existed carries a long-form inline `docker run … -e CANARY_JOB=` line, and matching only the marker left it in place — six entries, every job scheduled twice, one on the old image and one on the new. The per-job flock keeps that from doing damage and turns it into something worse to diagnose: which image runs becomes a coin toss. Found on the real box, whose crontab is exactly that shape. (#705) diff --git a/__tests__/ci/release-pipeline.test.ts b/__tests__/ci/release-pipeline.test.ts index a962a26a8..9e0bd44b0 100644 --- a/__tests__/ci/release-pipeline.test.ts +++ b/__tests__/ci/release-pipeline.test.ts @@ -749,6 +749,32 @@ describe("CI cost guards", () => { expect(script).not.toContain("-qq"); }); + it("retries every `bun run build` on the release path", () => { + // `bun --bun next build` is a demonstrated flake: on 2026-08-19 it took a + // SIGSEGV inside bun 1.3.14 during v1.0.1's TypeScript phase, exited 132, + // and skipped release-assets/publish/verify-install/announce — a release + // lost to a crash in the toolchain rather than anything being released. + // The same command had passed on the same commit in ci.yml minutes before. + // + // ci.yml's `build` job had 3 attempts from the start; publish.yml's two + // build steps had none, so the path where a spurious failure costs the most + // was the one without a net. Asserted rather than remembered. + const unretried: string[] = []; + for (const [id, job] of Object.entries(workflow("publish.yml").jobs) as [string, Record][]) { + for (const step of job.steps ?? []) { + const command = String(step.with?.command ?? ""); + const bare = String(step.run ?? ""); + if (bare.includes("bun run build")) { + unretried.push(`${id}: ${String(step.name ?? "(unnamed)")}`); + } else if (command.includes("bun run build")) { + expect(String(step.uses)).toContain("nick-fields/retry"); + expect(step.with.max_attempts).toBeGreaterThanOrEqual(2); + } + } + } + expect(unretried).toEqual([]); + }); + it("supersedes a superseded daemon build without cancelling a release", () => { const c = workflow("build-daemon.yml").concurrency; expect(c.group).toContain("github.ref"); From ae9d4a360ba6e0595800a35132dd55f08b9ed861 Mon Sep 17 00:00:00 2001 From: NiveditJain Date: Wed, 19 Aug 2026 14:42:36 +0530 Subject: [PATCH 2/2] Point the changelog entry at the real PR number Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01KbT6esM8A8mkSymH13keLt --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c7c8e7ca6..c3d86c644 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,7 +14,7 @@ ### Fixes -- **Retry `bun run build` on the release path, which is the half that never had a net.** `bun --bun next build` is a demonstrated flake, and on 2026-08-19 it proved it in the worst place: publishing v1.0.1, bun 1.3.14 took a SIGSEGV during the TypeScript phase — `oh no: Bun has crashed. This indicates a bug in Bun, not your code` — exited 132, and took `release-assets`, `publish`, `verify-install` and `announce` down with it, all skipped. Nothing about the code being released was wrong; the identical command had passed on the identical commit in `ci.yml` minutes earlier, which is the definition of a retryable failure. `ci.yml`'s `build` job has wrapped this in three attempts since it was written, and publish.yml's two `bun run build` steps — `cli-tarball`'s and `publish`'s — had none, so the path where a spurious failure costs the most was the one without protection. The second one matters more than the first: by the time `publish` builds, the release assets are already attached, so a crash there leaves a GitHub Release advertising daemon binaries whose npm package never shipped. Both are retried now, and `release-pipeline.test.ts` asserts it rather than trusting anyone to remember — a bare `run: bun run build` anywhere in publish.yml fails the suite. (#727) +- **Retry `bun run build` on the release path, which is the half that never had a net.** `bun --bun next build` is a demonstrated flake, and on 2026-08-19 it proved it in the worst place: publishing v1.0.1, bun 1.3.14 took a SIGSEGV during the TypeScript phase — `oh no: Bun has crashed. This indicates a bug in Bun, not your code` — exited 132, and took `release-assets`, `publish`, `verify-install` and `announce` down with it, all skipped. Nothing about the code being released was wrong; the identical command had passed on the identical commit in `ci.yml` minutes earlier, which is the definition of a retryable failure. `ci.yml`'s `build` job has wrapped this in three attempts since it was written, and publish.yml's two `bun run build` steps — `cli-tarball`'s and `publish`'s — had none, so the path where a spurious failure costs the most was the one without protection. The second one matters more than the first: by the time `publish` builds, the release assets are already attached, so a crash there leaves a GitHub Release advertising daemon binaries whose npm package never shipped. Both are retried now, and `release-pipeline.test.ts` asserts it rather than trusting anyone to remember — a bare `run: bun run build` anywhere in publish.yml fails the suite. (#728) - **Stop CI paying for work it throws away, and stop a stalled apt mirror holding a release for six hours.** Four costs, found by measuring a green run rather than a red one. **`bun install` was running a full Next.js production build**: `package.json`'s `prepare` is `bun run build`, which bun fires as an install lifecycle hook, so six of `ci.yml`'s eight jobs spent ~28s (14s compile + 13s TypeScript) building an application they never read — and the `build` job did it twice, since its own Build step then re-ran the same thing warm in 7s. `rust-quality` has passed `--ignore-scripts` since it landed and installs in **one second**, which is the control that proves the rest; `translate-docs.yml` already guards the same way with a comment naming this exact hazard. Every install now does. **The cargo cache cost more to move than the work it replaced**: one `cargo-Linux-*` entry had reached **5,727 MB** — 57% of the repo's entire 10 GiB quota in a single key, which is the LRU-eviction pressure the previous fix here was about and did not remove — and restoring it took **127 seconds** against the 74s `cargo test` it existed to avoid. The cause is `path: target` taken literally: it archives every intermediate the workspace ever produced, including this workspace's own crates, which recompile in seconds and are the artifacts most likely to be stale. `Swatinem/rust-cache` caches the dependency artifacts and prunes the rest. **`rust-quality` ran in full on every pull request**, including the many that touch no Rust; its `Detect crates` gate was written for a stage-1 empty workspace and, with all three crates present, had been answering `true` unconditionally for months. It now also diffs the merge commit against its first parent, so it still reports a status — no `needs:` edge, nothing serialised behind it — while finishing in seconds on a TypeScript-only branch. `docs` gained the same gate, and its `mintlify` install is pinned to `4.2.680` to match `translate-docs.yml`, where floating meant an upstream release could redden a branch that changed nothing. **And 190 of 208 unit test files were building a jsdom they never touched**: the config set `environment: "jsdom"` globally for the sake of 16 React files and two more that already opt in per-file, and the `test` matrix runs the suite three times, so it was paid three times per run. Split into `node`/`dom` projects on the file extension, jsdom construction drops from **40.96s to 13.67s** measured locally, and a new `.test.tsx` still gets a DOM without anyone remembering to ask. The release hang is the same story in one step: **nothing in this repo had a job timeout except `integration-suite.yml`**, so when the linux-x64 daemon leg hit a stalled Azure mirror on 2026-08-19 it sat in `apt-get update` through three runner re-dispatches with v1.0.1 blocked behind it, while the arm64 leg ran the identical step in seconds. Every job across five workflows now declares one; the apt step is retried and given real acquire timeouts (its defaults are long enough to be no timeout at all against a stall) and loses its `-qq`, which was suppressing the one thing worth having in the log — which mirror stalled. `build-daemon.yml` also gains a concurrency group, scoped to `pull_request` so the `workflow_call` legs that *are* a release's binaries are never cancelled. `musl-tools` itself stays: `-p failproofaid` reaches `rusqlite` with `bundled` and `ring` through rustls, so the `cc` crate needs `musl-gcc` on both musl legs. **The bound is `sudo timeout`, not `nick-fields/retry`**, and that distinction is the whole fix rather than a style choice — the retry action was tried first and CI rejected it: it bounds a step by killing the process tree **as the runner user**, and apt runs as root, so the four-minute timeout fired exactly as designed and the action then died with `kill EPERM` instead of retrying, converting a recoverable stall into a failed leg. `timeout` inside the `sudo` makes the killer root too. The same run also showed the stall is real and not a one-off: every `azure.archive.ubuntu.com` line came back `Ign`, apt fell back to `archive.ubuntu.com`, fetched the InRelease files and then sat for three and a half minutes emitting nothing — so the step now tries `apt-get install` **before** `apt-get update` at all, since the refresh is the part that stalls and the runner image's package lists usually make it unnecessary. Dropping the `prepare` hook does cost the `test` job one thing it was silently getting: the custom-policy loader tests resolve `import ... from 'failproofai'` through `findDistIndex()` and need a real `dist/index.js`, so the job now builds that one bundle explicitly — three milliseconds against the ~28s it replaces. Worth recording how that surfaced, because the check that should have caught it did not: running the **whole suite** with `dist/` moved aside passes, since an earlier test writes the file before the loader tests read it, and only running them alone fails. Test-order luck read as a clean bill of health. None of these regressions turns CI red on its own either, which is why `release-pipeline.test.ts` now asserts all four — the timeouts, the `--ignore-scripts`, the absence of a bare `target` cache path, and an apt step that is bounded by a killer running as the same user apt does. Net: **~4.0 min wall and ~15.7 runner-minutes per pull request down to ~1.7 min and ~9**, with ~5 GiB of cache quota returned. (#726)