Speed up the per-PR Lighthouse audit by jeffmerrick · Pull Request #20728 · pulumi/docs

jeffmerrick · 2026-08-05T18:56:50Z

Trying to speed up the build a bit!

Changes

One Chrome instead of six. scripts/run-lighthouse-pr.sh shelled out to npx lighthouse once per page/device combination, paying a cold Chrome boot each time. scripts/lighthouse/run-audits.mjs drives the Node API instead and runs all six against a single launched browser. Lighthouse still calls newPage() per run, closes it after, and clears storage, so the audits stay independent.

No screenshots. The report only reads the performance score plus FCP/LCP/TBT/CLS/SI, so every captured screenshot was encoded and thrown away. The expensive one is the full-page screenshot, and it can't be dropped via skipAudits — full-page-screenshot is a gatherer id, not an audit id, and FullPageScreenshot is in Lighthouse's filterResistantArtifactIds, so it runs regardless of which audits survive filtering. disableFullPageScreenshot is the only lever. The two real screenshot audits are skipped alongside it.

One page list. The audited pages move to scripts/lighthouse/pages.json, read by both the runner and the renderer, so the two can't drift.

chrome-launcher becomes a direct devDependency since run-audits.mjs imports it; it was already present as a Lighthouse transitive dep, and yarn.lock needed no change.

Numbers

Measured on ubuntu-latest against real preview buckets, bounded by the audit's own log markers so old and new are measured identically.

135s → 82s, ~39% faster, straight off buildSite's critical path.
Roughly three-quarters of that came from sharing the Chrome instance (135s → 97s); the rest from dropping the full-page screenshot (97s → 82s).

Single runs, not medians — see Follow-up.

Verified

The audit ran green on this PR with all six rows scoring normally and none falling back to an Error row, using a throwaway theme/ touch to trip the path filter (reverted; the diff above is unaffected). The disableFullPageScreenshot behavior was confirmed directly against Lighthouse 12.8.2 by resolving both the mobile and desktop configs through initializeConfig: the FullPageScreenshot artifact is present before the change and absent after, with the performance category and desktop form factor intact.

Considered and dropped: a separate `lighthouse` job

Moving the audit to its own job made nothing faster in absolute terms: the fresh job re-paid ~55s of setup it had been inheriting from buildSite, so total runner time went up and only the gating check's report time improved. At an 11% trigger rate (13 of the last 120 merged PRs), that bought 82s on 1 in 9 PRs in exchange for an artifact hand-off, a duplicate ESC fetch, and an npm install that scraped yarn.lock and symlinked node_modules. Cleanly additive later if it's ever worth it.

Follow-up

Median-of-3 is what would make these numbers trustworthy — the same page scored 89, 51, 45 and 34 across runs here. It's ~12 lines in run-audits.mjs, but it triples the audit load, so it belongs in its own PR.

🤖 Generated with Claude Code

The Lighthouse audit is advisory and gates nothing, but it ran at the tail of ci-pull-request.sh, so its 3-5 minutes were fully additive to the latency of buildSite -- the check people actually wait on. Extract it into its own `lighthouse` job in pull-request.yml. The job consumes the origin-bucket-metadata artifact buildSite already publishes, keeps the path filter as the identical bash regex list (no glob translation to drift), and installs only Lighthouse at the version yarn.lock resolves rather than running `make ensure`. It carries continue-on-error so a failure can't turn the run red, preserving the behavior of the `|| true` that wrapped the audit before. Also trim the audit itself. scripts/lighthouse/run-audits.mjs runs all six page/device combinations against a single shared Chrome instead of booting a cold one per audit via npx, and skips the three screenshot audits -- the report only reads the score plus FCP/LCP/TBT/CLS/SI, so the screenshots were captured, encoded, and thrown away. Measured against production: 98.7s -> 72.5s. The page list moves to scripts/lighthouse/pages.json so the runner and the renderer can't drift. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Pushing this branch triggers nothing else (both push-triggered workflows are master-only), so this exercises the new job's plumbing -- ESC secrets in a standalone job, the artifact hand-off, the path filter against three merged PRs with known file lists, the yarn.lock version extraction, and Chrome on ubuntu-latest -- without opening a PR or posting a bot comment. Delete before marking the real PR ready. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

It served its purpose: ESC secrets resolve in a standalone job with environment: testing, the buildSite -> lighthouse artifact hand-off works, the path filter matched all three merged-PR fixtures, and six audits ran in 99s on ubuntu-latest against a 36s Lighthouse install. What it couldn't reach -- the real preview URL and the comment post/update path -- needs a pull_request payload, which this PR provides. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The rest of this PR only touches .github/ and scripts/, none of which match the audit's path filter -- so without this the new job would correctly skip and prove nothing. Revert this commit before marking the PR ready. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

pulumi-bot · 2026-08-05T19:12:20Z

Your site preview for commit 43530e0 is ready! 🎉

http://www-testing-pulumi-docs-origin-pr-20728-43530e0b.s3-website.us-west-2.amazonaws.com

pulumi-bot · 2026-08-05T19:16:52Z

Lighthouse Performance Report

Commit: 51fdfb2 | Metric definitions

Page	Device	Score	FCP	LCP	TBT	CLS	SI
Homepage	Mobile	🟡 50	3.0s	4.6s	834ms	0.092	6.9s
Homepage	Desktop	🟢 94	0.8s	1.0s	169ms	0.014	0.9s
Install Pulumi	Mobile	🟡 57	7.5s	16.2s	124ms	0.000	7.5s
Install Pulumi	Desktop	🟡 86	1.1s	1.6s	0ms	0.147	1.1s
AWS Get Started	Mobile	🟡 59	6.1s	15.6s	87ms	0.000	6.1s
AWS Get Started	Desktop	🟢 93	1.1s	1.5s	0ms	0.019	1.1s

This reverts commit c3cd6b0.

github-actions · 2026-08-05T20:16:46Z

Pre-merge Review — Last updated 2026-08-06T14:10:17Z

Tip

Summary: This is an infrastructure PR. An earlier commit on this branch split the advisory Lighthouse audit into its own lighthouse job in .github/workflows/pull-request.yml (that's the revision the previous review covered, at d96e124); a later commit on the same branch reverted that split, so the diff against master now nets down to just the audit speedups: chrome-launcher added as a direct devDependency, the page list moved to scripts/lighthouse/pages.json, the runner rewritten as scripts/lighthouse/run-audits.mjs (one shared Chrome instance instead of six npx lighthouse cold boots, screenshots skipped), and scripts/run-lighthouse-pr.sh updated to read that shared page list via jq instead of hardcoded bash arrays. The failure mode that matters here is the bash→Node hand-off silently breaking or falling back to an all-Error report: the JSON filenames run-audits.mjs writes (<page.key>-<device>.json) match exactly what run-lighthouse-pr.sh reads back via the same pages.json; chrome-launcher@^1.2.0 was already resolved in yarn.lock by an existing transitive requirement at the identical semver range, so no lockfile change was needed; and the revised disableFullPageScreenshot: true setting — the actual fix in this revision, replacing the previous skipAudits-only approach that can't suppress the filter-resistant full-page-screenshot gatherer — is wired through correctly. Because the last-reviewed commit is no longer an ancestor of HEAD (the branch was rewritten, not just pushed forward), this run re-reads the whole current diff rather than a delta. .github/workflows/pull-request.yml and scripts/ci-pull-request.sh are unchanged from master in the final diff, so the job-wiring risk flagged last time no longer has anything to attach to.

Review confidence:

Dimension	Level	Notes
mechanics	HIGH	Filename hand-off, `pages.json` consumption, and `yarn.lock` resolution all cross-checked by reading; CI's build/test checks are green on this head.
facts	HIGH	Both extracted claims (page paths in `pages.json`) reused from the previous review — the file is byte-identical to the version already verified.
infra	MEDIUM	None of the files in this diff match the audit's own trigger paths (`layouts/`, `theme/`, `static/*`, `config/`, `hugo.`), so this head's own CI run wouldn't have re-executed the Lighthouse audit; the author's independent verification (a reverted throwaway `theme/` touch, plus direct `initializeConfig` resolution against Lighthouse 12.8.2) is the evidence for the runtime behavior, not this review's execution.

Investigation log

Cross-sibling reads: not run (not in a templated section)
External claim verification: 2 of 2 claims verified (0 unverifiable, 0 contradicted) · 4 specialists (numerical, cross-reference, capability, framing); 0 cross-specialist corroborations · routed: 2 inline, 0 Pass 1, 0 Pass 2, 0 Pass 3.
Cited-claim spot-checks: not run (no cited claims)
Frontmatter sweep: not run (no frontmatter in diff)
Temporal-trigger sweep: not run (no trigger words)
Code execution: not run (no static/programs/ change)
Code-examples checks: not run (no fenced code blocks in content files)
Editorial-balance pass: not run (not under content/blog/)

🚨 Outstanding	⚠️ Low-confidence	💡 Pre-existing	✅ Resolved
0	0	0	1

🔍 Verification trail

2 claims extracted · 2 verified · 0 unverifiable · 0 contradicted

L10 in scripts/lighthouse/pages.json "'path': '/docs/install/'" → ✅ verified (evidence: reused from the previous review — content/docs/install/_index.md exists and renders at /docs/install/, matching the lighthouse pages.json entry unchanged since last review; source: repo:content/docs/install/_index.md)
L15 in scripts/lighthouse/pages.json "'path': '/docs/iac/get-started/aws/'" → ✅ verified (evidence: reused from the previous review — content/docs/iac/get-started/aws/_index.md exists with no path override, confirming /docs/iac/get-started/aws/, matching the lighthouse pages.json entry unchanged since last review; source: repo:content/docs/iac/get-started/aws/_index.md)

🚨 Outstanding in this PR

No outstanding findings in this PR.

⚠️ Low-confidence

No low-confidence findings in this PR.

💡 Pre-existing issues in touched files (optional)

No pre-existing issues in touched files.

✅ Resolved since last review

[L201] .github/workflows/pull-request.yml — the "Install Lighthouse" step whose npm install scoping was flagged last review (resolved in 43530e0 — the separate lighthouse job it belonged to was dropped from the branch entirely; that file's diff against master is now empty, so this step isn't part of the PR anymore).

📜 Review history

2026-08-05T20:13:03Z — Reviewed the Lighthouse job split — artifact hand-off, permissions, path filter, and report loop all check out; one low-confidence note on whether the "Install Lighthouse" step installs only Lighthouse (d96e124)
2026-08-06T14:10:17Z — History rewritten since last review (the branch's own later commit reverted the separate lighthouse job split reviewed at d96e124); re-reviewed the full current diff against HEAD. Previous low-confidence item no longer applies — moved to ✅ Resolved. The revised disableFullPageScreenshot setting in run-audits.mjs and its hand-off with run-lighthouse-pr.sh's pages.json-driven report loop check out; no new findings (43530e0)

Refresh this review — comment @claude #update-review. Say what you fixed, or which finding you dispute and why; both work in the same mention.
Ask for anything else — comment @claude with no hashtag (questions, one-off fixes). Leaves this review untouched.

Important

Please don't hide, resolve, or delete this comment! It breaks things!

📖 How pre-merge review works — the full lifecycle, short-circuits, and escape hatches.

`npm install <pkg>` reifies the whole tree from whatever package.json it finds, so `npm install lighthouse@X` in the checkout root installed all 913 root dependencies rather than just Lighthouse. Running the install under an --prefix outside the checkout drops it to the 200 packages Lighthouse actually needs; symlinking the result back to ./node_modules keeps Node's resolution working for scripts/lighthouse/run-audits.mjs. Measured on ubuntu-latest, the install step goes from 39s (913 packages) to 9s (200), taking the lighthouse job from 3m00s to 2m25s. The audits are untouched: 97s before, 95s after. Also declare chrome-launcher, which run-audits.mjs imports directly but which was resolving only as a hoisted transitive dependency of lighthouse. The range matches the descriptor already keyed in yarn.lock, so the lockfile is unchanged and --frozen-lockfile still passes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`skipAudits: ["full-page-screenshot"]` was a no-op: that's the gatherer id for the FullPageScreenshot artifact, not an audit id, and Lighthouse drops unknown skipAudits entries silently. The artifact is also in filterResistantArtifactIds, so it runs even when no surviving audit requires it -- `disableFullPageScreenshot` is the only lever that turns it off. That's the expensive capture; screenshot-thumbnails and final-screenshot are computed from trace data and cost almost nothing. So the screenshot skip was contributing ~0 to the measured speedup, which came entirely from sharing one Chrome instance across the six audits. Verified against lighthouse 12.8.2 by resolving both configs through initializeConfig: the FullPageScreenshot artifact is present before and absent after, in both mobile and desktop, with the performance category and the desktop form factor intact. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Re-applied to re-measure the audit runtime now that the full-page screenshot is actually being skipped. Revert this commit before marking the PR ready. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

This reverts commit 51fdfb2.

…pr-lighthouse # Conflicts: # package.json

The audit goes back to running inline at the tail of ci-pull-request.sh, exactly where it was. .github/workflows/pull-request.yml and ci-pull-request.sh are now byte-identical to master. Measuring the split honestly made it hard to justify. The path filter trips on only 13 of the last 120 merged PRs (~11%), and the speedups in run-audits.mjs already take the audit from 135s to 82s -- so moving it to its own job was worth a further 82s on roughly 1 in 9 PRs. Carrying that were an artifact hand-off, a duplicate ESC fetch, and an isolated npm install that had to scrape the Lighthouse version out of yarn.lock and symlink node_modules back into the checkout. That install was the most fragile thing in the diff, and its only failure mode was quiet: a green advisory job with a table full of Error rows. What remains is the part that carried most of the win and none of the risk. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

jeffmerrick · 2026-08-06T14:02:09Z

@claude #update-review

sicarul

Nice!

github-actions · 2026-08-06T14:13:34Z

🤖 Review updated on @jeffmerrick's request.

CamSoper

Looks great, looking forward to this

jeffmerrick and others added 4 commits August 5, 2026 13:35

jeffmerrick temporarily deployed to testing August 5, 2026 18:56 — with GitHub Actions Inactive

jeffmerrick temporarily deployed to testing August 5, 2026 19:13 — with GitHub Actions Inactive

Revert "TEMP: touch theme/ to exercise the lighthouse job"

d96e124

This reverts commit c3cd6b0.

jeffmerrick marked this pull request as ready for review August 5, 2026 20:10

jeffmerrick temporarily deployed to testing August 5, 2026 20:10 — with GitHub Actions Inactive

github-actions Bot added review:no-blockers Claude review completed cleanly; outstanding is empty and removed review:in-progress Claude review is currently running labels Aug 5, 2026

jeffmerrick temporarily deployed to testing August 5, 2026 20:24 — with GitHub Actions Inactive

github-actions Bot added review:stale New commits since last Claude review; refresh on next ready-transition or @claude mention and removed review:no-blockers Claude review completed cleanly; outstanding is empty labels Aug 5, 2026

jeffmerrick temporarily deployed to testing August 5, 2026 20:40 — with GitHub Actions Inactive

jeffmerrick force-pushed the jeffmerrick/faster-pr-lighthouse branch from 3af28aa to 5373fd8 Compare August 5, 2026 20:44

jeffmerrick had a problem deploying to testing August 5, 2026 20:45 — with GitHub Actions Error

jeffmerrick and others added 2 commits August 5, 2026 15:59

TEMP: touch theme/ to exercise the lighthouse job

51fdfb2

Re-applied to re-measure the audit runtime now that the full-page screenshot is actually being skipped. Revert this commit before marking the PR ready. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

jeffmerrick temporarily deployed to testing August 5, 2026 21:00 — with GitHub Actions Inactive

jeffmerrick temporarily deployed to testing August 5, 2026 21:17 — with GitHub Actions Inactive

Revert "TEMP: touch theme/ to exercise the lighthouse job"

9f6d274

This reverts commit 51fdfb2.

jeffmerrick temporarily deployed to testing August 5, 2026 21:35 — with GitHub Actions Inactive

jeffmerrick temporarily deployed to testing August 5, 2026 21:51 — with GitHub Actions Inactive

Merge remote-tracking branch 'origin/master' into jeffmerrick/faster-…

57cc89a

…pr-lighthouse # Conflicts: # package.json

jeffmerrick temporarily deployed to testing August 6, 2026 13:22 — with GitHub Actions Inactive

jeffmerrick changed the title ~~Move the per-PR Lighthouse audit off the critical path~~ Speed up the per-PR Lighthouse audit Aug 6, 2026

jeffmerrick temporarily deployed to testing August 6, 2026 13:38 — with GitHub Actions Inactive

jeffmerrick requested review from CamSoper, cnunciato and sicarul August 6, 2026 13:53

github-actions Bot added review:in-progress Claude review is currently running and removed review:stale New commits since last Claude review; refresh on next ready-transition or @claude mention labels Aug 6, 2026

sicarul approved these changes Aug 6, 2026

View reviewed changes

github-actions Bot added review:no-blockers Claude review completed cleanly; outstanding is empty and removed review:in-progress Claude review is currently running labels Aug 6, 2026

CamSoper approved these changes Aug 6, 2026

View reviewed changes

jeffmerrick merged commit 4f38504 into master Aug 6, 2026
16 checks passed

jeffmerrick deleted the jeffmerrick/faster-pr-lighthouse branch August 6, 2026 15:09

jeffmerrick temporarily deployed to testing August 6, 2026 15:09 — with GitHub Actions Inactive

Navigation Menu

Uh oh!

Speed up the per-PR Lighthouse audit - #20728

Speed up the per-PR Lighthouse audit#20728
jeffmerrick merged 11 commits into
masterfrom
jeffmerrick/faster-pr-lighthouse

jeffmerrick commented Aug 5, 2026 •

edited

Loading

Uh oh!

pulumi-bot commented Aug 5, 2026 •

edited

Loading

Uh oh!

pulumi-bot commented Aug 5, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 5, 2026 •

edited by pulumi-bot

Loading

Uh oh!

jeffmerrick commented Aug 6, 2026

Uh oh!

sicarul left a comment

Uh oh!

github-actions Bot commented Aug 6, 2026

Uh oh!

CamSoper left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

jeffmerrick commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Numbers

Verified

Considered and dropped: a separate lighthouse job

Follow-up

Uh oh!

pulumi-bot commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pulumi-bot commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Lighthouse Performance Report

Uh oh!

github-actions Bot commented Aug 5, 2026 • edited by pulumi-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-merge Review — Last updated 2026-08-06T14:10:17Z

🔍 Verification trail

🚨 Outstanding in this PR

⚠️ Low-confidence

💡 Pre-existing issues in touched files (optional)

✅ Resolved since last review

📜 Review history

Uh oh!

jeffmerrick commented Aug 6, 2026

Uh oh!

sicarul left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 6, 2026

Uh oh!

CamSoper left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jeffmerrick commented Aug 5, 2026 •

edited

Loading

Considered and dropped: a separate `lighthouse` job

pulumi-bot commented Aug 5, 2026 •

edited

Loading

pulumi-bot commented Aug 5, 2026 •

edited

Loading

github-actions Bot commented Aug 5, 2026 •

edited by pulumi-bot

Loading