fix: externalize gRPC/pubsub packages so bundling can't break publish

[GregHilstonMozilla]

References:

Jira: MNTOR-5314

Description

Problem

Breach alert emails silently stopped in production. HIBP POSTs new breach hash ranges to POST /api/v1/hibp/notify, which publishes them to Pub/Sub for the breach alerts consumer to email affected users. The publish (pubsub.topic(...).publishMessage()) was failing ~100% of the time (≈ 19.7M/45d; HTTP 429 returned to HIBP), so nothing was queued, consumed, or emailed. Ingestion was unaffected, and breaches still appeared on the site, only alerting was dead.

Root cause

@grpc/grpc-js was being bundled by the Next.js build (it was not listed in serverExternalPackages). Next.js bundles server-side dependencies by default. Libraries that use Node.js native features must be marked external so they load via native require (Next.js docs).

Next ships a built-in allow-list of packages it auto-externalizes (server-external-packages.jsonc). That list includes packages like firebase-admin, pg, mongodb, dd-trace, etc., but not @grpc/grpc-js, google-gax, or @google-cloud/*, so those get bundled unless we add them.

Bundling grpc-js corrupts its error. StatusObject: grpc-js builds an error message as ${code} ${Status[code]}: ${details}, so when the status fields are stripped the failure renders as the useless undefined undefined: undefined, which is why the Sentry issues were titled uselessly and the cause stayed hidden. The same "bundling makes grpc's status undefined" failure is reported against @google-cloud/pubsub in nodejs-pubsub#1341 (a browser-bundling context, but the same failure mode).

Verified with a local A/B against real (stage) Pub/Sub using the same topic, only bundling differs. bundled → 429 undefined undefined: undefined; un bundled → 200 success (a plain gcloud publish to the same topic also succeeds).

Timeline (all UTC):

• 2026-05-13 — dependabot bumps protobufjs 7.5.5→7.5.8 (08715f8da) into main, the
  eventual trigger, dormant until it deploys.
• 2026-05-24 05:15 — last successfully alerted breach (7-Eleven), still running the prod-2026.05.06 image. Publishing is healthy.
• 2026-05-26 18:21 — prod-2026.05.26 deploys, carrying the protobufjs bump; the bundled grpc-js stack now corrupts the publish. Outage begins.
• 2026-05-26 22:03 — first missed breach (Ameriprise), ~3.7h after the deploy. Every breach from here on goes un-alerted.
• 2026-05-26 → 07-13 — ~7 weeks silent, 20+ notifiable breaches un-alerted. Undetected: the error was opaque (undefined undefined: undefined), there was no publish-failure alert, and the local/CI Pub/Sub emulator doesn't reproduce it.
• 2026-07-13 — outage discovered while verifying the Glendale Community College breach (ingested but never alerted).
• 2026-07-14 — root cause identified (Next bundling of grpc-js), trigger pinned to the protobufjs bump via a local dependency bisect (Next.js and @grpc/grpc-js versions ruled out), and the fix (this PR) validated against real (stage) Pub/Sub.

Why it wasn't caught

Local dev and CI use the Pub/Sub emulator (insecure transport); the failure only manifests against real Pub/Sub (TLS), so tests passed. The error was also opaque (undefined undefined: undefined), and there was no alert on the publish failure ratio (the dead letter queue can't catch a pre queue failure).

Fix

Add @grpc/grpc-js, google-gax, and @google-cloud/pubsub to serverExternalPackages in next.config.js so they load unbundled from node_modules, where the gRPC status, and the publish, work correctly. Adds a regression test asserting they stay external.

Recovery / backfill (follow up, not in this PR)

This PR stops the bleeding; it does not resend the ~7 weeks of missed alerts. HIBP does not automatically renotify, so we must retrigger. The missed breaches are enumerable (breaches added since 2026-05-26 with no email_notifications incident row). Two options:

1. ask HIBP to resend the breach callbacks for that window (cleanest, as it redrives the now fixed pipeline with the correct hash ranges. We did not persist the original ranges, so we can't replay them ourselves). Simplest approach, but requires communication and coordination with an external party.
2. A one off back fill script that, per missed breach, determines affected subscribers via the HIBP k-anon range API and sends the incident email, writing email_notifications so reruns don't double send. Something we can do all by ourselves.

How to test

1. Success path: docker compose --env-file .env.local up -d (postgres + pubsub emulator), npm run dev, then the README notify curl to http://localhost:6060/api/v1/hibp/notify → HTTP 200 + queued_breach_notification_success.
2. Reproduce the bug: temporarily remove the three packages from serverExternalPackages, restart, repeat → HTTP 429 with error_queuing_hibp_breach: undefined undefined: undefined.
3. Regression test: npx vitest run src/app/api/v1/hibp/notify/serverExternalPackages.test.ts → passes; fails if any of the three packages is dropped from serverExternalPackages.

Checklist (Definition of Done)

• Commits are minimal and descriptive (one config change + a regression test).
• I've added or updated the relevant code comments (why these packages must stay external).
• I've added a unit test to test for potential regressions of this bug
  (serverExternalPackages.test.ts). Note: a runtime bundler-config regression can't be
  exercised in a unit test; this guards against silently dropping the packages from the list.
  A real-transport publish smoke test is tracked as a follow-up.

[GregHilstonMozilla]

This unit test honestly seems pretty low value. I'll happily remove it if anyone else feels that way...

[joeherm]

Good find and really great explanation. Is it possible to test this with an E2E test against the stage environment in addition to / instead of the unit test?

[GregHilstonMozilla]

@joeherm I'm going to hold off on authoring an e2e test for right now, as I think it will push back resolving this issue another work day. I've added it as one of the top action items in the incident document. TY for reviewing this and brainstorming :)

[joeherm]

Fix LGTM- let's keep the regression guard in place and file a ticket to E2E test this pathway.