Skip to content

perf(collect): stop buffering every response body just to count bytes - #174

Open
JonasJesus42 wants to merge 1 commit into
mainfrom
fix/collect-avoid-body-buffering
Open

perf(collect): stop buffering every response body just to count bytes#174
JonasJesus42 wants to merge 1 commit into
mainfrom
fix/collect-avoid-body-buffering

Conversation

@JonasJesus42

@JonasJesus42 JonasJesus42 commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Problem

responseToEntry() in src/engine/collect.ts called safeBodySize()resp.body() for every response of every page, just to record byteLength for the network waterfall — even though content-length was already parsed as the fallback.

resp.body() forces Chromium to materialize the entire response body (images, fonts, video, …) and copy it into the Node heap. Done for every request, on every page, across up to 4 concurrent BrowserContexts on image-heavy commerce pages, this was one of the largest drivers of resident memory — contributing to the OOM pressure that SIGKILLs sibling processes (exit 137).

Fix

  • Prefer the content-length header (which is what a byte waterfall wants anyway — transfer size).
  • Only fall back to resp.body() for small text resources (document/script/xhr/fetch) and only when the header is absent.
  • Never buffer binary assets (image/media/font/stylesheet) just to count bytes.

Also a speed win: far less body I/O per page.

Verification

  • bun run check (tsc) passes.
  • No tests depend on the previous body-buffering behavior.

Part of the memory-pressure / OOM series (see #173).

🤖 Generated with Claude Code


Summary by cubic

Stop buffering response bodies in collect to compute transfer size. We now use content-length when present and only read bodies for small text responses, cutting memory use and speeding up runs.

  • Refactors
    • Prefer content-length in responseToEntry() for waterfall transfer size.
    • Only read the body when the header is missing and the resource is small text (document/script/xhr/fetch); skip binary assets.
    • Introduced isSmallTextResource() to gate the fallback.

Written for commit a27cd91. Summary will update on new commits.

Review in cubic

responseToEntry() called safeBodySize() → resp.body() for EVERY response of
every page, purely to measure byteLength — even though content-length was
already parsed as the fallback. resp.body() forces Chromium to materialize
the entire body (images, fonts, video) and copy it into the Node heap. Across
up to 4 concurrent contexts on image-heavy commerce pages this was a major RSS
driver contributing to OOM (exit 137).

Prefer the content-length header; only fall back to resp.body() for small text
resources (document/script/xhr/fetch) when the header is absent. Never buffer
binary assets to count bytes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant