fetchladder

Start with plain HTTP. Climb to a real browser only when you can prove the cheap answer was a lie.

Zero runtime dependencies (a clean install adds 1 package). Nothing is downloaded at install time.

The problem

A scraper watched a company careers page and returned zero postings for weeks. Nothing alerted, nothing errored, nothing looked wrong. The site had migrated to a different platform and the old selector matched nothing, but the request still returned HTTP 200 with well-formed HTML, so every layer of the stack reported success.

Zero postings is a perfectly valid answer. That is exactly why nobody caught it.

This is the failure mode fetchladder exists for. A cheap fetch can return 200 and still be a lie:

• an empty JavaScript shell that renders on the client
• a login wall where the content used to be
• a bot challenge page that looks like a complete document
• a soft 404 that returns 200 and says "nothing found"
• a rate limit page
• a page that renders perfectly and contains none of what you came for

HTTP clients return 200 and move on. Headless browsers render an empty shell without complaint. Markdown extractors will happily convert a login wall into tidy markdown. fetchladder's contract is the opposite: tell you when the cheap path lied, then climb until it stops lying.

How it works

Two tiers, because measurement supports two tiers and not four.

1. Tier 0, plain HTTP. Node's global fetch with an ordinary browser user agent. Most pages need nothing more.
2. Tier 1, Chromium over CDP. A real browser, launched only when a detector fires.

Between them sit the detectors. Escalation happens on a verifiable signal, never on a guess, never on a heuristic score, never on a model's opinion. There is exactly one escalation: HTTP, then browser, then a typed error. No retry loops against someone else's server.

Results

Measured live on 2026-07-31:

Target	Tier 0 result	What fetchladder did	Time
jobs.apple.com/en-us/search?sort=newest	HTTP 200, 310,994 bytes, 21,111 chars of text, 159 links	Answered from tier 0. No browser launched.	2.1s
metacareers.com/jobs	HTTP 400, 1,543 bytes, 110 chars of text	Detected blocked, climbed to Chromium, returned 604,130 bytes and 139 links	5.4s
metacareers.com/jobs (second visit)	skipped	Learned cache sent it straight to Chromium	2.5s

The learned cache is the actual payoff: once a site is known to need a browser, fetchladder stops paying for the failed HTTP probe first, which cuts the Meta round trip from 5.4s to 2.5s on every repeat visit.

It is not just one site

A separate benchmark measured curl, Lightpanda and Scrapling head to head across 5 live career-site targets (3 runs per tool, median wall-clock). It found the same shape fetchladder is built around: plain HTTP alone retrieved the complete posting list on 3 of 5 targets (Apple, the Linux Foundation, Synopsys). Only one target, Meta, was a genuinely JS-rendered page that needed a real browser. A tool that reached for a browser by default would have paid the Chromium cost on 3 sites that never needed it.

Footprint

• 70 tests, CI green on Ubuntu and macOS, across Node 22 and 24.
• The whole src/ tree is 1,952 lines of TypeScript. The CDP client that drives Chromium is 629 of them.

fetchladder vs plain fetch vs Playwright vs crawl4ai

Naming what the others are better at, not just what this one does:

	Plain fetch	Playwright	crawl4ai	fetchladder
Detects a lying HTTP 200 (empty shell, auth wall, soft 404)	No	No	Not a dedicated feature	Yes, typed signals (js_shell, auth_wall, expect_unmet, ...)
Full browser automation surface (iframes, shadow DOM, download interception, actionability retries)	No	Yes	Yes, built on Playwright	No, by design, see Limitations
LLM-ready markdown, chunking, extraction strategies	No	No	Yes, its core strength	No, out of scope, see Roadmap
Runtime dependencies	0	Playwright plus a downloaded browser	Playwright plus several Python packages	0
Escalates HTTP to a browser automatically, on a verified signal	No	N/A, already a browser	Not signal-driven	Yes, exactly one escalation

Install

Not on npm yet. Install from GitHub, which builds on install:

npm install github:Jeneidi/fetchladder

Node 22 or newer. There is no post-install step and no bundled browser. Tier 0 works immediately. Tier 1 uses a Chromium-family browser already on the machine (Chrome, Chromium, Edge or Brave), discovered at runtime.

Check what your machine can do, with or without installing first:

npx github:Jeneidi/fetchladder doctor   # no install
npx fetchladder doctor                  # once it is a dependency

fetchladder 0.1.0 (node v24.13.0, darwin arm64)

  [ok]      http         built in, no setup required
  [ok]      chromium     Google Chrome 150.0.7871.187 at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

  Cache: 0 sites learned (~/Library/Application Support/fetchladder/sites.json)

  Ready. Tier 0 (http) and tier 1 (chromium) available.

If no browser is found, doctor names exactly what is lost and how to fix it, rather than reporting a vague failure.

Quickstart

import { fetch as ladderFetch } from "fetchladder";

const page = await ladderFetch("https://example.com/products?sort=newest");

page.engine;     // "http" | "chromium"   which tier actually answered
page.escalated;  // false                 whether it had to climb
page.signals;    // []                    and why, if it did
page.text;       // readable text
page.links;      // every href on the page

The part that matters: tell it what a good answer looks like

Without an expectation, fetchladder can only catch structural lies. With one, it also catches the semantic lie that opened this README.

const page = await ladderFetch("https://example.com/careers", {
  expect: { minLinks: 5, matching: /\/job\/\d+/ },
});

If HTTP returns zero matching links, that is a verified failure signal, not an answer. fetchladder escalates to Chromium and checks again. If the browser also returns nothing, it throws NoResultError instead of handing back an empty array that your code will treat as "no jobs today".

import { NoResultError } from "fetchladder";

try {
  const page = await ladderFetch(url, { expect: { minLinks: 5, matching: /\/job\/\d+/ } });
  process(page.links);
} catch (err) {
  if (err instanceof NoResultError) {
    // Loud, typed, and it names the evidence:
    //   "neither tier produced an answer. HTTP: js_shell (...). Browser: expect_unmet (...)"
    alertMe(err.message, err.signals);
  }
}

From the command line

npx fetchladder fetch "https://jobs.apple.com/en-us/search?sort=newest" \
  --expect-links 10 --expect-matching "/en-us/details/"

200 https://jobs.apple.com/en-us/search?sort=newest
engine:    http
bytes:     310994
text:      21111 chars
links:     159
took:      2121ms

The signals

Every detector is a pure function over the response. They are unit tested against saved fixtures of each failure mode, which is the cheapest way to falsify this project's premise: if the detectors cannot separate real answers from lies, nothing else here matters.

Signal	Fires when
blocked	HTTP 400, 401, 403, 406 or 429
challenge	Cloudflare, Turnstile, hCaptcha, reCAPTCHA, DataDome, PerimeterX or Incapsula interstitial
js_shell	Under 200 characters of text plus an empty app root or a module script
auth_wall	Redirect chain ends at a sign-in path, or a small page whose main feature is a password field
expect_unmet	Your expect block is not satisfied. This is the silent-zero detector
shrinkage	This site normally returns far more bytes than it just did
transport	DNS, TLS, connection reset or timeout

signals names are part of the public API. Branch on them.

What does not trigger escalation, deliberately: a 404 (that is a real answer, and no browser will turn it into rows), a 200 with plenty of content and no expect block (nothing says it is wrong), and slow responses (slow is not wrong). Escalating on ambiguity would give back the cost the cheap tier exists to save.

Precision is tested as carefully as detection. A real article with a sign-in modal is not an auth wall. A real page whose contact form loads reCAPTCHA is not a challenge. A short JSON API response is not an empty shell.

API

fetch(url, options?): Promise<PageResult>

Aliased as ladderFetch for callers who do not want to shadow the global.

Option	Default	What it does
expect	none	{ minLinks, matching, contains, minText }. What a good answer looks like
browser	"auto"	"never" refuses to climb, "always" starts at the browser
ignoreRobots	false	Fetch a path robots.txt disallows. Deliberate, and logged
allowDomains	none	Hostnames this call may touch
minIntervalMs	1000	Minimum gap between requests to the same host
cache	true	Set false to neither read nor write the learned cache
timeoutMs	20000	Per request
settleMs	5000	How long the browser keeps re-reading a page that is still a shell
headless	true	Set false to watch it work
userAgent, headers, chromePath		Overrides

PageResult: { url, status, html, text, links, engine, escalated, signals, bytes, ms, fromCache }.

Errors: NoResultError, BrowserUnavailableError, RobotsDisallowedError, DomainNotAllowedError, all extending FetchladderError and carrying .signals plus the failing .result.

open(url, options?): Promise<Session>

The explicit escape hatch for anything stateful or interactive. No routing happens here, because "click this button" has exactly one tier that can do it.

import { open } from "fetchladder";

const s = await open("https://example.com/login", { session: "my-app" });
await s.fill("#email", "user@example.com");
await s.fill("#password", process.env.PW!);   // your string, see the scope note below
await s.click("button[type=submit]");
await s.waitFor({ url: /\/dashboard/ });
await s.screenshot("/tmp/out.png");
await s.close();                               // cookies persist under the session name

Session methods: goto, html, text, url, evaluate, click, fill, waitFor, screenshot, probe, close. Selectors are CSS.

Named sessions keep a browser profile under the state directory, so cookies survive between runs.

doctor(): Promise<DoctorReport>

What is installed, what is missing, what the missing pieces would unlock, and how many sites have been learned.

The learned-method cache

It ships empty. No bundled table, no seed data, no import path from any other tool. Every install learns its own sites from its own traffic.

Once fetchladder knows a site needs a browser, it stops paying for the failed HTTP probe first. On the Meta measurement above that is 2.9 seconds saved on every repeat visit, which for a monitoring agent is the entire workload.

• One entry per host, stored outside any repository (~/Library/Application Support/fetchladder/sites.json on macOS, $XDG_STATE_HOME/fetchladder/ on Linux, %LOCALAPPDATA% on Windows).
• Two consecutive failures delete the entry outright rather than decaying a score, because deleting self-heals after a site migration.
• Entries older than 30 days are ignored, so a stale lesson never becomes permanent. A cache that never re-probes is how the bug in the opening story becomes forever.

Set FETCHLADDER_NO_CACHE=1 to disable it entirely, or FETCHLADDER_CACHE to relocate it.

Scope boundary: this is not an anti-detection tool

Stated as hard design rules, not aspirations.

1. robots.txt is respected by default. Wildcards and $ anchors are honoured, longest match wins. Override per call with ignoreRobots: true, which is a deliberate act and is logged.
2. Per-domain rate limiting is on by default, one request per second, with Retry-After honoured.
3. An optional domain allowlist lets an operator constrain a scoped agent.
4. No credential handling anywhere in the library. No keychain integration, no credential store, no login helper, no cookie import from another tool. fill() receives a string you already have.
5. No fingerprint spoofing, no user agent rotation, no proxy rotation, no CAPTCHA solving, no challenge evasion. fetchladder detects a challenge and reports it as a signal. Detecting a wall is a compatibility feature. Climbing it is a different product, and not this one.

Limitations

This section is deliberately specific. Read it before adopting.

• The CDP driver is not a Playwright replacement and does not try to be. It navigates, waits, reads, clicks, fills, screenshots and persists cookies. It does not do out-of-process iframe traversal, shadow DOM piercing, download interception, or actionability auto-retry (waiting for an element to be stable, visible and unobscured). Clicks are dispatched through the element rather than as trusted input events, so a site that specifically checks for trusted events will not be fooled, by design.
• expect is what makes the semantic detector work. Without it, a soft 404 is structurally indistinguishable from a real page that genuinely has nothing on it today. The test suite asserts this limitation rather than hiding it.
• Windows tier 1 is unverified. The Chrome discovery paths are written but have not been run on Windows. Tier 0 is fine there. Tier 1 is exercised for real in CI on Linux and macOS, across Node 22 and 24, launching the runner's own Chrome.
• The detectors are regex readers, not a DOM parser. They answer "is this page a lie", which is a coarse question. They are not a general HTML parsing layer.
• Rate limiting is per process. Two processes will not coordinate.
• Site keys are hostnames, not registrable domains, so a.example.com and b.example.com learn separately. That is the safe direction to be wrong in, and it avoids a public suffix list dependency.
• Pre-1.0, so minor versions may change the API.

Roadmap

Deferred from v0.1 on purpose, in rough priority order:

• extract({ select, fields }): selector to rows. Needs a real DOM parser, which is the first dependency worth taking.
• snapshot(): accessibility tree with stable @ref handles, so an agent can address elements without writing CSS selectors.
• upload() via DOM.setFileInputFiles.
• An optional Playwright adapter for tier 1, used only if the host project already resolves playwright, covering the driver gaps listed above.
• Opt-in download of Chrome for Testing when no browser is found.
• A Claude Code plugin manifest and skill file.

Development

npm install
npm run build     # tsc to dist/
npm test          # 70 tests: detectors, router, cache, robots, doctor, live browser
npm run smoke     # live run against two real sites. Not part of CI

The browser integration tests skip themselves by name when no Chromium-family browser is present. They are never silently passed.

Why the name

A ladder you climb only as far as you have to. It is infrastructure, not circumvention.

License

MIT. See LICENSE.

fetchladder vendors no third party source and bundles nothing. The CDP client is original work speaking a public protocol.