Web scraping with Node, Axios and Cheerio

Scrapes the Hacker News front page, keeps stories newer than a cutoff you set, flags the ones matching your keywords, and writes the result to a CSV. Follows pagination until it runs past the cutoff.

Status: working, verified against live Hacker News · Node 18+

Companion code for Web scraping with Node and Cheerio, which walks through the same code line by line.

Quick start

git clone https://github.com/yum72/web-scraping-tutorial.git
cd web-scraping-tutorial
npm install
cp .env.example .env
npm start

Configuration

All of it lives in .env:

Variable	What it does
URL	Where to start. https://news.ycombinator.com/ for the front page, https://news.ycombinator.com/jobs for the jobs board.
FILTER_VALUES	JSON array of lowercase keywords. A story is flagged when its title contains any of them.
MAX_DAYS_BACK	How far back to go. Scraping stops at the first story older than this.

Keyword matching is a plain substring test, so short keywords catch more than you might expect: "ai" matches "Tailscale" as well as "AI". Use longer keywords when that matters.

Output

A timestamped CSV in the project root, one row per story:

Link,Title,Posted,Matched
https://tailscale.com/blog/hugging-face-intrusion,Tailscale didn't stop the Hugging Face intrusion,2026-07-31T19:03:45,true
https://john.fun/elevators,Elevators,2026-07-31T15:17:28,false

Posted is the exact ISO timestamp Hacker News puts in the title attribute of its .age element, not the "3 hours ago" text. Parsing the attribute instead of the display text avoids a whole category of date bugs.

How it works

Three files, one job each:

File	Job
request.js	Fetches a URL with Axios and a User-Agent header, returns the HTML.
processHTML.js	Loads the HTML into Cheerio, pulls out each story, applies the keyword and date filters, and finds the next page.
handler.js	Reads config, writes the CSV header, and recurses through pages until there is no next one.

A note on selectors

Scrapers break when the site changes, and this one did. It originally selected titles with .titlelink; Hacker News later renamed that class to .titleline > a, and every run after that returned zero rows while failing silently.

That is the normal failure mode for scrapers: not a crash, just empty output. If you run this and get no rows, open the page in a browser, inspect an actual story row, and compare the classes against processHTML.js. In production the fix is to monitor how many rows each run returns and alert when the count drops to zero, rather than waiting to notice.

License

MIT