Build from committed content snapshot instead of live APIs - #6
Merged
Conversation
Notion's private api/v3 and Substack's public API both sit behind
Cloudflare, which serves a 403 challenge to GitHub Actions runners. CI
could not fetch either, so the site shipped without essays.
Fetch the raw upstream data from a dev machine instead, commit it under
data/, and parse it at build time:
npm run fetch-content [-- --only=notion|substack]
Only fetching moves offline — all parsing still happens during the build,
so a stale snapshot means stale content, never a stale layout. Builds
select a source via CONTENT_SOURCE: auto (snapshot if present, else live),
snapshot (snapshot only, hard-fail if missing), or live. Both workflows
set snapshot so a missing one fails loudly instead of deploying an empty
site.
Also fixes two bugs found along the way:
- Notion returned 403 for *any* client without a browser User-Agent,
including local dev. lib/notion-api.ts now sends browser headers. It
also passed NOTION_API_TOKEN (an ntn_ integration token) as authToken,
which notion-client sends as the token_v2 browser cookie — the wrong
credential entirely. Read NOTION_TOKEN_V2 instead; the root page is
public so no auth is needed.
- notion-client 7.10 returns __version__ 3 recordMaps, nesting entries as
{value:{value,role}} rather than {role,value}. react-notion-x unwraps
either via getBlockValue, but nine call sites in this repo read .value
directly and silently got undefined — post pages rendered an empty <h1>
and a date of 12/31/1969. Use getBlockValue throughout and drop the
bespoke normalizeMap workaround.
package-lock.json carries a large diff because it still pinned notion-*
at ^6.16.0 while package.json had moved to ^7.10.0 — the patches are
named +7.10.0 and could not have applied to a fresh CI install.
Verified: clean build + export renders 7 Substack and 6 Notion entries
inline in one date-sorted list, and all 6 post pages render full bodies
with correct dates.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
✅ Deploy Preview for soft-swan-5d0ce7 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Deploy Preview for darling-dusk-441006 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
The Netlify build failed at `npm install` — patch-package could not apply either patch, so postinstall exited 1. Both patches had been generated against locally-linked, locally-built copies of the packages (the `nns-deps:link` workflow), so they carried .turbo logs, build/*.d.ts and .tsbuildinfo that the published tarballs do not ship, and their build/index.js hunk was diffed against a CJS bundle while 7.10.0 publishes ESM. They could never apply to a registry install. The only load-bearing change in ~200KB of patch noise was one feature: rendering the `caption` property on bookmark blocks, which upstream 7.10.0 does not do (it renders title and description only). Without it, 32 authored captions silently vanished from nigeria-agriculture and nigeria-agriculture-research. There is no `Bookmark` entry in NotionComponents, so no supported override hook exists — hence a patch. - notion-types patch: deleted. That package's build/index.js is 33 bytes of sourcemap comment and it is imported for types only, so the patch could not affect runtime. - react-notion-x patch: regenerated against the pristine tarball. Now 23 lines touching one file, versus 200KB+ of build artifacts. Verified with a from-scratch `npm install` in a clean directory: exits 0 and the patch applies. Rendering is unchanged except that captions on link-only bookmarks now show too — the previous fork dropped those, and nothing is lost relative to it. Also adds .nvmrc: Netlify was building on Node 18.20.8 against an engines field requiring >=22.9.0 (npm warned EBADENGINE), while GitHub Actions uses 22.9.0. Pin both to the same version. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
…resh next build and next export each flagged three config problems on every run: - assetPrefix was "" when no base path is set; next requires >=1 char. Pass undefined instead. - images.path was ""; same validator. Set "/". Verified all 74 rendered image URLs are byte-identical before and after. - swcMinify sat under `experimental`, where next 12 ignores it and warns. Moved to top level, which means it now actually applies — rendered text is unchanged on all 10 pages; only JS chunk hashes differ. Also documents the deploy setup that was previously tribal knowledge: - Netlify's four load-bearing UI settings, each with the failure it prevents. None live in this repo, so a fresh Netlify site wired to it hits all four in sequence. - Publishing is a two-step process — writing in Notion or Substack does nothing until `npm run fetch-content` is re-run and data/ committed. - Notion signs file-attachment URLs with a short expiry, so links to uploaded files go stale hours after a refresh. Images are unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Notion's private api/v3 and Substack's public API both sit behind Cloudflare, which serves a 403 challenge to GitHub Actions runners. CI could not fetch either, so the site shipped without essays.
This fetches the raw upstream data from a dev machine instead, commits it under
data/, and parses it at build time:Only fetching moves offline — all parsing still happens during the build, so a stale snapshot means stale content, never a stale layout. Builds select a source via
CONTENT_SOURCE:auto(default)data/when present, otherwise fetch live — good for local devsnapshotdata/only, fail the build if missing or incompletelivedata/and always fetchBoth workflows set
CONTENT_SOURCE=snapshotso a missing snapshot fails loudly instead of silently deploying an empty site.Two bugs fixed along the way
Notion returned 403 to any client without a browser User-Agent, including local dev — this was the failure behind the broken build, not just a CI problem.
lib/notion-api.tsnow sends browser headers. It also passedNOTION_API_TOKEN(anntn_integration token) asauthToken, which notion-client sends as thetoken_v2browser cookie — the wrong credential type entirely. Now readsNOTION_TOKEN_V2; the root page is public, so no auth is needed at all.notion-client 7.10 returns
__version__3 recordMaps, nesting entries as{value:{value,role}}instead of{role,value}. react-notion-x unwraps either viagetBlockValue, but nine call sites in this repo read.valuedirectly and silently gotundefined— post pages rendered an empty<h1>and a date of 12/31/1969. Fixed at all nine sites; the bespokenormalizeMapworkaround is gone.Note on package-lock.json
The large diff is a fix, not churn. The lock still pinned notion-* at
^6.16.0while package.json had moved to^7.10.0— and since the patches are named+7.10.0, they could not have applied to a fresh CI install. Installingtsxforced npm to reconcile it.Verified
next build && next exportrenders 7 Substack + 6 Notion entries inline in one date-sorted listCONTENT_SOURCEmodes exercised, including the hard-fail pathtsc --noEmitclean on every file touched;patch-packageapplies cleanlyKnown limitation
Notion signs file-attachment URLs with a ~hours expiry, baked in at fetch time. Exactly one rendered link is affected — a PDF inside a collapsed toggle on
nigeria-agriculture-research. This is inherent to static export rather than new here (a live build has the same problem, just refreshed more often). Images are unaffected: they route through thenotion.so/image/proxy with unsigned source URLs. Fix, if wanted: download attachments intopublic/at fetch time.🤖 Generated with Claude Code