feat/geolite2-runtime — runtime GeoLite2 fetch (#659) - #766
Merged
Conversation
The offline geo tier only reached a self-host two ways: baked into the image at build time, or mounted from the host. Both need something the operator of a stock `:latest` pull does not have. Add a third: set MAXMIND_LICENSE_KEY and the worker downloads GeoLite2-City + GeoLite2-ASN itself into GEOLITE2_DIR, so the offline tier starts working with no rebuild and no manual mount. A worker-only pg-boss queue drives it. Boot discovery fetches once when a key is set and no database is present yet; a monthly cron (past MaxMind's first-Tuesday reissue) keeps a keyed host current. The fetch is fail-soft: each edition extracts to a temp file in the target directory and is promoted with an atomic rename only after both are in hand, so a bad key or an upstream error leaves the previous databases exactly as they were and never writes a partial `.mmdb`. After a successful place the reader cache resets so a running process picks up the files without a restart. Extraction is pure Node (zlib + a minimal tar reader), not a shell-out: this module is reachable from the worker-boot import chain, and a child_process import there makes Next's file tracer pull the whole project into the standalone output. zlib keeps the trace bounded and drops the busybox runtime dependency. The download goes through safeFetch with the public-host pin and redirect-following, since the MaxMind permalink 302s to a signed CDN. The runtime image now owns /opt/geolite2 as the unprivileged app user so the worker can write it; a read-only bring-your-own mount at that path still wins, because the fetch cannot write over it and fails soft. Refs #659
Cover the new MAXMIND_LICENSE_KEY route in the self-hosting guide as the first of the three ways to get the offline databases (fetch, mount, bake), and note that the monthly runtime refresh keeps a keyed host current on its own. Add the variable to both env examples and to the env-check manifest so the optional group lists it. Refs #659
safeFetch embeds the full target URL in its thrown error, and the GeoLite2 download URL carries license_key=<secret> in its query string. The wide-event warning and job-failure paths do not redact it, so a timeout or network error on the runtime fetch printed the key to stdout, Loki and GlitchTip in clear. The download timeout is 120s for slow self-host links, so that path is expected, not rare. The fetch now re-throws with the edition and the error kind only, never the URL. As defence in depth, the central query-param redactor also learns license_key, so any other path that logs such a URL through redactSecrets is covered too. A regression test asserts the returned error carries neither the key nor the parameter name.
A self-hoster on the published image can set MAXMIND_LICENSE_KEY at runtime and the app fetches the GeoLite2 databases itself, no rebuild or manual mount. Refs #659.
The runtime fetch extracts databases in pure Node, but its unit test builds a MaxMind-shaped tar.gz fixture with the system tar, which the dead-code gate flagged as an unlisted binary. It is available in CI and dev, alongside the docker and pg_isready entries already listed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lets a self-hoster running the published image set MAXMIND_LICENSE_KEY at runtime; the worker fetches the GeoLite2 City and ASN databases itself (boot when absent, monthly refresh), extracts them in pure Node, and places them atomically. Fail-soft throughout: any download error leaves the previous databases untouched and the online lookup keeps answering; a read-only mounted database still wins.
Security: the licence key never reaches a log line on a fetch error (the URL carrying it is kept out of the reported error, and the central redactor also masks license_key).
No schema change, no migration. Semver patch. Refs #659 (leave open for the reporter to confirm).