build: fetch the bundled drivers, and prove a build needs nothing else - #708
Conversation
Groundwork for FTW holding no driver source at all. Nothing is deleted here and the committed files still ship; what changes is that fetching them becomes a first-class build step and CI proves it sufficient. make drivers fetch drivers/ from the pinned commit make drivers-present fail with the remedy when the directory is empty `test` depends on the presence check rather than on the fetch, so the inner loop keeps costing no network. The new drivers-from-pin job is the point. It deletes the committed drivers, confirms the guard fires, fetches from the pin, checks all 37 came back, runs the driver suites against the fetched copies, and asserts they are byte-identical to what was committed. While the files are still in git that is redundant with the drift check. The moment they leave git it is the only thing between a fresh clone and a build that cannot find a driver. The fetch writes to the same path the committed files occupy, so nothing that reads drivers/ needs to know where they came from. The fifteen Go test files that build that path by hand are untouched, and so are the Dockerfile and the release tarballs. Verified locally: rm drivers/*.lua, guard fires, make drivers restores 37 of 37, go test ./internal/drivers/... ./internal/driverinventory/... passes, git diff on drivers/ is empty. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Fredrik Ahlgren <fredrik@sourceful-labs.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 26adba3ce0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # ---- Testing ---- | ||
|
|
||
| test: optimizer/.venv/.installed | ||
| test: optimizer/.venv/.installed drivers-present |
There was a problem hiding this comment.
When the committed .lua files are removed—or a developer has discarded them as this workflow does—only test checks drivers-present; release still archives the metadata-only drivers/ directory successfully and produces artifacts with no recovery drivers. Make release depend on fetching or validating the complete snapshot before the planned deletion can land.
AGENTS.md reference: AGENTS.md:L12-L15
Useful? React with 👍 / 👎.
| drivers: | ||
| bash scripts/sync-bundled-drivers.sh |
There was a problem hiding this comment.
When a pin update removes a driver, make drivers invokes the existing sync script in write mode, which reports the old file as UNLISTED but exits successfully without deleting it; for example, dropping zap leaves drivers/zap.lua on disk while printing that 36 drivers were written. The new target therefore does not reliably recreate the pinned snapshot in an existing checkout, so it should remove unlisted files or fail instead.
AGENTS.md reference: AGENTS.md:L12-L15
Useful? React with 👍 / 👎.
drivers/*.lua leaves git and is gitignored. The files still ship -- offline recovery needs them in the image, the release tarballs carry them, the tests read them -- but they are fetched from the commit pinned in drivers/BUNDLED_SOURCE.json rather than committed here. The point is not the bytes. It is that there is no file to open a pull request against. Two contributors spent real effort, including thirty minutes on an SE17K and register-by-register probing on a Pixii, writing fixes against files that regenerate. A guard that fails the pull request recovers the codebase but not the afternoon. Absence teaches faster than a red check does. Everything that needed the files in git already stopped needing them: #708 made the fetch a build step and proved a build works from the pin alone, #709 moved the SemVer gate onto the pins. This is the small change those two were for. --check changes its question. It used to compare committed bytes against the pin; there are no committed bytes now, so it asks whether any driver is committed at all. Cheaper than a content diff, a stricter rule, and it needs no network. Every workflow that reads drivers/ fetches first: core, drivers and e2e in test.yml, the image build in beta.yml, and both the tarball and image jobs in release-assets.yml. e2e, release and driver-repository-validate depend on drivers-present locally, so a missing fetch fails with the remedy rather than a confusing file-not-found. Verified: rm -rf the drivers, make drivers restores 37 of 37, go test ./... passes uncached, and the full-stack e2e is green. The guard is proven to fire -- git add -f on one driver and --check reports it. Signed-off-by: Fredrik Ahlgren <fredrik@sourceful-labs.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
1 participant
First step toward FTW holding no driver source at all. Nothing is deleted here. The committed files still ship, the drift check still guards them, and this is reversible with one
git revert.What changes
testdepends on the presence check, not the fetch, so the inner development loop keeps costing no network.The part that matters
The new
drivers build from the pin alonejob deletes the committed drivers and rebuilds without them:rm -f drivers/*.luamake drivers-presentfails — the guard has to actually firemake driversgo test ./internal/drivers/... ./internal/driverinventory/...against the fetched copiesWhile the files are still in git this is redundant with the drift check. The moment they leave git it is the only thing standing between a fresh clone and a build that cannot find a driver. Proving it now means the deletion is a small, boring change rather than a leap.
One thing turned out cheaper than expected
The earlier sketch assumed the fifteen Go test files that build
drivers/<id>.luapaths by hand would each need editing. They do not.sync-bundled-drivers.shwrites to${ROOT}/drivers, which is exactly where those tests already look, so a fetched directory is indistinguishable from a committed one. The Dockerfile and the release tarballs are untouched for the same reason.That removes most of the estimated work from the remaining steps.
Verified locally
What is left
drivers/*.lua, gitignore them, keep the pin.luamay be committed here"Independent of #704 (moves the pin) and #705 (watches for a stale pin) — no overlapping files, any merge order works.
🤖 Generated with Claude Code