Skip to content

Make model snapshot builds deterministic - #960

Open
fettpl wants to merge 3 commits into
PrimeIntellect-ai:mainfrom
fettpl:agent/930-deterministic-model-snapshot
Open

Make model snapshot builds deterministic#960
fettpl wants to merge 3 commits into
PrimeIntellect-ai:mainfrom
fettpl:agent/930-deterministic-model-snapshot

Conversation

@fettpl

@fettpl fettpl commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

What

  • Compile the committed packages/ai/src/models.generated.ts snapshot during normal, CI, package, and release builds without fetching catalogs.
  • Add explicit root/package refresh-models commands.
  • Make refreshes reject failed, malformed, and zero-model required sources; render deterministic source provenance; and replace the snapshot atomically only after all sources succeed.
  • Refresh the committed snapshot through the updater and add focused regression coverage.

Why

Build artifacts should be determined by the reviewed commit. A transient catalog outage or upstream change must not silently alter or truncate a release build.

Impact

Builds are network-independent and compile the reviewed catalog snapshot. Maintainers opt into catalog changes with npm run refresh-models, then review both model changes and the canonical source digests in the generated diff. A failed refresh leaves the existing snapshot intact and exits nonzero.

Root cause

The packages/ai build script ran live generation before every compile. Catalog helpers converted network and parsing failures to empty arrays, while the top-level handler logged errors without setting a failing exit status, allowing a partial snapshot to overwrite the committed file.

Checks

  • cd packages/ai && npx tsx ../../node_modules/vitest/dist/cli.js --run test/model-snapshot-generation.test.ts (6 passed)
  • npm run check
  • cd packages/ai && npm run refresh-models (four required live catalogs loaded; snapshot and provenance generated)

Fixes #930

Note

Make model snapshot builds deterministic with provenance tracking and strict validation

  • Model refresh is decoupled from npm run build; snapshots must now be explicitly regenerated via npm run refresh-models (or the new root-level proxy).
  • generate-models.ts now canonicalizes and SHA-256-hashes each upstream catalog (models.dev, OpenRouter, Vercel AI Gateway, Prime Inference) and embeds provenance in the generated file header.
  • Parsing is rewritten with strict validators that throw on malformed, empty, or HTTP-error responses, replacing silent fallbacks and empty-array returns.
  • Output is validated (every model field + TypeScript compilation) and written atomically via a temp file rename, so a failed run preserves the last reviewed snapshot.
  • A new Vitest test suite covers determinism, provenance, Kimi alias precedence, transactional rollback on failure, and validation error paths.
  • Risk: any CI step that relied on npm run build regenerating models will no longer do so; the snapshot is now a committed artifact.

Macroscope summarized 6d9155c.

@fettpl
fettpl marked this pull request as ready for review August 8, 2026 12:38
@fettpl

fettpl commented Aug 8, 2026

Copy link
Copy Markdown
Contributor Author

@snimu CI is fully green and independent review found no blocking issues. This is ready for maintainer review. It fixes #930 by making production builds consume a committed validated model snapshot and moving network refresh into an explicit deterministic atomic command.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make builds compile a reviewed, deterministic model snapshot

1 participant