ci: verify committed models match regeneration from the pinned UCP spec - #62
Merged
damaz91 merged 3 commits intoAug 5, 2026
Merged
Conversation
The Tests workflow installs the committed generated models and runs the unit suite, but never executes the generation pipeline itself (generate_models.sh + preprocess_schemas.py + postprocess_models.py end-to-end). A defect in the pipeline's wiring stays invisible as long as the committed models are fine, and models edited or left stale without regeneration are equally invisible. This is not hypothetical: with the floating datamodel-code-generator>=0.50.0 bound, the 0.72.0 release (2026-08-03) mis-resolves the spec's remote $refs (HTTP 404 on a doubled path), generation emits incomplete models, postprocess_models.py exits 1 -- and CI stays green because it only ever tests the committed artifacts. Add a model-drift job that regenerates the models against the pinned UCP spec version for this SDK line (0.4.x -> 2026-04-08, per the README compatibility table), normalizes file endings the way pre-commit does, and fails if the result is not byte-identical to the committed models. This catches a broken generator and uncommitted regeneration in one check, without changing the release flow of committing generated artifacts. Pin the codegen toolchain (datamodel-code-generator==0.71.0, ruff==0.16.1) so regeneration is reproducible; 0.71.0 + ruff 0.16.1 reproduce the committed models byte-for-byte after end-of-file normalization.
damaz91
added
status:needs-triage
damaz91
approved these changes
Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Observed
tests.ymlinstalls the committed models and runsunittest discover; nothing in CI exercises the generation pipeline end to end. Three concrete consequences, all reproduced on current main:./generate_models.sh 2026-04-08fails on a clean checkout because the dev dependency bound is a floatingdatamodel-code-generator>=0.50.0, and 0.72.0 (released yesterday) mis resolves remote refs (HTTP 404 .../shopping/shopping/shopping/types/product_option.json), after whichpostprocess_models.pyexits 1 ('Description' has no generated class). The exact CI steps still pass._patch_unique_items()call inpostprocess_models.py(committed models untouched) leaves CI green:Ran 45 tests ... OK.skipUnless(HAVE_SDK, ...)and unittest reportsOK (skipped=15), exit 0.The unit tests added with the recent validator work cover the postprocess functions in isolation and the committed models behaviorally; the wiring between the generator, the postprocessor and the committed artifacts is what has no coverage.
Change
A
model-driftCI job: regenerate against the pinned spec release the 0.4.x line targets (./generate_models.sh 2026-04-08, per the README compatibility table), normalize end of file (the only delta pre-commit introduces on generated output), and fail with an actionable message if the result differs from the committed models. Plus exact pins for the two generation-critical dev dependencies (datamodel-code-generator==0.71.0,ruff==0.16.1), with a comment documenting the 0.72.0 breakage; these versions reproduce the committed models byte for byte.This deliberately does NOT regenerate at build time or auto commit anything (the committed artifact flow stays the release path, consistent with the direction in #10): it only asserts that what is committed and what the pipeline produces are the same thing, which catches both a broken generator and a forgotten regeneration.
Verification
Green on clean HEAD (twice, including an end to end run of the exact job commands). Red in both failure directions: the gutted injector mutation (30 deleted validator lines detected) and a deliberately staled committed model (6 line diff detected). Full existing suite on the final tree:
Ran 45 tests ... OK. One caveat stated up front: therelease/2026-04-08branch has received cherry picks before (for example the attribution feature); if it moves again this job goes red on unrelated PRs, which is the intended signal that the committed models are stale relative to the pinned spec, and regenerating clears it.