build: couple the grpcio-tools floor to the committed protobuf gencode - #19
Merged
Conversation
ADR-0001 makes the generated bindings and the protobuf runtime pin one unit, but nothing enforced it. grpcio-tools carries its own protoc, so the installed version decides the gencode written into the bindings, and the dev extra's floor was >=1.68 - whose protoc emits gencode 5.28.1 against bindings committed at 7.35.1. The protobuf runtime only validates runtime >= gencode, so a regeneration that walked the pin seven minors backwards would have imported cleanly and passed the whole suite. Raise the floor to grpcio-tools>=1.83.0, the oldest release whose protoc emits gencode 7.35.1. The number is not derivable from package metadata: 1.82.1 declares protobuf>=7.35.1 and still emits gencode 7.35.0, so each candidate was run and the stamp it writes was read. Two guards, because prevention and detection are different jobs: - scripts/compile_protos.sh now generates into a temporary directory, compares the gencode against the committed bindings and refuses to install anything older. On refusal the tree is untouched. It is the only place that can stop the downgrade before it lands, and it is where the mistake is made. - tests/test_packaging.py proves the committed state on every PR, with no toolchain: one generator across all bindings, the pin floor equal to the committed gencode, the ceiling one major above it. It covers gencode that arrived by any other route. The bindings themselves are unchanged - regenerating is its own change with its own pin bump (ADR-0001). Recorded as ADR-0008.
- The gencode gate could fail OPEN. Both helpers piped into `head -1`, and under `set -o pipefail` a producer that takes SIGPIPE when head exits early fails the pipeline; inside `$(...)` that comes back as an empty string, which the comparison reads as "not older". The one case the gate exists to catch is the one that could slip through it. Both are single awk processes now, with no pipe to race. - `_protobuf_pin` matched on `protobuf\b`, which also matches `protobuf-stubs` and any other hyphenated name. Anchored on a version operator instead.
- The gate could fail OPEN on the case it should trust least. `gencode_of` returns an empty string both for "file is missing" and for "file has no version stamp", and the loop read them as one: nothing to compare, skip. Bindings from a pre-stamp protoc carry no stamp line at all, so that path would have waved in any generator. Verified before the fix: with both stamps stripped, a 7.35.0 generator wrote over 7.35.1 bindings and exited 0. Missing and unstamped are now separate answers, and unstamped refuses. Deleting the file is the deliberate way to say "replace this". - ADR-0008 said a resolver was free to install grpcio-tools 1.68.0 and fall seven gencode minors. It was not: 1.68.0 requires protobuf<6, which the project's own protobuf>=7.35.1 rules out, so `pip install -e ".[dev]"` cannot land on it. Resolved at its lowest the old floor gives 1.82.1, emitting gencode 7.35.0 - one patch back, and reachable through the documented install, which is the better argument anyway. 1.68.0 needs a venv that got grpcio-tools some other way, or the system-protoc fallback, which no floor constrains. Corrected in ADR.md, STEERING.md and changelog.md. - ADR-0008 claimed the floor is "proved in CI". Only the bindings-to-pin half is. Nothing machine-checks the floor itself, because judging a floor means running that generator and the offline suite has none. Title says what CI actually checks, and Consequences names the gap: a gencode bump that forgets the floor leaves CI green until the next regeneration, where the script catches it. - protoc exiting 0 having written nothing reached the `cp` as a bare "No such file or directory". It names the likely cause now (a `package` statement in the schema sending output into a subdirectory). - The gate's messages said "committed" while comparing against the file on disk, which need not be what is in HEAD. They say "the tree". Not changed: `_bound`'s parsing of pin shapes it does not expect (`<=`, `~=`, environment markers). Every one of them fails the assert rather than passing it, and the message prints the pin string it parsed, so the misdirection is one line deep. ADR-0001 fixes the shape of this pin; handling others is speculative.
jonathanstokes
marked this pull request as ready for review
August 12, 2026 23:05
jonathanstokes
added a commit
that referenced
this pull request
Aug 12, 2026
PR #19 landed first and took ADR-0008 for the generator floor, so the listener record is renumbered ADR-0009 - in ADR.md, and in every reference to it (transport.py, CLAUDE.md, STEERING.md, architecture.md, changelog.md). The two review commits on this branch predate the renumber and still say 0008; STEERING's change-log entry says so rather than leaving a reader to work it out. Both change-log entries and both ADR table rows are kept. Nothing else conflicted: main's changes are the build/gencode guard and its docs, which touch no file this branch changes for a shared reason. Offline suite after the merge, including main's new packaging tests: 529 passed, 1 skipped.
jonathanstokes
added a commit
that referenced
this pull request
Aug 13, 2026
Two text conflicts and one substantive one. ADR numbering: main took 0008 (grpcio-tools floor) and 0009 (persistent listeners), so this branch's record renumbers to ADR-0010. The lifecycle is append-only, and theirs landed first. Every reference that meant this branch's ADR moved with it; the grpcio ones are left alone. The substantive conflict is that PR #20 shipped exactly the hook this branch's harness needed. state_snapshot._Tap monkey-patched transport._dispatch, reaching past the public surface and unhookable out of order; it now subscribes through Transport.add_listener (ADR-0009). Verified the two are equivalent rather than assuming it: run side by side against the same 14-second window they saw the identical 478 messages, the same 23 types, in the same order. docs/capture.md's listener recipe monkey-patched _dispatch too, and it is the document that teaches this. Rewritten onto add_listener, with the two rules that come with a listener - do not block, do not read from the device - both of which the transport now enforces. That comparison turned up a real defect. A capture's type coverage varies run to run, because the handshake's reply burst lands lazily (the File enumeration takes 10-25 s) and a fixed window catches a different tail each time: two runs here saw 23 types and 12. diff() was rendering a type present in one snapshot and absent from the other field by field as "<absent> -> value", which reads exactly like a discovery. It now reports one line in the noise bucket naming what actually happened. Hardware, after the merge: the capture and the write regression both pass through the listener-based harness, and the unit is left in PRESET. 557 offline.
jonathanstokes
added a commit
that referenced
this pull request
Aug 13, 2026
Three change logs conflicted, all in the same place: main's Unreleased section and this branch's both grew an entry after the `Device` one. Kept both, in the order they landed. - changelog.md: the broadcast listener (#20) then the translation groundwork - docs/STEERING.md: the boundary entry sits above TEMPO MODE (#22) and the listener (#20), which is where the newest entry goes in that file - docs/domain-model.md: TEMPO MODE closing then principle 5 being built, which is the order that file reads in
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The trap
ADR-0001 says the generated bindings and the
protobufpin are one unit. Nothing enforced it.grpcio-toolscarries its own copy of protoc, so whichever version is installed is what decides the gencode written intopyquadcortex/protocol/proto/*_pb2.py. The committed bindings are stamped 7.35.1. The dev extra's floor wasgrpcio-tools>=1.68, and resolved at its lowest that gives 1.82.1, whose protoc emits 7.35.0.So running
scripts/compile_protos.shafter a cleanpip install -e ".[dev]"could quietly move the bindings one patch backwards, and nothing anywhere would say so. The protobuf runtime validatesruntime >= gencodeand nothing else, so older bindings import cleanly and pass the entire suite whilepyproject.tomlkeeps claiming a pin the tree no longer matches.The failure is quiet in the direction that matters. A gencode that is too NEW is a hard
ImportErrorfor every user and gets caught immediately. A gencode that is too OLD looks exactly like success.It gets worse than one patch outside a clean resolve. A venv that picked up
grpcio-toolssome other way, or the script's fallback to a systemprotoc, is bounded by nothing at all - grpcio-tools 1.68.0 emits gencode 5.28.1, seven minors back.What the versions actually are
The floor is not derivable from package metadata, and the two version lines do not track each other. Each of these was installed and run, and the stamp it writes was read:
protobuf>=5.26.1,<6protobuf>=6.33.5,<7protobuf>=6.33.5,<8protobuf>=7.35.1,<8protobuf>=7.35.1,<8Look at 1.82.1. It declares a
protobuffloor of 7.35.1 and still emits gencode 7.35.0. The declared dependency is a runtime floor, not the gencode stamp, and reading it as the mapping gets the wrong answer by one patch - which is exactly enough to walk the pin backwards. Ask the compiler, never the metadata.Two corrections to the report that started this, since both change what the fix has to be:
protobuf<6, which the project's ownprotobuf>=7.35.1,<8rules out, so the resolver cannot land on it. The reachable exposure was 1.82.1 at one patch. The hole is real either way, and the reachable case is the better argument for closing it.What changed
The floor.
grpcio-tools>=1.68becomesgrpcio-tools>=1.83.0, the oldest release whose protoc emits gencode 7.35.1, with the reason written next to it inpyproject.toml. 1.83.0 is the current latest (released 2026-07-23; wheels for cp310 through cp314 on Linux, macOS and Windows), so the floor is the newest release. That is not a problem to route around: this is a dev-only extra, and the floor has to move with the gencode anyway.A gate at regeneration.
scripts/compile_protos.shnow generates into a temporary directory, compares the gencode it produced against the bindings already in the tree, and refuses to install anything older. On refusal it writes nothing, so the tree is exactly as it was, and the message names both versions and how to fix it. Because the gate reads generator OUTPUT rather than a package version, it covers the system-protocfallback too.A check in CI.
tests/test_packaging.pyasserts three things about the committed state, offline and with no toolchain: all bindings carry one gencode, the pin's floor equals that gencode, and the pin's ceiling is one major above it.Why both, and not one
They catch different routes and neither covers the other.
The script is the only guard that can stop the downgrade before it reaches the tree, and it is where the mistake is actually made, so it is the right place for the explanation. But it never runs in CI, so it cannot police what lands on main, and it only sees gencode that arrives through the script.
The test is the only guard that runs on every PR, needs no protoc, and holds for gencode that arrived any other way - a hand edit, an IDE-run protoc, a pin bumped without regenerating. But it fires after the files are already overwritten, and it cannot say "your generator is too old" because the offline suite has no generator to ask.
Recorded as ADR-0008, with the rejected options.
Two things the ADR is explicit about, because both are easy to overread:
grpcio-toolsfloor itself - judging a floor means running that generator, and the offline suite has none. A future gencode bump that updates the bindings and the pin but forgets the floor leaves CI green until the next regeneration, where the script catches it and names the stale floor as the cause. That residual exposure is a delay, not a silent pass.Test plan
514 passed, 1 skippedon the offline suite, in a fresh worktree venv built from the new dev extra (uv venv && uv pip install -e ".[dev]", resolving to grpcio-tools 1.83.0 / protobuf 7.35.1)git checkoutafter eachscripts/compile_protos.shagainst grpcio-tools 1.83.0: byte-identical to what is committed, tree cleangit statuson the bindings directory empty in both casesolder_thanchecked against 8 cases, including unequal component counts (7.35 vs 7.35.1) and two-digit fields (7.9.0 vs 7.35.1, 10.0.0 vs 9.9.9)uv pip compile --resolution lowest-directon the old constraint set, and by trying to install 1.68.0 alongside the project's own protobuf pinThe bindings are not regenerated here. They are byte-identical to main. Regenerating is its own change with its own pin bump, per ADR-0001 and CLAUDE.md.
Reviewer notes
Seven findings across a pre-PR self-review and a triage pass. Six fixed, in
c927428andb26572a; both commits are worth reading on their own. The two that mattered:gencode_ofreturned an empty string both for "file is missing" and for "file carries no version stamp", and the loop read them as one: nothing to compare, skip. Bindings from a pre-stamp protoc have no stamp line at all, so that path would have waved in any generator. Reproduced before fixing: with both stamps stripped, a 7.35.0 generator wrote over 7.35.1 bindings and exited 0. Missing and unstamped are separate answers now, and unstamped refuses.head -1, and underset -o pipefaila producer that takes SIGPIPE when head exits early fails the whole pipeline; inside$(...)that returns an empty string, which the comparison reads as "not older". Both helpers are singleawkprocesses now, with no pipe to race.Also fixed: the ADR's overstated magnitude and its "proved in CI" claim (both above);
protobuf\bin the test matchingprotobuf-stubs; a bare "No such file or directory" when protoc writes nothing; and messages saying "committed" while comparing against the working tree.Not fixed, deliberately:
_boundmis-parses pin shapes it does not expect (<=,~=, environment markers). Every one of them fails the assert rather than passing it, and the message prints the pin string it parsed, so the misdirection is one line deep. ADR-0001 fixes the shape of this pin; handling others is speculative generality.