Skip to content

build: couple the grpcio-tools floor to the committed protobuf gencode - #19

Merged
jonathanstokes merged 3 commits into
mainfrom
claude/loving-jones-3e52a5
Aug 12, 2026
Merged

build: couple the grpcio-tools floor to the committed protobuf gencode#19
jonathanstokes merged 3 commits into
mainfrom
claude/loving-jones-3e52a5

Conversation

@jonathanstokes

@jonathanstokes jonathanstokes commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

The trap

ADR-0001 says the generated bindings and the protobuf pin are one unit. Nothing enforced it.

grpcio-tools carries its own copy of protoc, so whichever version is installed is what decides the gencode written into pyquadcortex/protocol/proto/*_pb2.py. The committed bindings are stamped 7.35.1. The dev extra's floor was grpcio-tools>=1.68, and resolved at its lowest that gives 1.82.1, whose protoc emits 7.35.0.

So running scripts/compile_protos.sh after a clean pip install -e ".[dev]" could quietly move the bindings one patch backwards, and nothing anywhere would say so. The protobuf runtime validates runtime >= gencode and nothing else, so older bindings import cleanly and pass the entire suite while pyproject.toml keeps claiming a pin the tree no longer matches.

The failure is quiet in the direction that matters. A gencode that is too NEW is a hard ImportError for every user and gets caught immediately. A gencode that is too OLD looks exactly like success.

It gets worse than one patch outside a clean resolve. A venv that picked up grpcio-tools some other way, or the script's fallback to a system protoc, is bounded by nothing at all - grpcio-tools 1.68.0 emits gencode 5.28.1, seven minors back.

What the versions actually are

The floor is not derivable from package metadata, and the two version lines do not track each other. Each of these was installed and run, and the stamp it writes was read:

grpcio-tools libprotoc gencode it emits what its metadata declares
1.68.0 (the old floor) 28.1 5.28.1 protobuf>=5.26.1,<6
1.81.1 33.5 6.33.5 protobuf>=6.33.5,<7
1.82.0 35.0 7.35.0 protobuf>=6.33.5,<8
1.82.1 35.0 7.35.0 protobuf>=7.35.1,<8
1.83.0 (latest) 35.1 7.35.1 protobuf>=7.35.1,<8

Look at 1.82.1. It declares a protobuf floor of 7.35.1 and still emits gencode 7.35.0. The declared dependency is a runtime floor, not the gencode stamp, and reading it as the mapping gets the wrong answer by one patch - which is exactly enough to walk the pin backwards. Ask the compiler, never the metadata.

Two corrections to the report that started this, since both change what the fix has to be:

  • grpcio-tools 1.83.0 does not emit gencode 5.29.3. It emits 7.35.1, which is why the venv on this machine was never actually wrong. The hole was in the floor, which allowed versions nobody had checked.
  • 1.68.0 was not reachable through the documented install. It requires protobuf<6, which the project's own protobuf>=7.35.1,<8 rules out, so the resolver cannot land on it. The reachable exposure was 1.82.1 at one patch. The hole is real either way, and the reachable case is the better argument for closing it.

What changed

The floor. grpcio-tools>=1.68 becomes grpcio-tools>=1.83.0, the oldest release whose protoc emits gencode 7.35.1, with the reason written next to it in pyproject.toml. 1.83.0 is the current latest (released 2026-07-23; wheels for cp310 through cp314 on Linux, macOS and Windows), so the floor is the newest release. That is not a problem to route around: this is a dev-only extra, and the floor has to move with the gencode anyway.

A gate at regeneration. scripts/compile_protos.sh now generates into a temporary directory, compares the gencode it produced against the bindings already in the tree, and refuses to install anything older. On refusal it writes nothing, so the tree is exactly as it was, and the message names both versions and how to fix it. Because the gate reads generator OUTPUT rather than a package version, it covers the system-protoc fallback too.

A check in CI. tests/test_packaging.py asserts three things about the committed state, offline and with no toolchain: all bindings carry one gencode, the pin's floor equals that gencode, and the pin's ceiling is one major above it.

Why both, and not one

They catch different routes and neither covers the other.

The script is the only guard that can stop the downgrade before it reaches the tree, and it is where the mistake is actually made, so it is the right place for the explanation. But it never runs in CI, so it cannot police what lands on main, and it only sees gencode that arrives through the script.

The test is the only guard that runs on every PR, needs no protoc, and holds for gencode that arrived any other way - a hand edit, an IDE-run protoc, a pin bumped without regenerating. But it fires after the files are already overwritten, and it cannot say "your generator is too old" because the offline suite has no generator to ask.

Recorded as ADR-0008, with the rejected options.

Two things the ADR is explicit about, because both are easy to overread:

  • The pin floor and the committed gencode must be equal, not merely compatible. A floor above the gencode still imports for everybody, which is precisely why it needs catching.
  • What CI proves is the bindings-to-pin half. Nothing machine-checks the grpcio-tools floor itself - judging a floor means running that generator, and the offline suite has none. A future gencode bump that updates the bindings and the pin but forgets the floor leaves CI green until the next regeneration, where the script catches it and names the stale floor as the cause. That residual exposure is a delay, not a silent pass.

Test plan

  • 514 passed, 1 skipped on the offline suite, in a fresh worktree venv built from the new dev extra (uv venv && uv pip install -e ".[dev]", resolving to grpcio-tools 1.83.0 / protobuf 7.35.1)
  • The three new tests were each made to fail on purpose by temporarily editing the committed stamps: one file at 7.35.0 (mismatched generators), both at 7.35.0 (floor above gencode), both at 8.36.0 (floor below gencode, and a ceiling left behind a major). Every message named the problem and what to move. Bindings restored with git checkout after each
  • scripts/compile_protos.sh against grpcio-tools 1.83.0: byte-identical to what is committed, tree clean
  • Against 1.82.1 (7.35.0, one patch back) and 1.68.0 (5.28.1): refused, exit 1, git status on the bindings directory empty in both cases
  • Against a tree whose bindings carry no version stamp at all: refused. This one had to be fixed first - see below
  • older_than checked against 8 cases, including unequal component counts (7.35 vs 7.35.1) and two-digit fields (7.9.0 vs 7.35.1, 10.0.0 vs 9.9.9)
  • The resolver claims above were checked with uv pip compile --resolution lowest-direct on the old constraint set, and by trying to install 1.68.0 alongside the project's own protobuf pin

The bindings are not regenerated here. They are byte-identical to main. Regenerating is its own change with its own pin bump, per ADR-0001 and CLAUDE.md.

Reviewer notes

Seven findings across a pre-PR self-review and a triage pass. Six fixed, in c927428 and b26572a; both commits are worth reading on their own. The two that mattered:

  1. The gate could fail open on the case it should trust least. gencode_of returned an empty string both for "file is missing" and for "file carries no version stamp", and the loop read them as one: nothing to compare, skip. Bindings from a pre-stamp protoc have no stamp line at all, so that path would have waved in any generator. Reproduced before fixing: with both stamps stripped, a 7.35.0 generator wrote over 7.35.1 bindings and exited 0. Missing and unstamped are separate answers now, and unstamped refuses.
  2. The gate could also fail open on a SIGPIPE race. Both version helpers piped into head -1, and under set -o pipefail a producer that takes SIGPIPE when head exits early fails the whole pipeline; inside $(...) that returns an empty string, which the comparison reads as "not older". Both helpers are single awk processes now, with no pipe to race.

Also fixed: the ADR's overstated magnitude and its "proved in CI" claim (both above); protobuf\b in the test matching protobuf-stubs; a bare "No such file or directory" when protoc writes nothing; and messages saying "committed" while comparing against the working tree.

Not fixed, deliberately: _bound mis-parses pin shapes it does not expect (<=, ~=, environment markers). Every one of them fails the assert rather than passing it, and the message prints the pin string it parsed, so the misdirection is one line deep. ADR-0001 fixes the shape of this pin; handling others is speculative generality.

ADR-0001 makes the generated bindings and the protobuf runtime pin one unit,
but nothing enforced it. grpcio-tools carries its own protoc, so the installed
version decides the gencode written into the bindings, and the dev extra's
floor was >=1.68 - whose protoc emits gencode 5.28.1 against bindings committed
at 7.35.1. The protobuf runtime only validates runtime >= gencode, so a
regeneration that walked the pin seven minors backwards would have imported
cleanly and passed the whole suite.

Raise the floor to grpcio-tools>=1.83.0, the oldest release whose protoc emits
gencode 7.35.1. The number is not derivable from package metadata: 1.82.1
declares protobuf>=7.35.1 and still emits gencode 7.35.0, so each candidate was
run and the stamp it writes was read.

Two guards, because prevention and detection are different jobs:

- scripts/compile_protos.sh now generates into a temporary directory, compares
  the gencode against the committed bindings and refuses to install anything
  older. On refusal the tree is untouched. It is the only place that can stop
  the downgrade before it lands, and it is where the mistake is made.
- tests/test_packaging.py proves the committed state on every PR, with no
  toolchain: one generator across all bindings, the pin floor equal to the
  committed gencode, the ceiling one major above it. It covers gencode that
  arrived by any other route.

The bindings themselves are unchanged - regenerating is its own change with its
own pin bump (ADR-0001). Recorded as ADR-0008.
- The gencode gate could fail OPEN. Both helpers piped into `head -1`, and
  under `set -o pipefail` a producer that takes SIGPIPE when head exits early
  fails the pipeline; inside `$(...)` that comes back as an empty string, which
  the comparison reads as "not older". The one case the gate exists to catch is
  the one that could slip through it. Both are single awk processes now, with
  no pipe to race.
- `_protobuf_pin` matched on `protobuf\b`, which also matches `protobuf-stubs`
  and any other hyphenated name. Anchored on a version operator instead.
- The gate could fail OPEN on the case it should trust least. `gencode_of`
  returns an empty string both for "file is missing" and for "file has no
  version stamp", and the loop read them as one: nothing to compare, skip.
  Bindings from a pre-stamp protoc carry no stamp line at all, so that path
  would have waved in any generator. Verified before the fix: with both stamps
  stripped, a 7.35.0 generator wrote over 7.35.1 bindings and exited 0. Missing
  and unstamped are now separate answers, and unstamped refuses. Deleting the
  file is the deliberate way to say "replace this".
- ADR-0008 said a resolver was free to install grpcio-tools 1.68.0 and fall
  seven gencode minors. It was not: 1.68.0 requires protobuf<6, which the
  project's own protobuf>=7.35.1 rules out, so `pip install -e ".[dev]"` cannot
  land on it. Resolved at its lowest the old floor gives 1.82.1, emitting
  gencode 7.35.0 - one patch back, and reachable through the documented install,
  which is the better argument anyway. 1.68.0 needs a venv that got
  grpcio-tools some other way, or the system-protoc fallback, which no floor
  constrains. Corrected in ADR.md, STEERING.md and changelog.md.
- ADR-0008 claimed the floor is "proved in CI". Only the bindings-to-pin half
  is. Nothing machine-checks the floor itself, because judging a floor means
  running that generator and the offline suite has none. Title says what CI
  actually checks, and Consequences names the gap: a gencode bump that forgets
  the floor leaves CI green until the next regeneration, where the script
  catches it.
- protoc exiting 0 having written nothing reached the `cp` as a bare "No such
  file or directory". It names the likely cause now (a `package` statement in
  the schema sending output into a subdirectory).
- The gate's messages said "committed" while comparing against the file on
  disk, which need not be what is in HEAD. They say "the tree".

Not changed: `_bound`'s parsing of pin shapes it does not expect (`<=`, `~=`,
environment markers). Every one of them fails the assert rather than passing it,
and the message prints the pin string it parsed, so the misdirection is one line
deep. ADR-0001 fixes the shape of this pin; handling others is speculative.
@jonathanstokes
jonathanstokes marked this pull request as ready for review August 12, 2026 23:05
@jonathanstokes
jonathanstokes merged commit e4a40b3 into main Aug 12, 2026
4 checks passed
@jonathanstokes
jonathanstokes deleted the claude/loving-jones-3e52a5 branch August 12, 2026 23:18
jonathanstokes added a commit that referenced this pull request Aug 12, 2026
PR #19 landed first and took ADR-0008 for the generator floor, so the listener
record is renumbered ADR-0009 - in ADR.md, and in every reference to it
(transport.py, CLAUDE.md, STEERING.md, architecture.md, changelog.md). The two
review commits on this branch predate the renumber and still say 0008; STEERING's
change-log entry says so rather than leaving a reader to work it out.

Both change-log entries and both ADR table rows are kept. Nothing else conflicted:
main's changes are the build/gencode guard and its docs, which touch no file this
branch changes for a shared reason.

Offline suite after the merge, including main's new packaging tests: 529 passed,
1 skipped.
jonathanstokes added a commit that referenced this pull request Aug 13, 2026
Two text conflicts and one substantive one.

ADR numbering: main took 0008 (grpcio-tools floor) and 0009 (persistent
listeners), so this branch's record renumbers to ADR-0010. The lifecycle
is append-only, and theirs landed first. Every reference that meant this
branch's ADR moved with it; the grpcio ones are left alone.

The substantive conflict is that PR #20 shipped exactly the hook this
branch's harness needed. state_snapshot._Tap monkey-patched
transport._dispatch, reaching past the public surface and unhookable
out of order; it now subscribes through Transport.add_listener
(ADR-0009). Verified the two are equivalent rather than assuming it: run
side by side against the same 14-second window they saw the identical
478 messages, the same 23 types, in the same order.

docs/capture.md's listener recipe monkey-patched _dispatch too, and it
is the document that teaches this. Rewritten onto add_listener, with the
two rules that come with a listener - do not block, do not read from the
device - both of which the transport now enforces.

That comparison turned up a real defect. A capture's type coverage
varies run to run, because the handshake's reply burst lands lazily (the
File enumeration takes 10-25 s) and a fixed window catches a different
tail each time: two runs here saw 23 types and 12. diff() was rendering
a type present in one snapshot and absent from the other field by field
as "<absent> -> value", which reads exactly like a discovery. It now
reports one line in the noise bucket naming what actually happened.

Hardware, after the merge: the capture and the write regression both
pass through the listener-based harness, and the unit is left in PRESET.
557 offline.
jonathanstokes added a commit that referenced this pull request Aug 13, 2026
Three change logs conflicted, all in the same place: main's Unreleased section
and this branch's both grew an entry after the `Device` one. Kept both, in the
order they landed.

- changelog.md: the broadcast listener (#20) then the translation groundwork
- docs/STEERING.md: the boundary entry sits above TEMPO MODE (#22) and the
  listener (#20), which is where the newest entry goes in that file
- docs/domain-model.md: TEMPO MODE closing then principle 5 being built, which
  is the order that file reads in
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant