fix(ci): lint and SAST the same files locally and in CI, and pin the two together - #11
Merged
Conversation
…two together
The pre-commit hooks were BROADER than CI, which is the dangerous direction. `ruff check` in CI named
six explicit paths; the ruff-check hook has no `exclude` and so runs on every changed Python file.
bandit in CI scanned `-r messagefoundry tee`; its hook scanned everything but tests/harness/samples.
So harness/ (73 findings), tee/ (12), samples/ and docker/ were gated locally and by NOTHING in CI.
Touching any file there failed the commit on errors no CI run could report and no green build had ever
been able to reveal. I hit it twice in one day: a harness import-sort during an unrelated flake fix,
and a one-line UP017 in scripts/ that I reverted rather than drag four unrelated bandit findings into
a release PR. A hook LOOSER than CI is an annoyance -- CI still catches it. A hook STRICTER than CI
blocks work on a standard the project does not enforce, and that is what makes people reach for
`--no-verify` and lose every other gate with it.
Scope now has ONE definition per tool instead of being re-stated, differently, in two files:
* ruff -- CI runs `ruff check .` (matching `ruff format --check .`, which was ALREADY repo-wide
while lint was not). pyproject's [tool.ruff] extend-exclude is the single source of truth.
* bandit -- CI runs `-r .` with an --exclude list identical to the hook's, and the hook's exclude
was widened to match. Widening surfaced 29 real findings that CI had never been able to see:
- 9 in scripts/ (subprocess in claim_check / vuln_metrics / assert_semgrep_handler_taint, one
urlopen, two try/except/continue in a hand-run Kerberos lab probe). All are established,
already-reviewed patterns -- two of them ALREADY carried `# noqa: S603`/`S310` for ruff's copy
of the same rules; bandit does not read noqa, so they now carry `# nosec` as well.
- 20 in packaging/messagefoundry-webconsole/tests/, literal test credentials (B105/B106). That
is the same "tests carry non-production idioms" case as tests/, one directory deeper, so the
exclude on BOTH sides now names it.
Recording a mistake worth not repeating: I first measured this as "0 findings, widening is free"
and said so. bandit was not installed in the worktree venv, `python -m bandit` failed, and the
JSON parse of its error swallowed the failure as an empty result set. The corrected run reports
`Total lines of code: 114206` -- a scanned-volume number the broken run never printed, and the
thing that distinguishes a clean scan from one that never ran.
99 findings cleared. 88 by safe auto-fix; the rest deliberately, because they were NOT interchangeable:
* B905 `zip()` without strict= -- three sites, three different answers, which is exactly why a blanket
`strict=False` would have been a fake fix. failover.py and sender.py build both operands from the
same source one line above, so strict=True is correct and will catch a future divergence that
zip's default would silently truncate. remote.py is the pairwise idiom `zip(x, x[1:])` where the
lengths differ BY DESIGN -- strict=True would raise on every non-empty input -- so it became
itertools.pairwise, which says "adjacent pairs" outright and drops the slice copy.
* SIM105 -- five sites ruff refused to auto-fix, all for the same reason: each carries a comment
(`# pragma: no cover`, "underlying socket already deleted", a two-line rationale) that the
contextlib.suppress transform would DESTROY. Two more would need a new import purely to satisfy a
style preference. Grandfathered per-line with the reason, which is the approach [tool.ruff.lint]
already documents for exactly this case.
docs/benchmarks/results/ is now ruff-excluded: those are archived measurement ARTIFACTS -- the exact
scripts a published number was produced with. Reformatting them edits the record, and a result you
cannot reproduce from the script filed beside it is worthless.
tests/test_lint_scope_parity.py reads both configurations and compares them, so narrowing one forces
narrowing the other. Mutation-verified in BOTH directions: re-narrowing CI's ruff fails it, and adding
an `exclude` to the ruff hook fails it. It asserts the CONTRACT, not a particular scope -- widen or
narrow freely, provided both sides move together.
Full suite green: 8914 passed, 818 skipped.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The pre-commit hooks were broader than CI — the dangerous direction.
ruff checkin CI named six explicit paths; theruff-checkhook has noexclude, so it runs on every changed Python file. bandit in CI scanned-r messagefoundry tee; its hook scanned everything excepttests/harness/samples.So
harness/(73 findings),tee/(12),samples/anddocker/were gated locally and by nothing in CI. Touching any file there failed the commit on errors no CI run could report, and that no green build had ever been able to reveal. I hit it twice in one day: aharness/import-sort during an unrelated flake fix, and a one-lineUP017inscripts/that I reverted rather than drag four unrelated bandit findings into a release PR.A hook looser than CI is an annoyance — CI still catches it. A hook stricter than CI blocks work on a standard the project doesn't enforce, and that's what makes people reach for
--no-verifyand lose every other gate with it.Scope now has one definition per tool
ruff check ., matchingruff format --check ., which was already repo-wide while lint was not.pyproject's[tool.ruff] extend-excludeis the single source of truth.-r .with an--excludelist identical to the hook's; the hook's exclude was widened to match.99 ruff findings cleared — 88 auto, 11 deliberately
They were not interchangeable:
B905—zip()withoutstrict=. Three sites, three different answers, which is exactly why a blanketstrict=Falsewould have been a fake fix.failover.py/sender.pybuild both operands from the same source one line above →strict=True, which will catch a future divergence that zip's default would silently truncate.remote.pyis the pairwise idiomzip(x, x[1:])where lengths differ by design —strict=Truewould raise on every non-empty input → becameitertools.pairwise.SIM105— five sites ruff refused to auto-fix, all for the same reason: each carries a comment (# pragma: no cover,"underlying socket already deleted", a two-line rationale) that thecontextlib.suppresstransform would destroy. Two more would need a new import purely for a style preference. Grandfathered per-line with the reason — the approach[tool.ruff.lint]already documents for this case.docs/benchmarks/results/is now ruff-excluded: archived measurement artifacts, the exact scripts a published number came from. Reformatting them edits the record.Widening bandit surfaced 29 findings CI had never seen
scripts/— subprocess, one urlopen, twotry/except/continuein a hand-run Kerberos lab probe. All established reviewed patterns; two already carried# noqa: S603/S310for ruff's copy of the same rules, and bandit doesn't readnoqa, so they now carry# nosectoo.packaging/messagefoundry-webconsole/tests/— literal test credentials (B105/B106). Same "tests carry non-production idioms" case astests/, one directory deeper; both excludes now name it.A mistake worth recording
I first measured this as "0 findings, widening is free" and said so. bandit wasn't installed in the worktree venv —
python -m banditfailed, and my JSON parse of its error swallowed the failure as an empty result set. The corrected run reportsTotal lines of code: 114206, a scanned-volume number the broken run never printed. That number is what distinguishes a clean scan from one that never ran, which is the same failure this PR exists to remove.Drift prevention
tests/test_lint_scope_parity.pyreads both configurations and compares them, so narrowing one forces narrowing the other. Mutation-verified in both directions: re-narrowing CI's ruff fails it, and adding anexcludeto the ruff hook fails it. It asserts the contract, not a particular scope — widen or narrow freely, provided both sides move together.Full suite green: 8914 passed, 818 skipped.
🤖 Generated with Claude Code