Skip to content

The verdict says how much of the documentation it actually covered - #21

Merged
melbinjp merged 1 commit into
mainfrom
docs/verdict-carries-coverage
Aug 19, 2026
Merged

The verdict says how much of the documentation it actually covered#21
melbinjp merged 1 commit into
mainfrom
docs/verdict-carries-coverage

Conversation

@melbinjp

Copy link
Copy Markdown
Owner

The README promises this, and until now it was only half true:

a clean report over two files in a project with three hundred cannot be mistaken for a clean
report over three hundred

The coverage note printed in the header. The verdict printed forty lines below it and said
Nothing contradicted. 149 claims checked on its own. The line anyone quotes from a CI log is
the last one.

The measurement that prompted it

I ran docproof over nine public repositories and compared what it read against what was there:

repo read not read coverage
zeroclaw 209 55 79%
paperclip 145 231 39%
sentry-react-native 8 18 31%
tutti 92 234 28%
hive 54 162 25%
cherry-studio 120 418 22%
gsd-core 307 1218 20%
opensre 26 102 20%
langwatch 11 372 3%

972 of 3,782, or 25.7%.

Then I re-ran two of them with --docs '**/*.md'. langwatch went from 1 broken to 19.
cherry-studio went from 23 to 111.
Those runs were never clean. They were narrow, and only
the header said so.

What changed

Report.render() takes read and unread and appends one sentence to the verdict:

1 broken, 149 checked, 497 not judged.
This judged 11 of 383 documentation file(s). 372 were never read, so this verdict
covers 2% of the documentation in this project.

Attached to the broken verdict as well as the clean one, because "19 broken" over a fifth
of a tree misleads exactly as much as "nothing contradicted" over a fifth. A project whose
documentation was entirely read gains no sentence at all, because a reassurance printed on
every clean run is the kind of noise that teaches a reader to skip the whole block.

Three tests, 193 to 196, and the README now shows the real output.

One of those tests failed the first time and deserved to. I built a fixture with no deleted
path and asserted the run was broken. make_repo's own docstring is emphatic that a later
removal is what separates real drift from an illustration, and I had ignored it. Rewritten with
documented_before plus deleted.

This does not change the default scope, which is deliberate and argued for in find_docs. It
changes what a narrow run is allowed to sound like.

Measured, not imagined. Across nine public repositories docproof read 972 of 3,782 documentation files under the default scope - 25.7%. Re-running two of them over the whole tree took langwatch from 1 broken to 19 and cherry-studio from 23 to 111. Those runs were never clean; they were narrow.

The information was already printed, in the header. The verdict was forty lines below it and said 'Nothing contradicted. 149 claims checked' on its own, which is the exact thing the README promises cannot happen: 'a clean report over two files in a project with three hundred cannot be mistaken for a clean report over three hundred.' Printed forty lines apart it can be, and the line anyone quotes from a CI log is the last one.

So render() now takes read/unread and appends the coverage to the verdict itself - to the BROKEN verdict too, because '19 broken' over a fifth of a tree misleads exactly as much as 'nothing contradicted' over a fifth. Full coverage adds no sentence, because a reassurance on every clean run is the noise that teaches people to skip the block.

Three tests. The second one failed first time because I built a fixture with no deleted path and asserted it was broken - the make_repo docstring is emphatic that a later removal is the whole rule, and I had ignored it. Fixed to use documented_before plus deleted, which is real drift rather than an illustration.
@melbinjp
melbinjp merged commit 6be8d84 into main Aug 19, 2026
21 checks passed
@melbinjp
melbinjp deleted the docs/verdict-carries-coverage branch August 19, 2026 09:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant