The verdict says how much of the documentation it actually covered - #21
Merged
Conversation
Measured, not imagined. Across nine public repositories docproof read 972 of 3,782 documentation files under the default scope - 25.7%. Re-running two of them over the whole tree took langwatch from 1 broken to 19 and cherry-studio from 23 to 111. Those runs were never clean; they were narrow. The information was already printed, in the header. The verdict was forty lines below it and said 'Nothing contradicted. 149 claims checked' on its own, which is the exact thing the README promises cannot happen: 'a clean report over two files in a project with three hundred cannot be mistaken for a clean report over three hundred.' Printed forty lines apart it can be, and the line anyone quotes from a CI log is the last one. So render() now takes read/unread and appends the coverage to the verdict itself - to the BROKEN verdict too, because '19 broken' over a fifth of a tree misleads exactly as much as 'nothing contradicted' over a fifth. Full coverage adds no sentence, because a reassurance on every clean run is the noise that teaches people to skip the block. Three tests. The second one failed first time because I built a fixture with no deleted path and asserted it was broken - the make_repo docstring is emphatic that a later removal is the whole rule, and I had ignored it. Fixed to use documented_before plus deleted, which is real drift rather than an illustration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The README promises this, and until now it was only half true:
The coverage note printed in the header. The verdict printed forty lines below it and said
Nothing contradicted. 149 claims checkedon its own. The line anyone quotes from a CI log isthe last one.
The measurement that prompted it
I ran docproof over nine public repositories and compared what it read against what was there:
972 of 3,782, or 25.7%.
Then I re-ran two of them with
--docs '**/*.md'. langwatch went from 1 broken to 19.cherry-studio went from 23 to 111. Those runs were never clean. They were narrow, and only
the header said so.
What changed
Report.render()takesreadandunreadand appends one sentence to the verdict:Attached to the broken verdict as well as the clean one, because "19 broken" over a fifth
of a tree misleads exactly as much as "nothing contradicted" over a fifth. A project whose
documentation was entirely read gains no sentence at all, because a reassurance printed on
every clean run is the kind of noise that teaches a reader to skip the whole block.
Three tests, 193 to 196, and the README now shows the real output.
One of those tests failed the first time and deserved to. I built a fixture with no deleted
path and asserted the run was broken.
make_repo's own docstring is emphatic that a laterremoval is what separates real drift from an illustration, and I had ignored it. Rewritten with
documented_beforeplusdeleted.This does not change the default scope, which is deliberate and argued for in
find_docs. Itchanges what a narrow run is allowed to sound like.