Skip to content

Restate the verification report's preamble, and declare its charset - #110

Open
jonfroehlich wants to merge 2 commits into
mainfrom
docs/eval-verification-preamble
Open

Restate the verification report's preamble, and declare its charset#110
jonfroehlich wants to merge 2 commits into
mainfrom
docs/eval-verification-preamble

Conversation

@jonfroehlich

@jonfroehlich jonfroehlich commented Aug 5, 2026

Copy link
Copy Markdown
Member

docs/eval_protocol_verification.html opened by quoting the chat instruction that produced it, verbatim and attributed by name:

Requested by Jon: "Are you absolutely certain there was an error in the original paper and our experimental approach? Double- and triple-check, produce visual examples of what we double-counted, and explain in greater detail."

That was fine for an internal note. It is now linked from the v1.1-corrected-eval release notes and from #9, so it reads as a public document, and an instruction quoted out of its conversation is both informal and less informative than the thing it asked for. The preamble now states the work's scope directly, keeping every element of the original ask — certainty, independent confirmation rather than a single line of evidence, worked visual examples, and a quantified effect:

A re-examination of the matching rule used to score RampNet's detections, prompted by a discrepancy identified after publication. Its scope: to establish whether predictions were genuinely over-counted, to confirm that finding by independent means rather than rest on a single line of evidence, to document the over-count with worked examples drawn from the released model's own output, and to quantify its effect on the published metrics.

The <h1> is neutralised in the same spirit (624056d): "Did we really over-count? A re-verification of the published evaluation protocol" → "Re-verification of the published evaluation protocol". The old heading was the question the investigation started from, phrased in the first person and addressed to whoever commissioned it. The subject it carried — the over-count — is stated by the preamble immediately below, so the document loses nothing. The <title> was already neutral and is unchanged.

Second change, one line: the file never declared a charset. It is valid UTF-8, but with no <meta charset> a browser opening it from file:// falls back to the locale's legacy encoding and renders every em dash and curly quote as â€". Since the release notes tell readers to "open it in a browser", that is the path most likely to be hit. The bytes are untouched — this adds a declaration, it does not re-encode.

No change to any finding, number, figure or section of the report; the diff is the two header lines plus the meta tag.

Worth landing before the v1.1-corrected-eval tag is cut, since the tag will freeze whatever this file says.

🤖 Generated with Claude Code (claude-opus-5[1m])

jonfroehlich and others added 2 commits August 5, 2026 05:36
The report opened by quoting, verbatim and by name, the chat instruction that
prompted it. That was fine for an internal note; it is now linked from the
paper-boundary release notes, where it reads as a public document. The preamble
states the work's scope in its own right instead, preserving what the original
instruction actually asked for: certainty, independent confirmation rather than a
single line of evidence, worked visual examples, and a quantified effect.

Also adds the missing <meta charset="utf-8">. The file is valid UTF-8 but never
declared it, so a browser opening it from file:// falls back to the locale's
legacy encoding and renders every em dash and curly quote as mojibake. The bytes
are untouched — this is a one-line declaration, not a re-encode.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"Did we really over-count?" was the question the investigation started from, and
it read as one — first person, rhetorical, addressed to the person who asked for
the work. The heading now names the document; the subject it used to carry (the
over-count) is stated by the preamble directly below it, so nothing is lost.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant