docs: warn packagers that a system uchardet detects non-UTF-8 less accurately

[wbarnha]

Stacked on #64 (same pattern as #65), so it targets that branch rather than master.

Why

#64's README section presents enabled as the recommended choice for distro packaging. It does not mention that the resulting build detects non-UTF-8 input materially worse than the wheels do.

The bundled copy compiles the encoding-only replacement for nsMBCSGroupProber added in #62. A system build links upstream's prober, so it loses both the throughput fix and the rejection of non-UTF-8 sequences that upstream reports as UTF-8.

Measured over the in-tree corpus (benchmarks/make_nonutf8_corpus.py, 1650 documents, 7 encodings, median of 3 runs; system library built from our pinned uchardet commit so the overlay is the only variable):

Build	Throughput	Non-UTF-8 reported as UTF-8
bundled (wheel default)	3.10 MB/s	0.0%
system uchardet	1.44 MB/s	16.2%

That 16.2% is the defect v3.0.0 was yanked from PyPI for. Packagers should not have to discover it from a downstream bug report. Full method in #64 (comment).

What changed

Docs only — no build or code changes.

• Qualifies the enabled row so it points at the caveat instead of reading as an unconditional recommendation.
• Adds a warning block with the measured numbers.
• Attributes the cause upstream rather than to the packaging option: nsUTF8Prober never rejects invalid sequences on its own and its confidence never falls far enough for the candidate to be discarded. This is not something @mgorny's patch introduced.
• Points packagers who cannot ship bundled libraries at the overlay as a distro patch in the interim, and notes that the corpus generator and benchmark are in-tree so they can verify independently.

Caveat on the number

make_nonutf8_corpus.py builds documents from random mid-word slices and mixes languages within a document, which is adversarial for language-model detection — 16.2% is a corpus-specific rate, not a field rate. Reporting bytes as UTF-8 that do not decode as UTF-8 is a hard error either way, and the bundled build scores 0.0% on identical input.

Related

A fix for the underlying upstream defect has been prepared (nsUTF8Prober handling eError), which takes a system build to 0.0% mislabel. It is not merged upstream yet, and it exposes a separate Big5 false positive that needs its own fix, so the caveat stands for now. Once upstream carries both, the meson probe can move from has_function() to the version: constraint the existing # TODO anticipates and this block can be deleted.

---

Generated by Claude Code