From 78faeb1795b337645c5f3a24e1ee20d913083a6b Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 4 Aug 2026 11:31:21 +0000 Subject: [PATCH] docs: warn packagers that a system uchardet detects non-UTF-8 less accurately The system-uchardet section presented `enabled` as the recommended choice for distro packaging without noting that it produces a measurably worse build. The bundled copy compiles an encoding-only replacement for uchardet's multi-byte group prober; a system build links upstream's, so it loses both the throughput fix and the rejection of non-UTF-8 sequences that upstream reports as UTF-8. Over the in-tree non-UTF-8 corpus (1650 documents, 7 encodings, median of 3 runs) a system build reports 16.2% of documents as UTF-8 against 0.0% for the bundled build, at roughly half the throughput. That is the defect v3.0.0 was yanked for, so packagers should not discover it from a bug report. State the numbers, attribute the cause upstream rather than to the packaging option, and point packagers who cannot ship a bundled library at the overlay as a distro patch in the meantime. --- README.md | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index e76f2c6..b276a14 100644 --- a/README.md +++ b/README.md @@ -56,7 +56,7 @@ Meson [feature option](https://mesonbuild.com/Build-options.html#features): | `-Dsystem-uchardet=` | Behaviour | | --- | --- | | `disabled` (default) | Always build the bundled copy (used by the wheels). | -| `enabled` | Require the system library; the build **fails** if it is missing or too old. Recommended for distro packaging. | +| `enabled` | Require the system library; the build **fails** if it is missing or too old. Intended for distro packaging — read the detection-quality caveat below first. | | `auto` | Use the system library if it is new enough, otherwise fall back to the bundled copy. | Pass the option through your build front-end. With `pip` / `build` the @@ -72,6 +72,35 @@ pip install . --config-settings=setup-args=-Dsystem-uchardet=enabled > release provides it yet — so `enabled`/`auto` probe for it and reject an > older library. +> [!WARNING] +> **A system build currently detects non-UTF-8 input less accurately than the +> bundled build.** The bundled copy compiles a cChardet-specific replacement +> for uchardet's multi-byte group prober, which both restores detection +> throughput and rejects non-UTF-8 byte sequences that upstream reports as +> UTF-8. A system build links upstream's prober, so neither applies. +> +> Measured over 1650 non-UTF-8 documents spanning 7 encodings +> (`benchmarks/make_nonutf8_corpus.py`, median of 3 runs): +> +> | Build | Throughput | Non-UTF-8 reported as UTF-8 | +> | --- | ---: | ---: | +> | bundled (wheel default) | 3.10 MB/s | 0.0% | +> | system `uchardet` | 1.44 MB/s | **16.2%** | +> +> Reporting non-UTF-8 bytes as UTF-8 is the defect that caused v3.0.0 to be +> yanked from PyPI, so this is not a cosmetic difference. It is an upstream +> issue rather than a packaging mistake: `nsUTF8Prober` does not reject +> invalid sequences on its own, and its confidence never falls low enough for +> the candidate to be discarded. Patches have been sent upstream; once they +> land, the build can require a `uchardet` version that includes them and this +> caveat goes away. +> +> If you are packaging cChardet for a distribution that forbids bundled +> libraries, consider carrying the overlay +> (`src/cchardet/uchardet-overlay/nsMBCSGroupProber.cpp`) as a patch against +> your system `uchardet` until then. The corpus generator and benchmark above +> are in-tree, so you can verify the result yourself. + ## Supported Languages/Encodings - International (Unicode)