Transliterate HTML entities decoded after the unidecode pass#187
Open
chuenchen309 wants to merge 1 commit into
Open
Transliterate HTML entities decoded after the unidecode pass#187chuenchen309 wants to merge 1 commit into
chuenchen309 wants to merge 1 commit into
Conversation
In the non-unicode path, HTML entity / decimal / hexadecimal decoding runs
after `unidecode()`, and the decoded characters are then only NFKD-normalized.
A decoded character that NFKD-decomposes to ASCII survives (the docstring's
`Ž -> Ž -> z`), but one that needs transliteration (ß, æ, ©, CJK, ...) is
left as raw unicode and stripped by the disallowed-chars pattern, producing an
empty/lossy slug: `slugify("ß") == ""` while `slugify("ß") == "ss"`.
Run `unidecode()` again after normalizing, mirroring the first normalization
block. Idempotent on already-ASCII (entity-free) input.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Slugifying an HTML entity that decodes to a non-ASCII character silently loses it:
Root cause (
slugify/slugify.py): in the non-unicode path, HTML entity / decimal / hexadecimal decoding runs after theunidecode()pass, and the decoded characters are then onlyNFKD-normalized. A decoded char that NFKD-decomposes to ASCII survives — that's why the docstring'sŽ → Ž → zworks — but a decoded char that needs transliteration (ß, æ, ©, CJK, …) is left as raw unicode and stripped by the disallowed-chars pattern, yielding an empty/lossy slug.The docstring documents the intended pipeline as entity → unicode char → transliterated ASCII slug (
:param decimal: converts html decimal to unicode (Ž -> Ž -> z)), and the direct-character path already producesss/ying/c/ae— so this is data loss, not a transliteration taste difference.Fix (1 line): run
unidecode()again after normalizing the decoded text, mirroring the first normalization block. Idempotent on already-ASCII (entity-free) input.Verification (re-runnable from the diff):
''. After:ss/ying/c/ae.test_html_decimal_on_transliterated/_hexadecimal_/_entities_(assert decoded entity slug == the direct-character slug). They fail on the current tree and pass with the fix;test.py→ 85 passed.Ž→z,Ž→z,foo & bar → foo-bar.Orthogonal to #181 (which reworks how invalid numeric references are decoded, in the block above this one) — this touches only the normalization step and applies cleanly regardless.
Disclosure: This PR was authored by an AI coding agent (Claude Code) running on this account: the AI found the bug, ran the repro, wrote the test, and wrote this description. The human account holder reviews every change and is accountable for it. The verification above is real and re-runnable from the diff. If this isn't the kind of contribution you want, say so and I'll close it.