Skip to content

Transliterate HTML entities decoded after the unidecode pass#187

Open
chuenchen309 wants to merge 1 commit into
un33k:masterfrom
chuenchen309:fix/transliterate-decoded-html-entities
Open

Transliterate HTML entities decoded after the unidecode pass#187
chuenchen309 wants to merge 1 commit into
un33k:masterfrom
chuenchen309:fix/transliterate-decoded-html-entities

Conversation

@chuenchen309

Copy link
Copy Markdown

Slugifying an HTML entity that decodes to a non-ASCII character silently loses it:

slugify("ß")    # ''    — expected 'ss'   (slugify("ß") == 'ss')
slugify("影")  # ''    — expected 'ying' (slugify("影") == 'ying')
slugify("©")    # ''    — expected 'c'    (slugify("©")  == 'c')
slugify("æ")    # ''    — expected 'ae'   (slugify("æ")  == 'ae')

Root cause (slugify/slugify.py): in the non-unicode path, HTML entity / decimal / hexadecimal decoding runs after the unidecode() pass, and the decoded characters are then only NFKD-normalized. A decoded char that NFKD-decomposes to ASCII survives — that's why the docstring's Ž → Ž → z works — but a decoded char that needs transliteration (ß, æ, ©, CJK, …) is left as raw unicode and stripped by the disallowed-chars pattern, yielding an empty/lossy slug.

The docstring documents the intended pipeline as entity → unicode char → transliterated ASCII slug (:param decimal: converts html decimal to unicode (Ž -> Ž -> z)), and the direct-character path already produces ss/ying/c/ae — so this is data loss, not a transliteration taste difference.

Fix (1 line): run unidecode() again after normalizing the decoded text, mirroring the first normalization block. Idempotent on already-ASCII (entity-free) input.

     else:
         text = unicodedata.normalize('NFKD', text)
+        text = unidecode.unidecode(text)

Verification (re-runnable from the diff):

  • Before: the four cases above return ''. After: ss/ying/c/ae.
  • Added test_html_decimal_on_transliterated / _hexadecimal_ / _entities_ (assert decoded entity slug == the direct-character slug). They fail on the current tree and pass with the fix; test.py → 85 passed.
  • The docstring examples are unchanged: Ž→z, Ž→z, foo & bar → foo-bar.

Orthogonal to #181 (which reworks how invalid numeric references are decoded, in the block above this one) — this touches only the normalization step and applies cleanly regardless.


Disclosure: This PR was authored by an AI coding agent (Claude Code) running on this account: the AI found the bug, ran the repro, wrote the test, and wrote this description. The human account holder reviews every change and is accountable for it. The verification above is real and re-runnable from the diff. If this isn't the kind of contribution you want, say so and I'll close it.

In the non-unicode path, HTML entity / decimal / hexadecimal decoding runs
after `unidecode()`, and the decoded characters are then only NFKD-normalized.
A decoded character that NFKD-decomposes to ASCII survives (the docstring's
`Ž -> Ž -> z`), but one that needs transliteration (ß, æ, ©, CJK, ...) is
left as raw unicode and stripped by the disallowed-chars pattern, producing an
empty/lossy slug: `slugify("ß") == ""` while `slugify("ß") == "ss"`.

Run `unidecode()` again after normalizing, mirroring the first normalization
block. Idempotent on already-ASCII (entity-free) input.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant