Skip to content

fix(perplexity): use all_special_tokens for GPT-2 tokenizer compatibility - #789

Open
akaashsa wants to merge 1 commit into
huggingface:mainfrom
akaashsa:fix/perplexity-gpt2-tokenizer
Open

fix(perplexity): use all_special_tokens for GPT-2 tokenizer compatibility#789
akaashsa wants to merge 1 commit into
huggingface:mainfrom
akaashsa:fix/perplexity-gpt2-tokenizer

Commits

Commits on Aug 16, 2026