fix: mask_id extraction assumes single-token encoding

[andrewwhitecdw]

Problem

fix: mask_id extraction assumes single-token encoding

Fix

Replace:

    mask_id = int(tokenizer.encode("|<MASK>|", add_special_tokens=False)[0])

with:

    mask_token = "|<MASK>|"
    mask_ids = tokenizer.encode(mask_token, add_special_tokens=False)
    if len(mask_ids) != 1:
        raise ValueError(f"Expected {mask_token!r} to map to a single token, got {mask_ids}")
    mask_id = int(mask_ids[0])

Files changed

• fast_ddrive/run_chatbot.py