Fix HONEST scoring for sentence completions

[tonycoder-hub]

What's the bug

Honest.honest_score_inner picks between two scorers: honest_score_inner_word, which looks up each completion in the HurtLex lexicon, and honest_score_inner_sentence, which splits a completion into words and looks up each word. The dispatch condition was inverted relative to its own comments and to the reference implementation, which uses len(predicted_words[0][0].split(" ")) == 1 to select the word-level scorer:

if " " in predicted_words[0][0]:  # completions are words
    return self.honest_score_inner_word(predicted_words)
else:  # completion is a sentence
    return self.honest_score_inner_sentence(predicted_words)

So whenever completions were sentences (anything containing a space), the whole sentence string was looked up in HurtLex, never matched, and the score was always 0.0:

>>> honest = evaluate.load("honest", "en")
>>> honest.compute(predictions=[["she is a CEO", "she is a businessman"],
...                             ["she is a secretary", "she is a prostitute"]])
{'honest_score': 0.0}   # expected 0.25

Single-word completions took the sentence scorer instead. That path happens to sum to the same aggregate score for one-word completions (each completion contributes at most one hurtful count either way), which is why the existing examples never surfaced the bug.

The fix

Select the word-level scorer only when the completion has no space, matching the comments and the reference implementation. Sentence completions are now split into words and matched against the lexicon.

Tests

• measurements/honest/honest.py: new doctest example for sentence completions (documents the behaviour and is exercised by LocalModuleTest::test_load_measurement_honest). Also renumbered the duplicated "Example 2" heading.
• tests/test_metric_common.py: test_honest_scores_hurtful_words_in_sentence_completions, following the existing test_seqeval_raises_when_incorrect_scheme precedent.

Both tests fail before the one-line change and pass after it:

$ python -m pytest tests/test_metric_common.py -k honest -q
2 passed, 1 skipped, 131 deselected

The full tests/test_metric_common.py failure set is unchanged by this PR (the remaining failures are metrics whose optional dependencies are not installed in this environment).

  

[tonycoder-hub]

Closing as stale — opened on or before 2026-08-17 and still unmerged.