Make text_duplicates list the duplicates it counts

[tonycoder-hub]

Problem

text_duplicates can return two fields that contradict each other in a single call:

>>> from evaluate import load
>>> load("text_duplicates").compute(data=["hello sun", "hello sun ", "hello moon"], list_duplicates=True)
{'duplicate_fraction': 0.33333333333333337, 'duplicates_dict': {}}

duplicate_fraction is derived from get_hash, which strips surrounding whitespace, so "hello sun" and "hello sun " are counted as duplicates. duplicates_dict, however, was built with Counter(data) over the raw strings, so those same duplicates never show up in the listing. The caller is told that a third of the data is duplicated and simultaneously that there are no duplicates.

Fix

Count the duplicates over the same stripped strings that the fraction is based on:

c = Counter(d.strip() for d in data)

Inputs without stray whitespace are unaffected, so the documented examples and the module doctests keep their current output.

Testing

Added test_text_duplicates_lists_the_duplicates_it_counts in tests/test_metric_common.py, next to the existing per-module test for seqeval. It fails on main (assert {} == {'hello sun': 2}) and passes with the fix.

python -m pytest tests/test_metric_common.py -k text_duplicates
# 2 passed, 1 skipped (test_load_measurement_text_duplicates doctest + new test)

Wider offline run of tests/test_metric_common.py tests/test_metric.py tests/test_save.py tests/test_file_utils.py: 26 failed / 69 passed with the change versus 26 failed / 68 passed without it. The 26 failures are pre-existing ImportErrors for optional metric dependencies that are not installed in this environment (nltk, sacrebleu, jiwer, torch, ...) and are identical before and after.

  

[tonycoder-hub]

Closing as stale — opened on or before 2026-08-17 and still unmerged.