Fix BERTScore OverflowError when the tokenizer has no model_max_length

[tonycoder-hub]

Fixes huggingface#739

Problem

bertscore.compute(..., model_type="microsoft/deberta-xlarge-mnli") crashes with OverflowError: int too big to convert under transformers>=5.

Tokenizers whose config omits model_max_length (DeBERTa being the example from the issue) report transformers' VERY_LARGE_INTEGER sentinel instead, and bert_score.utils.sent_encode forwards tokenizer.model_max_length straight into tokenizer.encode(max_length=...). In transformers>=5 that reaches the Rust tokenizers backend, which cannot represent the value.

Verified against the current main (a7dd338) with transformers==5.15.0 / tokenizers==0.22.2 / bert-score==0.3.13:

model_max_length 1000000000000000019884624838656
max_position_embeddings 512
REPRO OverflowError int too big to convert

Fix

BERTScore._compute() now replaces the sentinel with 512 — the sequence length of the BERT-family models bert_score recommends, and the actual max_position_embeddings of microsoft/deberta-xlarge-mnli — and logs a warning when it does so. A max_length argument is also exposed so a different truncation length can be requested, which is useful because the fallback would otherwise be silent.

The cap has to be applied before any tokenization happens. BERTScorer.__init__ tokenizes idf_sents via compute_idf(), so idf=True would still overflow if the tokenizer were only fixed up after construction. idf_sents are therefore passed to compute_idf() after the tokenizer has been adjusted rather than through the constructor; _idf_dict is None at that point, so this is equivalent to what the constructor did. The cap is re-applied on cached scorers so that max_length also takes effect on subsequent compute() calls.

Models that declare a real model_max_length are untouched.

Tests

Two tests in tests/test_metric_common.py, run with idf=True so that real tokenization happens rather than being mocked away:

• test_bertscore_caps_undefined_model_max_length — reverting bertscore.py makes it fail with the exact error from the issue (OverflowError: int too big to convert at transformers/tokenization_utils_tokenizers.py).
• test_bertscore_max_length_overrides_model_max_length — covers the new argument.

$ python -m pytest tests/test_metric_common.py -k bertscore -q
3 passed, 1 skipped, 131 deselected

make quality (black, isort, flake8) is clean on the changed files. The remaining failures in tests/test_metric_common.py in my environment are identical before and after this change (21 in both cases) and come from running on Python 3.12 with numpy>=2 and without the optional dependencies from additional-tests-requirements.txt; CI pins numpy<2.0.0 and Python 3.9.

I also confirmed the fix against the real microsoft/deberta-xlarge-mnli tokenizer with the model forward mocked out:

real deberta model_max_length before: 1000000000000000019884624838656
after: 512
result: {'precision': [1.0], 'recall': [1.0], 'f1': [1.0]}

Note

#756 previously proposed a fix for this issue but was closed without merging. That version applied the cap only after BERTScorer construction, so the idf=True path still overflowed.

  

[tonycoder-hub]

Closing as stale — opened on or before 2026-08-17 and still unmerged.