Don't let generation_kwargs leak into the class-level PIPELINE_KWARGS

[tonycoder-hub]

Problem

Text2TextGenerationEvaluator.compute() merged the caller's generation_kwargs into PIPELINE_KWARGS with an in-place update:

if generation_kwargs is not None:
    self.PIPELINE_KWARGS.update(generation_kwargs)

PIPELINE_KWARGS is a class attribute, so self.PIPELINE_KWARGS.update(...) mutates the dict that is shared by the class and every one of its instances. A generation_kwargs value passed to a single compute() call is therefore forwarded to the pipeline on every later call in the same process, including calls made through a completely separate evaluator object.

Reproduced on main (a7dd338) with a dummy pipeline that records the kwargs it is called with:

first call kwargs: {'truncation': True, 'max_length': 5}
class PIPELINE_KWARGS after: {'truncation': True, 'max_length': 5}
second (fresh evaluator, no generation_kwargs) call kwargs: {'truncation': True, 'max_length': 5}

The second compute() call was made on a brand new evaluator("text2text-generation") and passed no generation_kwargs, yet the pipeline still received max_length=5. The same applies to SummarizationEvaluator and TranslationEvaluator, which each define their own PIPELINE_KWARGS.

Fix

Build a per-call dict from the class defaults instead of mutating them, so generation_kwargs are scoped to the compute() call that supplied them:

self.PIPELINE_KWARGS = {**type(self).PIPELINE_KWARGS, **(generation_kwargs or {})}

Using type(self) keeps the subclass defaults (SummarizationEvaluator, TranslationEvaluator) intact, and assigning to self shadows rather than mutates the class attribute.

Tests

tests/test_evaluator.py:

• DummyText2TextGenerationPipeline now records the kwargs it was called with, so tests can assert what actually reached the pipeline.
• test_generation_kwargs checks that generation_kwargs are forwarded to the pipeline alongside the class defaults.
• test_generation_kwargs_are_not_persisted checks that the class attribute is untouched and that a later compute() without generation_kwargs no longer sees them. This test fails on main and passes with the fix.

$ python -m pytest tests/test_evaluator.py -k Text2Text -v
# on main: 1 failed, 6 passed, 1 skipped
# with fix: 7 passed, 1 skipped

$ python -m pytest tests/test_evaluator.py
43 passed, 14 skipped

black, isort and flake8 are clean on both changed files.

  

[tonycoder-hub]

Closing as stale — opened on or before 2026-08-17 and still unmerged.