Task Text Summarization does not work

[DominikHil]

The example code shown in https://huggingface.co/docs/evaluate/main/en/package_reference/evaluator_classes#evaluate.SummarizationEvaluator fails with:

KeyError: "Unknown task summarization, available tasks are ['any-to-any', 'audio-classification', 'automatic-speech-recognition', 'depth-estimation', 'document-question-answering', 'feature-extraction', 'fill-mask', 'image-classification', 'image-feature-extraction', 'image-segmentation', 'image-text-to-text', 'keypoint-matching', 'mask-generation', 'ner', 'object-detection', 'sentiment-analysis', 'table-question-answering', 'text-classification', 'text-generation', 'text-to-audio', 'text-to-speech', 'token-classification', 'video-classification', 'zero-shot-audio-classification', 'zero-shot-classification', 'zero-shot-image-classification', 'zero-shot-object-detection']"

Enviroment:

# Name                    Version                   Build  Channel
aiohappyeyeballs          2.6.1                    pypi_0    pypi
aiohttp                   3.13.4                   pypi_0    pypi
aiosignal                 1.4.0                    pypi_0    pypi
annotated-doc             0.0.4                    pypi_0    pypi
anyio                     4.13.0                   pypi_0    pypi
attrs                     26.1.0                   pypi_0    pypi
bzip2                     1.0.8                h2bbff1b_6  
ca-certificates           2025.12.2            haa95532_0  
certifi                   2026.2.25                pypi_0    pypi
charset-normalizer        3.4.6                    pypi_0    pypi
click                     8.3.1                    pypi_0    pypi
colorama                  0.4.6                    pypi_0    pypi
cuda                      13.0.2                        0    nvidia/label/cuda-13.0.2
cuda-cccl_win-64          13.0.85              hc667259_1    nvidia/label/cuda-13.0.2
cuda-command-line-tools   13.0.2                        0    nvidia/label/cuda-13.0.2
cuda-compiler             13.0.2                        0    nvidia/label/cuda-13.0.2
cuda-crt-dev_win-64       13.0.88                       0    nvidia/label/cuda-13.0.2
cuda-crt-tools            13.0.88                       0    nvidia/label/cuda-13.0.2
cuda-ctadvisor            13.0.85              hde976d8_0    nvidia/label/cuda-13.0.2
cuda-cudart               13.0.96              hde976d8_1    nvidia/label/cuda-13.0.2
cuda-cudart-dev           13.0.96              hde976d8_1    nvidia/label/cuda-13.0.2
cuda-cudart-dev_win-64    13.0.96              hde976d8_1    nvidia/label/cuda-13.0.2
cuda-cudart-static        13.0.96              hde976d8_1    nvidia/label/cuda-13.0.2
cuda-cudart-static_win-64 13.0.96              hde976d8_1    nvidia/label/cuda-13.0.2
cuda-cudart_win-64        13.0.96              hde976d8_1    nvidia/label/cuda-13.0.2
cuda-cuobjdump            13.0.85              h17533db_0    nvidia/label/cuda-13.0.2
cuda-cupti                13.0.85              h17533db_0    nvidia/label/cuda-13.0.2
cuda-cupti-dev            13.0.85              h17533db_0    nvidia/label/cuda-13.0.2
cuda-cuxxfilt             13.0.85              h17533db_0    nvidia/label/cuda-13.0.2
cuda-libraries            13.0.2                        0    nvidia/label/cuda-13.0.2
cuda-libraries-dev        13.0.2                        0    nvidia/label/cuda-13.0.2
cuda-nvcc                 13.0.88              h9a4f5bf_0    nvidia/label/cuda-13.0.2
cuda-nvcc-dev_win-64      13.0.88                       0    nvidia/label/cuda-13.0.2
cuda-nvcc-impl            13.0.88              h87bdf4e_0    nvidia/label/cuda-13.0.2
cuda-nvcc-tools           13.0.88              h82bb817_0    nvidia/label/cuda-13.0.2
cuda-nvcc_win-64          13.0.88              h8adc4ec_0    nvidia/label/cuda-13.0.2
cuda-nvdisasm             13.0.85              h17533db_0    nvidia/label/cuda-13.0.2
cuda-nvml-dev             13.0.87              h7827d72_0    nvidia/label/cuda-13.0.2
cuda-nvprune              13.0.85              h17533db_0    nvidia/label/cuda-13.0.2
cuda-nvrtc                13.0.88              h4c03020_0    nvidia/label/cuda-13.0.2
cuda-nvrtc-dev            13.0.88              hde976d8_0    nvidia/label/cuda-13.0.2
cuda-nvvm-dev_win-64      13.0.88                       0    nvidia/label/cuda-13.0.2
cuda-nvvm-impl            13.0.88              h82bb817_0    nvidia/label/cuda-13.0.2
cuda-nvvm-tools           13.0.88              h82bb817_0    nvidia/label/cuda-13.0.2
cuda-opencl               13.0.85              h17533db_0    nvidia/label/cuda-13.0.2
cuda-opencl-dev           13.0.85              hde976d8_0    nvidia/label/cuda-13.0.2
cuda-profiler-api         13.0.85              h5221881_0    nvidia/label/cuda-13.0.2
cuda-runtime              13.0.2                        0    nvidia/label/cuda-13.0.2
cuda-sanitizer-api        13.0.85              h17533db_0    nvidia/label/cuda-13.0.2
cuda-toolkit              13.0.2                        0    nvidia/label/cuda-13.0.2
cuda-tools                13.0.2                        0    nvidia/label/cuda-13.0.2
cuda-version              13.0                          3    nvidia/label/cuda-13.0.2
cuda-visual-tools         13.0.2                        0    nvidia/label/cuda-13.0.2
datasets                  4.8.4                    pypi_0    pypi
dill                      0.4.1                    pypi_0    pypi
evaluate                  0.4.6                    pypi_0    pypi
expat                     2.7.5                hd7fb8db_0
filelock                  3.25.2                   pypi_0    pypi
fontconfig                2.15.0               hd211d86_0
freetype                  2.14.1               hfbffc0b_0
frozenlist                1.8.0                    pypi_0    pypi
fsspec                    2026.2.0                 pypi_0    pypi
h11                       0.16.0                   pypi_0    pypi
hf-xet                    1.4.2                    pypi_0    pypi
httpcore                  1.0.9                    pypi_0    pypi
httpx                     0.28.1                   pypi_0    pypi
huggingface-hub           1.8.0                    pypi_0    pypi
idna                      3.11                     pypi_0    pypi
jinja2                    3.1.6                    pypi_0    pypi
khronos-opencl-icd-loader 2025.07.22           h79b28c9_0
libcublas                 13.1.0.3             hde976d8_0    nvidia/label/cuda-13.0.2
libcublas-dev             13.1.0.3             hde976d8_0    nvidia/label/cuda-13.0.2
libcufft                  12.0.0.61            h8be46e5_0    nvidia/label/cuda-13.0.2
libcufft-dev              12.0.0.61            hde976d8_0    nvidia/label/cuda-13.0.2
libcurand                 10.4.0.35            h86a452a_1    nvidia/label/cuda-13.0.2
libcurand-dev             10.4.0.35            hde976d8_1    nvidia/label/cuda-13.0.2
libcusolver               12.0.4.66            hde976d8_0    nvidia/label/cuda-13.0.2
libcusolver-dev           12.0.4.66            hde976d8_0    nvidia/label/cuda-13.0.2
libcusparse               12.6.3.3             hb96d541_0    nvidia/label/cuda-13.0.2
libcusparse-dev           12.6.3.3             hde976d8_0    nvidia/label/cuda-13.0.2
libexpat                  2.7.5                hd7fb8db_0
libffi                    3.4.4                hd77b12b_1
libglib                   2.86.3               h9bccc14_0
libiconv                  1.18                 hc89ec93_0
libnpp                    13.0.1.2             h831e19e_0    nvidia/label/cuda-13.0.2
libnpp-dev                13.0.1.2             h831e19e_0    nvidia/label/cuda-13.0.2
libnvfatbin               13.0.85              h17533db_0    nvidia/label/cuda-13.0.2
libnvfatbin-dev           13.0.85              h17533db_0    nvidia/label/cuda-13.0.2
libnvjitlink              13.0.88              h4c03020_0    nvidia/label/cuda-13.0.2
libnvjitlink-dev          13.0.88              h4c03020_0    nvidia/label/cuda-13.0.2
libnvjpeg                 13.0.1.86            hae75015_0    nvidia/label/cuda-13.0.2
libnvjpeg-dev             13.0.1.86            hc667259_0    nvidia/label/cuda-13.0.2
libnvptxcompiler-dev_win-64 13.0.88                       0    nvidia/label/cuda-13.0.2
libpng                    1.6.56               h2854ad3_0
libxml2                   2.13.9               h6201b9f_0
libzlib                   1.3.1                h02ab6af_0
markdown-it-py            4.0.0                    pypi_0    pypi
markupsafe                3.0.3                    pypi_0    pypi
mdurl                     0.1.2                    pypi_0    pypi
mpmath                    1.3.0                    pypi_0    pypi
multidict                 6.7.1                    pypi_0    pypi
multiprocess              0.70.19                  pypi_0    pypi
networkx                  3.6.1                    pypi_0    pypi
nsight-compute            2025.3.1.4           hc3e9d23_0    nvidia/label/cuda-13.0.2
numpy                     2.4.3                    pypi_0    pypi
opencl-headers            2025.07.22           h885b0b7_0
openssl                   3.5.5                hbb43b14_0
packaging                 25.0            py311haa95532_1
pandas                    3.0.1                    pypi_0    pypi
pcre2                     10.46                h5740b90_0
pillow                    12.1.1                   pypi_0    pypi
pip                       26.0.1             pyhc872135_0
propcache                 0.4.1                    pypi_0    pypi
pyarrow                   23.0.1                   pypi_0    pypi
pygments                  2.19.2                   pypi_0    pypi
python                    3.11.15              h1044e36_0
python-dateutil           2.9.0.post0              pypi_0    pypi
pyyaml                    6.0.3                    pypi_0    pypi
regex                     2026.3.32                pypi_0    pypi
requests                  2.33.0                   pypi_0    pypi
rich                      14.3.3                   pypi_0    pypi
safetensors               0.7.0                    pypi_0    pypi
scipy                     1.17.1                   pypi_0    pypi
setuptools                80.10.2         py311haa95532_0
shellingham               1.5.4                    pypi_0    pypi
six                       1.17.0                   pypi_0    pypi
sqlite                    3.51.2               hee5a0db_0
sympy                     1.14.0                   pypi_0    pypi
tk                        8.6.15               hf199647_0
tokenizers                0.22.2                   pypi_0    pypi
torch                     2.11.0+cu130             pypi_0    pypi
torchvision               0.26.0+cu130             pypi_0    pypi
tqdm                      4.67.3                   pypi_0    pypi
transformers              5.4.0                    pypi_0    pypi
typer                     0.24.1                   pypi_0    pypi
typing-extensions         4.15.0                   pypi_0    pypi
tzdata                    2025.3                   pypi_0    pypi
ucrt                      10.0.22621.0         haa95532_0
urllib3                   2.6.3                    pypi_0    pypi
vc                        14.3                h2df5915_10
vc14_runtime              14.44.35208         h4927774_10
vs2015_runtime            14.44.35208         ha6b5a95_10
vs2017_win-64             19.16.27032.1        hb4161e2_3
vswhere                   3.1.7                haa95532_0
wheel                     0.46.3          py311haa95532_0
xxhash                    3.6.0                    pypi_0    pypi
xz                        5.8.2                h53af0af_0
yarl                      1.23.0                   pypi_0    pypi
zlib                      1.3.1                h02ab6af_0

[braydenkrus]

I tested this in a Colab notebook, and am also experiencing this. Additionally, I've noticed that alongside summarization, there are also issues with text2text-generation and translation, all of which come from the same import in their implementation:

evaluate/src/evaluate/evaluator/__init__.py

Line 32 in a7dd338

from .text2text_generation import SummarizationEvaluator, Text2TextGenerationEvaluator, TranslationEvaluator

.

I think this issue may actually trace back to SUPPORTED_TASKS in https://github.com/huggingface/transformers/blob/eed95d8c445b8679ba342cffa947a3ed2b8d7fbc/src/transformers/pipelines/__init__.py, at least for summarization, and text2text-generation.

As for translation, the error thrown is KeyError: "Invalid translation task translation, use 'translation_XX_to_YY' format", which I have not looked into yet.

In short, this seems to be an external issue in transformers, where some tasks are not implemented under SUPPORTED_TASKS.

EDIT: huggingface/transformers@fc53224
Basically, the underlying pipelines were removed as they are outdated and can be misleading for beginners. So these should actually be removed. I will work on a PR that achieves this.