From 41f30b4c52a7e61d500e5c41957bd51cf818b1c1 Mon Sep 17 00:00:00 2001 From: Cursor Agent Date: Mon, 17 Aug 2026 09:09:55 +0000 Subject: [PATCH] Clarify that the BLEURT checkpoint is a load, not a compute, argument Co-authored-by: Tony Coder <407243179@qq.com> --- metrics/bleurt/README.md | 4 +++- metrics/bleurt/bleurt.py | 5 ++++- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/metrics/bleurt/README.md b/metrics/bleurt/README.md index b14094e5..5e4cc13b 100644 --- a/metrics/bleurt/README.md +++ b/metrics/bleurt/README.md @@ -41,6 +41,8 @@ This metric takes as input lists of predicted sentences and reference sentences: >>> results = bleurt.compute(predictions=predictions, references=references) ``` +The BLEURT checkpoint is chosen when the metric is loaded, with the `config_name` argument of `load`, and cannot be changed in `compute`. + ### Inputs For the `load` function: @@ -96,7 +98,7 @@ The [original BLEURT paper](https://arxiv.org/pdf/2004.04696.pdf) showed that BL Furthermore, currently BLEURT only supports English-language scoring, given that it leverages models trained on English corpora. It may also reflect, to a certain extent, biases and correlations that were present in the model training data. -Finally, calculating the BLEURT metric involves downloading the BLEURT model that is used to compute the score, which can take a significant amount of time depending on the model chosen. Starting with the default model, `bleurt-tiny`, and testing out larger models if necessary can be a useful approach if memory or internet speed is an issue. +Finally, calculating the BLEURT metric involves downloading the BLEURT model that is used to compute the score, which can take a significant amount of time depending on the model chosen. Starting with a small checkpoint, such as the default `"bleurt-base-128"` or the even smaller `"bleurt-tiny-128"`, and testing out larger models if necessary can be a useful approach if memory or internet speed is an issue. ## Citation diff --git a/metrics/bleurt/bleurt.py b/metrics/bleurt/bleurt.py index 11ad20a7..300bf28e 100644 --- a/metrics/bleurt/bleurt.py +++ b/metrics/bleurt/bleurt.py @@ -45,10 +45,12 @@ _KWARGS_DESCRIPTION = """ BLEURT score. +The BLEURT checkpoint is not an argument of `compute`: it is selected when the metric is loaded, with the +`config_name` argument of `evaluate.load`, and defaults to `bleurt-base-128`. + Args: `predictions` (list of str): prediction/candidate sentences `references` (list of str): reference sentences - `checkpoint` BLEURT checkpoint. Will default to BLEURT-tiny if None. Returns: 'scores': List of scores. @@ -57,6 +59,7 @@ >>> predictions = ["hello there", "general kenobi"] >>> references = ["hello there", "general kenobi"] >>> bleurt = evaluate.load("bleurt") + >>> # bleurt = evaluate.load("bleurt", config_name="BLEURT-20") # you can also choose which checkpoint to use >>> results = bleurt.compute(predictions=predictions, references=references) >>> print([round(v, 2) for v in results["scores"]]) [1.03, 1.04]