Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion metrics/bleurt/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ This metric takes as input lists of predicted sentences and reference sentences:
>>> results = bleurt.compute(predictions=predictions, references=references)
```

The BLEURT checkpoint is chosen when the metric is loaded, with the `config_name` argument of `load`, and cannot be changed in `compute`.

### Inputs

For the `load` function:
Expand Down Expand Up @@ -96,7 +98,7 @@ The [original BLEURT paper](https://arxiv.org/pdf/2004.04696.pdf) showed that BL

Furthermore, currently BLEURT only supports English-language scoring, given that it leverages models trained on English corpora. It may also reflect, to a certain extent, biases and correlations that were present in the model training data.

Finally, calculating the BLEURT metric involves downloading the BLEURT model that is used to compute the score, which can take a significant amount of time depending on the model chosen. Starting with the default model, `bleurt-tiny`, and testing out larger models if necessary can be a useful approach if memory or internet speed is an issue.
Finally, calculating the BLEURT metric involves downloading the BLEURT model that is used to compute the score, which can take a significant amount of time depending on the model chosen. Starting with a small checkpoint, such as the default `"bleurt-base-128"` or the even smaller `"bleurt-tiny-128"`, and testing out larger models if necessary can be a useful approach if memory or internet speed is an issue.


## Citation
Expand Down
5 changes: 4 additions & 1 deletion metrics/bleurt/bleurt.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,12 @@
_KWARGS_DESCRIPTION = """
BLEURT score.

The BLEURT checkpoint is not an argument of `compute`: it is selected when the metric is loaded, with the
`config_name` argument of `evaluate.load`, and defaults to `bleurt-base-128`.

Args:
`predictions` (list of str): prediction/candidate sentences
`references` (list of str): reference sentences
`checkpoint` BLEURT checkpoint. Will default to BLEURT-tiny if None.

Returns:
'scores': List of scores.
Expand All @@ -57,6 +59,7 @@
>>> predictions = ["hello there", "general kenobi"]
>>> references = ["hello there", "general kenobi"]
>>> bleurt = evaluate.load("bleurt")
>>> # bleurt = evaluate.load("bleurt", config_name="BLEURT-20") # you can also choose which checkpoint to use
>>> results = bleurt.compute(predictions=predictions, references=references)
>>> print([round(v, 2) for v in results["scores"]])
[1.03, 1.04]
Expand Down