Proposal: Add ECE to evaluate

[itsfernn]

Expected Calibration Error (ECE) is a widely-used metric for evaluating how well a classifier's predicted confidence scores align with actual outcomes (Guo et al., 2017). It is a standard diagnostic in modern deep learning pipelines, on par with Brier score (already in evaluate) for calibration assessment, but is not yet available in evaluate.

Happy to implement this myself, just wanted to check on scope and design before I open a PR.

Proposed API

At minimum, this metric requires predictions and references as inputs.

>>> import numpy as np
>>> ece = evaluate.load("ece")
>>> references = np.array([0, 0, 1, 1])
>>> predictions = np.array([0.25, 0.25, 0.75, 0.75])
>>> results = ece.compute(references=references, predictions=predictions,
n_bins=2)
>>> print(results)
{'ece': 0.25, 'mce': 0.25, 'adaptive_ece': 0.25}

Inputs

• references: array-like of shape (n_samples,), representing the ground truth labels. Can be numeric (0/1 or -1/1) or strings.
• predictions: numeric array-like of shape (n_samples,), representing the predicted confidence scores (probabilities in [0, 1]) for the positive class.

Optional arguments:

• n_bins: number of bins (default is 10).
• strategy: binning strategy "uniform" (equal-width) or "adaptive" (equal-mass) (default is "uniform").
• pos_label: int or str, default=None. Label of the positive class. pos_label will be inferred as follows:
  • if references in {-1, 1} or {0, 1}, pos_label defaults to 1;
  • else if references contains strings, pos_label must be explicitly specified (an error is raised otherwise);
  • otherwise, pos_label defaults to the greater label, i.e. np.unique(references)[-1].
• return_detailed: if True, returns per-bin reliability diagram data (default is False).

Output Values

This metric returns a dictionary with the following keys:

• ece (float): Expected Calibration Error.
• mce (float): Maximum Calibration Error.
• adaptive_ece (float): Adaptive ECE using equal-mass binning.
• reliability_diagram (list[dict], optional): per-bin data when return_detailed=True.

Implementation approach

Brier score uses scikit-learn. ECE is not in scikit-learn, so either:

A. ECE is implemented from scratch using numpy
B. Or a dependency is added and the netcal implementation is used.

References

• Guo et al., On Calibration of Modern Neural Networks, ICML 2017
• Nixon et al., Measuring Calibration in Deep Learning, CoRR 2020
• netcal package: https://github.com/EFS-OpenSource/calibration-framework

[itsfernn]

Actually in hindsight I think this should be simplified quite a bit to be more inline with the rest of evaluate.

I think a simple netcal wrapper makes the most sense.

MCE and ACE should probably be separate metrics and I also dropped the return_detailed parameter because there isn't a clean way to do this using netcal (might also be out of scope for evaluate).

API is now dead simple:

>>> ece = evaluate.load("ece")
>>> results = ece.compute(references=refs, predictions=preds, n_bins=10)
>>> print(results)
{'ece': 0.25}

Parameters:

• references
• predictions
• n_bins (default 10)
• equal_intervals (bool, default True; selects between equal-width vs equal-mass bins)
• pos_label (optional, same inference logic as brier_score)