Exact Interval-Based Segmentation Metrics by tomxi · Pull Request #436 · mir-evaluation/mir_eval

tomxi · 2026-07-31T17:32:16Z

Implements issue #432.

Adds frame_size=None as an option for the following segmentation metrics: segment.pairwise, segment.nce, segment.vmeasure, and hierarchy.lmeasure: this option will calculate these metrics without using framing/sampling, thus provide more accurate results and are generally much faster.

Also capped the matplotlib version in github workflow to be less than 3.11, so that the display tests are passing: matplotlib 3.11 seems to have made very subtle changes to plot axis text layouts, and thus the display tests are not passing against the reference plots if using mpl 3.11.

…ure); capped matplotlib ci version to keep display tests passing (mir-evaluation#432)

Copilot

Pull request overview

Adds “frameless” (exact interval-duration) evaluation paths for key segmentation metrics by allowing frame_size=None, aiming to improve both accuracy (no sampling approximation) and runtime (scale with number of segments rather than track duration/frames). Also adjusts CI dependencies to keep matplotlib-based image-baseline tests stable.

Changes:

Add frame_size=None (“frameless”) execution paths for segment.pairwise, segment.nce/segment.vmeasure, and hierarchy.lmeasure.
Add new unit tests covering frameless behavior and duration-weighted hierarchy ranking comparisons.
Cap matplotlib-base to <3.11 in the GitHub Actions conda environment to keep display/baseline tests passing.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`tests/test_segment.py`	Adds targeted tests for frameless `pairwise` and `nce`.
`tests/test_hierarchy.py`	Extends `lmeasure` tests to include `frame_size=None` and adds weighted-ranking unit tests.
`mir_eval/segment.py`	Implements frameless logic for `pairwise`/`nce` (+ doc updates) and adds weighted contingency support.
`mir_eval/hierarchy.py`	Implements frameless `lmeasure` via atomic intervals and adds duration-weighted ranking support.
`.github/environment.yml`	Pins matplotlib to `<3.11` to stabilize mpl image comparison tests in CI.

Suppressed comments (2)

mir_eval/segment.py:1089

nce() unconditionally emits a FutureWarning on every call, even when frame_size is explicitly set (including frame_size=None). This can create a lot of warning noise (especially via segment.evaluate) and contradicts the intent of warning only for default behavior changes. Gate the warning so it only triggers when using the default frame_size behavior.

    # raise FutureWarning for change of default frame_size to None
    warnings.warn(
        "Default `frame_size` will change from 0.1 to None in a future version. "
        "Set `frame_size` explicitly for consistent results.",
        FutureWarning,
        stacklevel=2,
    )

mir_eval/hierarchy.py:706

lmeasure() unconditionally emits a FutureWarning on every call, even when frame_size is explicitly set by the caller. This will spam users and makes it hard to opt out of the warning by setting frame_size explicitly. Gate the warning so it only triggers when relying on the default frame_size behavior.

    # raise FutureWarning for change of default frame_size to None
    warnings.warn(
        "Default `frame_size` will change from 0.1 to None in a future version. "
        "Set `frame_size` explicitly for consistent results.",
        FutureWarning,
        stacklevel=2,
    )

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tomxi · 2026-08-01T02:34:46Z

+    # Optionally weight by segment duration
+    if seg_durations is None:
+        seg_durations = np.ones(len(reference_indices))
    # Using coo_matrix is faster than histogram2d
    return scipy.sparse.coo_matrix(
-        (np.ones(ref_class_idx.shape[0]), (ref_class_idx, est_class_idx)),
+        (seg_durations, (ref_class_idx, est_class_idx)),
        shape=(n_ref_classes, n_est_classes),
-        dtype=np.int64,
+        dtype=np.float64,
    ).toarray()


will preserve int dtype when seg_durations is not provided

tomxi · 2026-08-01T01:45:12Z

+    # raise FutureWarning for change of default frame_size to None
+    warnings.warn(
+        "Default `frame_size` will change from 0.1 to None in a future version. "
+        "Set `frame_size` explicitly for consistent results.",
+        FutureWarning,
+        stacklevel=2,
+    )


Will change all similar FutureWarnings

tomxi · 2026-08-01T02:34:13Z

+    frame_size : float > 0 or None
        length (in seconds) of frames.  The frame size cannot be longer than
-        the window.
+        the window. If None, use exact segment duration instead of framing.


Will remove mention of window in lmeasure doc string

tomxi · 2026-08-01T02:19:33Z

Suggestions from Copilot's code review all seem reasonable, and I'll incorporate them in the next commit.

However, the CI checks are failing with various issues that I hope to get some guidance on how best to resolve:

for the part of the test matrix with python=3.13, some unit tests in test_display.py are failing due to minor layout differences from reference baseline.png's. This is because these environments don't follow .github/environment.yml and uses matplotlib=3.11. Updating all the baseline.png's probably belong to another PR, but what's a sensible patch right now to keep the display tests passing?
for the other test environments (python<3.13), all the unit tests are passing, but CI check is failing at the upload coverage step with this message:
error - 2026-07-31 20:31:36,292 -- Upload queued for processing failed: {"message":"Token required - not valid tokenless upload"}
any advice on how to resolve?
CI Docs is failing at the link checking step: code.soundsoftware.ac.uk doesn't work anymore and music-ir.org has SSL issues. This also seems out of scope of this PR; any suggestions on how to proceed?

Conditional FutureWarning for frame_size default change; backward compatible contingency matrix dtype handling; No mention of window in lmeasure doc strings.

craffel

Maybe @bmcfee can check for correctness?

craffel · 2026-08-03T19:15:27Z

+    # raise FutureWarning for change of default frame_size to None
+    if frame_size == 0.1:
+        warnings.warn(
+            "Default `frame_size` will change from 0.1 to None in a future version. "


Historically we only deprecate/make changes like this if the broad community consensus supports it, e.g. MIREX changing their approach or virtually all papers migrating to it. I'm not sure we're at that point for these metrics yet?

craffel · 2026-08-03T19:16:11Z

+    # raise FutureWarning for change of default frame_size to None
+    if frame_size == 0.1:
+        warnings.warn(
+            "Default `frame_size` will change from 0.1 to None in a future version. "


craffel · 2026-08-03T19:16:34Z

+    # raise FutureWarning for change of default frame_size to None
+    if frame_size == 0.1:
+        warnings.warn(
+            "Default `frame_size` will change from 0.1 to None in a future version. "


craffel · 2026-08-03T19:19:19Z

  - numpy >=1.20.3
  - scipy >=1.4.0
-  - matplotlib-base>=3.6.0
+  - matplotlib-base>=3.6.0,<3.11


Should we change the matplotlib entry under test dependencies in options.extras_require in setup.cfg too?

tomxi added 2 commits July 31, 2026 10:16

Frameless versions of flat segmentation metrics (pairwise, nce, vmeas…

89013b7

…ure); capped matplotlib ci version to keep display tests passing (mir-evaluation#432)

implements frameless L-measure (mir-evaluation#432)

4a3ef10

bmcfee requested a review from Copilot July 31, 2026 20:36

Copilot started reviewing on behalf of bmcfee July 31, 2026 20:37 View session

Copilot AI reviewed Jul 31, 2026

View reviewed changes

Incorporate code review suggestions (mir-evaluation#436)

f51eeaf

Conditional FutureWarning for frame_size default change; backward compatible contingency matrix dtype handling; No mention of window in lmeasure doc strings.

craffel reviewed Aug 3, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Exact Interval-Based Segmentation Metrics - #436

Exact Interval-Based Segmentation Metrics#436
tomxi wants to merge 3 commits into
mir-evaluation:mainfrom
tomxi:fast-segment

tomxi commented Jul 31, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

tomxi Aug 1, 2026

Uh oh!

tomxi Aug 1, 2026

Uh oh!

tomxi Aug 1, 2026

Uh oh!

tomxi commented Aug 1, 2026

Uh oh!

craffel left a comment

Uh oh!

craffel Aug 3, 2026

Uh oh!

craffel Aug 3, 2026

Uh oh!

craffel Aug 3, 2026

Uh oh!

craffel Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

tomxi commented Jul 31, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

tomxi Aug 1, 2026

Choose a reason for hiding this comment

Uh oh!

tomxi Aug 1, 2026

Choose a reason for hiding this comment

Uh oh!

tomxi Aug 1, 2026

Choose a reason for hiding this comment

Uh oh!

tomxi commented Aug 1, 2026

Uh oh!

craffel left a comment

Choose a reason for hiding this comment

Uh oh!

craffel Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

craffel Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

craffel Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

craffel Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants