Skip to content

Avoid retaining evaluate out arrays#560

Merged
FrancescAlted merged 3 commits into
pydata:masterfrom
gaoflow:fix-558-out-cache-weakref
Jun 24, 2026
Merged

Avoid retaining evaluate out arrays#560
FrancescAlted merged 3 commits into
pydata:masterfrom
gaoflow:fix-558-out-cache-weakref

Conversation

@gaoflow

@gaoflow gaoflow commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Fixes #558.

validate() cached the out= array directly in _numexpr_last so re_evaluate() could reuse it. That kept the most recent output array alive even after the caller deleted their last reference.

This stores a weak reference for cached out= instead. re_evaluate() resolves it when called, so live output arrays are still reused, but released arrays are no longer pinned by the cache.

Tests:

  • .venv/bin/python -m pytest numexpr/tests/test_numexpr.py -q
  • .venv/bin/python -m pytest --pyargs numexpr -q
  • .venv/bin/python -m compileall -q numexpr

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses a memory-retention issue in NumExpr’s re_evaluate() caching by avoiding strong references to arrays provided via out= in the thread-local “last expression” cache, preventing large output buffers from being unintentionally pinned.

Changes:

  • Cache out= as a weakref in validate() so the most recent output array isn’t kept alive solely by _numexpr_last.
  • Resolve the weak reference in re_evaluate() to reuse live out= arrays when still available.
  • Add regression tests covering both “doesn’t keep out alive” and “reuses live out”.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
RELEASE_NOTES.rst Documents the behavioral change for out= retention in the re_evaluate cache.
numexpr/tests/test_numexpr.py Adds tests asserting out= isn’t pinned and that re_evaluate() still reuses a live out.
numexpr/necompiler.py Implements weakref-based caching for out= and resolves it during re_evaluate().

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread numexpr/necompiler.py Outdated
FrancescAlted and others added 2 commits June 24, 2026 10:47
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@FrancescAlted

Copy link
Copy Markdown
Contributor

LGTM. I had to disable providing free-threaded wheels for Python 3.13t, because NumPy already does that, so let's play the same game. Thanks @gaoflow !

@FrancescAlted FrancescAlted merged commit b065396 into pydata:master Jun 24, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

evaluate(..., out=arr) keeps arr pinned in the thread-local _numexpr_last cache

3 participants