Skip to content

FEAT: allow use of different numbers of samples per event - #142

Merged
ColmTalbot merged 4 commits into
mainfrom
variable-sample-number
Jul 17, 2026
Merged

FEAT: allow use of different numbers of samples per event#142
ColmTalbot merged 4 commits into
mainfrom
variable-sample-number

Conversation

@ColmTalbot

Copy link
Copy Markdown
Owner

This uses an approach suggested by @mdmould to allow different numbers of samples per event.

@mdmould, do you think you could take a look at this?

@ColmTalbot ColmTalbot added the enhancement New feature or request label Jul 16, 2026
@ColmTalbot
ColmTalbot requested a review from Copilot July 16, 2026 21:14
@codecov

codecov Bot commented Jul 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.21%. Comparing base (16b209d) to head (3b8045d).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #142      +/-   ##
==========================================
+ Coverage   95.15%   95.21%   +0.06%     
==========================================
  Files          14       14              
  Lines         907      940      +33     
==========================================
+ Hits          863      895      +32     
- Misses         44       45       +1     
Flag Coverage Δ
python3.10 95.21% <100.00%> (+0.06%) ⬆️
python3.11 95.21% <100.00%> (+0.06%) ⬆️
python3.12 95.21% <100.00%> (+0.06%) ⬆️
python3.13 95.21% <100.00%> (+0.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds an opt-in/opt-out mechanism to support different numbers of posterior samples per event in HyperparameterLikelihood, while retaining a backward-compatible mode that enforces equal sample counts per event.

Changes:

  • Add require_equal_samples flag and implement per-event sample-count handling via flattened storage + event transitions.
  • Refactor expectation/variance calculations to work with either equal-sample (2D) or variable-sample (flattened) representations.
  • Update tests to explicitly request backward-compatible behavior and to parametrize the new flag in the example test.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 9 comments.

File Description
gwpopulation/hyperpe.py Adds require_equal_samples flag, implements variable-sample resampling, and centralizes expectation logic.
gwpopulation/experimental/numpyro.py Switches expectation computation to reuse likelihood._weight_expectation for consistency with new storage modes.
test/likelihood_test.py Updates existing tests to enforce backward-compatible equal-sample behavior explicitly.
test/example_test.py Parametrizes tests over require_equal_samples (and JIT) to exercise both modes.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread gwpopulation/hyperpe.py Outdated
Comment thread gwpopulation/hyperpe.py
Comment thread gwpopulation/hyperpe.py Outdated
Comment thread gwpopulation/hyperpe.py Outdated
Comment thread gwpopulation/hyperpe.py Outdated
Comment thread gwpopulation/hyperpe.py
Comment thread gwpopulation/hyperpe.py
Comment thread gwpopulation/hyperpe.py
Comment thread test/example_test.py
@mdmould

mdmould commented Jul 16, 2026

Copy link
Copy Markdown

I'll take a look. But I'll note that I've run into issues with this approach in practice, which manifests differently depending on the sampler:

  • For nested sampling (specifically, dynesty via bilby), things will steadily converge, but eventually dlogz will keep jumping up.
  • For HMC (specifically, NUTS in numpyro), warmup adaptation will start working, but eventually chains will get stuck (step size become basically zero).

I'm still not sure why this happens. A couple of ideas are that I'm using way too many samples overall, or that summing in probability space instead of log space over all those samples is just not stable. I need to do some more systematic tests though, because it does work in some cases 🤷

@ColmTalbot

Copy link
Copy Markdown
Owner Author

Thanks for the comments!
Interesting, I think I've seen the same thing with the equal samples version, sometimes increasing the number of samples makes MC effects worse.
I usually put it down to being more likely to get a large outlier if you have more samples, but have never gone through the process to show it.

@ColmTalbot
ColmTalbot merged commit b3a34f9 into main Jul 17, 2026
9 checks passed
@ColmTalbot
ColmTalbot deleted the variable-sample-number branch July 17, 2026 14:38
@mdmould

mdmould commented Jul 20, 2026

Copy link
Copy Markdown

I've also seen that issue with the equal samples version. I compared the evaluation of the likelihood and variance with my implementation (on GWTC-5 BBHs) - the variance matched but only to 1 d.p. (even in 64 bit, maybe expected given such a large cumulative sum?), and the log likelihoods differed but by a fixed constant (again to around 1 d.p., I'm not sure where the difference in normalization lies but it doesn't matter anyway)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants