Skip to content

Reduce memory usage of sample-level result accumulators - #109

Open
akrivi wants to merge 3 commits into
mainfrom
al/fix_threads_samples_memory
Open

Reduce memory usage of sample-level result accumulators#109
akrivi wants to merge 3 commits into
mainfrom
al/fix_threads_samples_memory

Conversation

@akrivi

@akrivi akrivi commented May 20, 2026

Copy link
Copy Markdown
Collaborator

Previously, threaded execution created worker-local sample accumulators sized for the full Monte Carlo sample count. For sample-based result specs like ShortfallSamples this caused memory usage to scale as O(regions x timesteps x samples x threads).

This PR changes sample-based result accumulation so each threaded worker's recorder stores only that worker's assigned sample range. The partitions are then combined during finalization to produce the same full sample result.

Example for 3 threaded workers:

Before, each worker accumulator allocated the whole sample matrix:

Worker 1 accumulator: [total number of samples] 

Worker 2 accumulator: [total number of samples]

Worker 3 accumulator: [total number of samples]

Now, samples are split into ranges:

Worker 1 accumulator: [first 1/3 of total number of samples] 

Worker 2 accumulator: [second 1/3 of total number of samples]

Worker 3 accumulator: [last 1/3 of total number of samples]

Benchmarks

System: Guam 2028, 13 regions, 8760 timestamps, hourly resolution
Simulation: Run on HPC, using standard nodes (104 cores, 250 GB)
Result: ShortfallSamples()

1000 MC Samples

threads Elapsed time(s) - main Elapsed time(s) - PR Max RSS(GB) - main Max RSS(GB) - PR
1 23.61 23.75 1.57 1.57
2 13.04 12.96 2.37 2.38
4 8.28 8.16 4.09 2.38
8 6.68 5.24 7.47 2.41
16 6.84 3.87 14.31 2.43
24 7.39 3.48 21.14 2.46
32 9.59 3.28 27.99 2.49
48 11.16 3.06 41.69 2.62
64 14.53 2.97 55.28 2.61
80 15.04 2.96 69.01 2.75
96 21.48 3.06 82.61 2.79

10000 MC Samples

threads Elapsed time(s) - main Elapsed time(s) - PR Max RSS(GB) - main Max RSS(GB) - PR
1 210.73 208.58 9.21 9.20
2 109.29 108.45 17.67 17.65
4 61.39 56.88 34.64 17.64
8 46.96 33.21 68.58 17.66
16 43.08 18.88 136.53 17.69
24 51.95 13.68 204.37 17.77

@codecov-commenter

codecov-commenter commented May 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.59155% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 83.28%. Comparing base (f49c56e) to head (8adce48).

Files with missing lines Patch % Lines
PRASCore.jl/src/Simulations/Simulations.jl 97.14% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #109      +/-   ##
==========================================
- Coverage   84.14%   83.28%   -0.86%     
==========================================
  Files          45       45              
  Lines        2491     2537      +46     
==========================================
+ Hits         2096     2113      +17     
- Misses        395      424      +29     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@akrivi
akrivi force-pushed the al/fix_threads_samples_memory branch from f6313c9 to b4b5ba3 Compare July 5, 2026 21:26
@akrivi
akrivi force-pushed the al/fix_threads_samples_memory branch from b113d5e to 8adce48 Compare August 3, 2026 17:47
@akrivi akrivi changed the title fix: High memory usage from sample-level result accumulators Reduce memory usage of sample-level result accumulators Aug 3, 2026
@akrivi
akrivi requested a review from scdhulipala August 5, 2026 19:11

abstract type ResultAccumulator{R<:ResultSpec} end

issamplebased(::ResultSpec) = false

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth it to just do this the other way round, so you don't have to define this for all the non sample ResultSpec s?

) where {A<:ResultAccumulator}

field = fieldnames(A)[1]
xarr = getfield(x, field)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this leads to some performance issues. Have you tested this on very high sample results?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If only grabbing the 1st filed is correct (I don't thnk it is), repalcing with this could work:

`function copy_sample_partition!(
x::A,
y::A,
sampleids::UnitRange{Int},
) where {A<:ResultAccumulator}

# Using a literal integer makes this completely type-stable
xarr = getfield(x, 1)
yarr = getfield(y, 1)

@views xarr[:, :, sampleids] .= yarr
return nothing

end`

This explicitly tells the complier the field and the type of the field you want to grab.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants