Skip to content

corrections around data checker GPU and transfer optimisation for save_intermediate - #750

Draft
dkazanc wants to merge 4 commits into
mainfrom
data_reducer_order
Draft

corrections around data checker GPU and transfer optimisation for save_intermediate #750
dkazanc wants to merge 4 commits into
mainfrom
data_reducer_order

Conversation

@dkazanc

@dkazanc dkazanc commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

pls ignore the name for the branch :)

  1. It removes the insertion of data_checker (GPU) after save_intermediate_data method. Data is being checked before save_intermediate_data. So the order is more logical now:
    total_variation_PD (httomolibgpu)
    --- output check with data_checker --- (httomolibgpu)
    save_intermediate_data (httomo)
    calculate_stats (httomo)

compared to:

    total_variation_PD (httomolibgpu)
    save_intermediate_data (httomo)
    --- output check with data_checker --- (httomolibgpu)
    calculate_stats (httomo)
  1. More major change around the transfer in SaveIntermediateFilesWrapper. Now the block can be transferred wholly to CPU, only if the method that follows is a CPU method. That saves of transferring from Device to Host twice. If the method after is a GPU method or there is no method then we only copy block to CPU in order to be saved, while the block itself remains on the GPU.

  2. I also removed nan_to_num bit from calculate stats wrapper as all the data should be corrected with data checker beforehand.

One of the main reasons of doing this change is trying to understand the GPU transfers related numbers for the paper.
So to add more context, this was the result on the Blackwell node before the changes and this stuff around calculate_stats really confused me, like why is this the GPU kernel time, why it is so long as well.
blackwell4_119647_lprec_TV_blackwell_part1

And this after the changes I've made. calculate_stats is purely on the CPU as the block-data already converted beforehand in save_intermediate_data method. It is indeed faster (few seconds for this small data but when we scale up it will be more noticeable).
blackwell4_119647_lprec_TV_blackwell_part1_optimised

Checklist

  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have made corresponding changes to the documentation
  • I have added the user-release-note label in order to include this PR in the "Notable
    Changes for Users" section in release notes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant