corrections around data checker GPU and transfer optimisation for save_intermediate - #750
Draft
dkazanc wants to merge 4 commits into
Draft
corrections around data checker GPU and transfer optimisation for save_intermediate #750dkazanc wants to merge 4 commits into
dkazanc wants to merge 4 commits into
Conversation
…ng the bug on the time kernel counter
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pls ignore the name for the branch :)
data_checker(GPU) aftersave_intermediate_datamethod. Data is being checked beforesave_intermediate_data. So the order is more logical now:compared to:
More major change around the transfer in
SaveIntermediateFilesWrapper. Now the block can be transferred wholly to CPU, only if the method that follows is a CPU method. That saves of transferring from Device to Host twice. If the method after is a GPU method or there is no method then we only copy block to CPU in order to be saved, while the block itself remains on the GPU.I also removed
nan_to_numbit from calculate stats wrapper as all the data should be corrected with data checker beforehand.One of the main reasons of doing this change is trying to understand the GPU transfers related numbers for the paper.

So to add more context, this was the result on the Blackwell node before the changes and this stuff around
calculate_statsreally confused me, like why is this the GPU kernel time, why it is so long as well.And this after the changes I've made.

calculate_statsis purely on the CPU as the block-data already converted beforehand insave_intermediate_datamethod. It is indeed faster (few seconds for this small data but when we scale up it will be more noticeable).Checklist
user-release-notelabel in order to include this PR in the "NotableChanges for Users" section in release notes