p2pBandwidthLatencyTest shows large run-to-run variance with P2P enabled due to low repeat count

[zhiguangni01-lang]

Description

I observed large run-to-run variance in p2pBandwidthLatencyTest when measuring unidirectional bandwidth with P2P enabled.

In my test environment, I ran the unidirectional P2P-enabled bandwidth test 10 times. For each run, the result is a 2D bandwidth matrix. I then compared the same (src GPU, dst GPU) data point across the 10
runs and calculated:
range / mean

The variance is relatively large for many GPU pairs. Increasing the internal repeat count in outputBandwidthMatrix() from 5 to 500 made the results much more stable in my environment.

The current code uses only 5 repeats:
void outputBandwidthMatrix(int numElems, int numGPUs, bool p2p, P2PDataTransfer p2p_method, bool needsFallback)
{
int repeat = 5;
...
}

With this setting, the unidirectional P2P-enabled results show noticeable run-to-run fluctuation.

Across 10 runs, the unidirectional P2P-enabled results show noticeable variance.
The range / mean is mostly between 5% and 8%, with a maximum of 10.0%.
Increasing repeat from 5 to 500 made the results much more stable in my environment.
Experimented change

Changing the repeat count from 5 to 500 significantly improved result stability in my environment:

diff --git a/cpp/5_Domain_Specific/p2pBandwidthLatencyTest/p2pBandwidthLatencyTest.cu b/cpp/5_Domain_Specific/p2pBandwidthLatencyTest/p2pBandwidthLatencyTest.cu
index 0c642fcd..6c1470d6 100644
--- a/cpp/5_Domain_Specific/p2pBandwidthLatencyTest/p2pBandwidthLatencyTest.cu
+++ b/cpp/5_Domain_Specific/p2pBandwidthLatencyTest/p2pBandwidthLatencyTest.cu
@@ -228,7 +228,7 @@ void performP2PCopy(int *dest,

void outputBandwidthMatrix(int numElems, int numGPUs, bool p2p, P2PDataTransfer p2p_method, bool needsFallback)
{

• int repeat = 5;

• int repeat = 500;
  volatile int *flag = NULL;
  vector<int *> buffers(numGPUs);
  vector<int *> buffersD2D(numGPUs); // buffer for D2D, that is, intra-GPU copy