p2pBandwidthLatencyTest shows large run-to-run variance with P2P enabled due to low repeat count

Description

I observed large run-to-run variance in p2pBandwidthLatencyTest when measuring unidirectional bandwidth with P2P enabled.

In my test environment, I ran the unidirectional P2P-enabled bandwidth test 10 times. For each run, the result is a 2D bandwidth matrix. I then compared the same (src GPU, dst GPU) data point across the 10 runs and calculated: range / mean

The variance is relatively large for many GPU pairs. Increasing the internal repeat count in outputBandwidthMatrix() from 5 to 500 made the results much more stable in my environment.

The current code uses only 5 repeats: void outputBandwidthMatrix(int numElems, int numGPUs, bool p2p, P2PDataTransfer p2p_method, bool needsFallback) { int repeat = 5; ... }

With this setting, the unidirectional P2P-enabled results show noticeable run-to-run fluctuation.

Across 10 runs, the unidirectional P2P-enabled results show noticeable variance. The range / mean is mostly between 5% and 8%, with a maximum of 10.0%. Increasing repeat from 5 to 500 made the results much more stable in my environment. Experimented change

Changing the repeat count from 5 to 500 significantly improved result stability in my environment:

diff --git a/cpp/5_Domain_Specific/p2pBandwidthLatencyTest/p2pBandwidthLatencyTest.cu b/cpp/5_Domain_Specific/p2pBandwidthLatencyTest/p2pBandwidthLatencyTest.cu index 0c642fcd..6c1470d6 100644 --- a/cpp/5_Domain_Specific/p2pBandwidthLatencyTest/p2pBandwidthLatencyTest.cu +++ b/cpp/5_Domain_Specific/p2pBandwidthLatencyTest/p2pBandwidthLatencyTest.cu @@ -228,7 +228,7 @@ void performP2PCopy(int *dest,

void outputBandwidthMatrix(int numElems, int numGPUs, bool p2p, P2PDataTransfer p2p_method, bool needsFallback) {

• int repeat = 5;

• int repeat = 500; volatile int *flag = NULL; vector<int *> buffers(numGPUs); vector<int *> buffersD2D(numGPUs); // buffer for D2D, that is, intra-GPU copy