PCIe - GPU Bandwidth Plugin#

The PCIe plugin detects underperforming or unreliable host-to-GPU and peer-to-peer GPU transfer paths. Depending on the enabled subtests, it flags link downtraining, low bandwidth, high latency, excessive PCIe recovery activity, and data corruption; GPU peer paths can use NVLink when topology allows.

Availability and prerequisites#

The canonical test name is pcie. It runs on selected physical GPUs. Host-to-device subtests require one GPU; peer-to-peer and concurrent subtests require a selection and topology from which the applicable GPU pairs can be formed. Run the plugin on idle GPUs so competing transfers do not distort bandwidth or latency measurements.

Behavior#

The plugin consists of several self-tests that each measure a different aspect of bandwidth or latency. Each subtest has either a pinned/unpinned pair or a p2p enabled/p2p disabled pair of identical tests. Pinned/unpinned tests use either pinned or unpinned memory when copying data between the host and the GPUs.

This plugin will use NvLink to communicate between GPUs when possible. Otherwise, communication between GPUs will occur over PCIe

Each sub test is represented with a tag that is used both for specifying configuration parameters for the sub test and for outputting stats for the sub test. P2p enabled/p2p disabled tests enable or disable GPUs writing and reading to and from each other directly rather than through the PCIe bus.

Subtests#

Sub Test Tag	Pinned/Unpinned P2P Enabled/P2P Disabled	Description
h2d_d2h_single_pinned	Pinned	Device <-> Host Bandwidth, one GPU at a time
h2d_d2h_single_unpinned	Unpinned	Device <-> Host Bandwidth, one GPU at a time
h2d_d2h_concurrent_pinned	Pinned	Device <-> Host bandwidth, with all selected GPUs transferring concurrently
h2d_d2h_concurrent_unpinned	Unpinned	Device <-> Host bandwidth, with all selected GPUs transferring concurrently
h2d_d2h_latency_pinned	Pinned	Device <-> Host Latency, one GPU at a time
h2d_d2h_latency_unpinned	Unpinned	Device <-> Host Latency, one GPU at a time
p2p_bw_p2p_enabled	P2P Enabled	Device <-> Device bandwidth one GPU pair at a time
p2p_bw_p2p_disabled	P2P Disabled	Device <-> Device bandwidth one GPU pair at a time
p2p_bw_concurrent_p2p_enabled	P2P Enabled	Device <-> Device bandwidth, concurrently, focusing on bandwidth between GPUs between GPUs likely to be directly connected to each other -> for each (index / 2) and (index / 2)+1
p2p_bw_concurrent_p2p_disabled	P2P Disabled	Device <-> Device bandwidth, concurrently, focusing on bandwidth between GPUs between GPUs likely to be directly connected to each other -> for each (index / 2) and (index / 2)+1
1d_exch_bw_p2p_enabled	P2P Enabled	Device <-> Device bandwidth, concurrently, focusing on bandwidth between gpus, every GPU either sending to the gpu with the index higher than itself (l2r) or to the gpu with the index lower than itself (r2l)
1d_exch_bw_p2p_disabled	P2P Disabled	Device <-> Device bandwidth, concurrently, focusing on bandwidth between gpus, every GPU either sending to the gpu with the index higher than itself (l2r) or to the gpu with the index lower than itself (r2l)
p2p_latency_p2p_enabled	P2P Enabled	Device <-> Device Latency, one GPU pair at a time
p2p_latency_p2p_disabled	P2P Disabled	Device <-> Device Latency, one GPU pair at a time

Parameters#

Execution selectors#

The following table lists the global parameters for the PCIe plugin.

Parameter	Type	Default	Description
test_pinned	Bool	True	Include subtests that test using pinned memory.
test_unpinned	Bool	True	Include subtests that test using unpinned memory.
test_p2p_on	Bool	True	Run relevant subtests with peer to peer (P2P) memory transfers between GPUs enabled.
test_p2p_off	Bool	True	Run relevant subtests with peer to peer (P2P) memory transfers between GPUs disabled.
test_broken_p2p	Bool	True	Check peer-to-peer copies for data corruption and identify the reader or writer GPU involved when possible.
is_allowed	Bool	False	Allow the plugin to run. Numbered-suite configuration can override this product-dependent default.
max_pcie_replays	Float	80.0	Maximum number of PCIe replays to allow per GPU during the plugin run.

Subtest thresholds#

The following table lists parameters that apply to specific PCIe subtests.

Parameter	Type	Default	Applies to	Description
min_bandwidth	Number	0	h2d_d2h_single_pinned, h2d_d2h_single_unpinned, h2d_d2h_concurrent_pinned, h2d_d2h_concurrent_unpinned	Minimum bandwidth in GB/s that must be reached for this sub-test to pass.
max_latency	Number	100,000	h2d_d2h_latency_pinned, h2d_d2h_latency_unpinned	Latency in microseconds that cannot be exceeded for this sub-test to pass.
min_pci_generation	Number	1.0	h2d_d2h_single_pinned, h2d_d2h_single_unpinned	Minimum allowed PCI generation that the GPU must be at or exceed for this sub-test to pass.
min_pci_width	Number	1.0	h2d_d2h_single_pinned, h2d_d2h_single_unpinned	Minimum allowed PCI width that the GPU must be at or exceed for this sub-test to pass. For example, 16x = 16.0.

Examples#

Run the PCIe plugin on two GPUs so that both host-to-device and applicable peer-to-peer paths can be exercised:

$ dcgmi diag --run pcie --entity-id gpu:0,gpu:1

Results and failure conditions#

PCIe plugin results#
Condition	Result or code	Interpretation
A bandwidth subtest is below its configured `min_bandwidth`.	`DCGM_FR_LOW_BANDWIDTH`; fail	The message identifies the GPU, direction, measured value, and threshold.
A latency subtest exceeds its configured `max_latency`.	`DCGM_FR_HIGH_LATENCY`; fail	The tested transfer path did not meet the configured latency limit.
Negotiated PCIe generation or width is below the configured minimum.	`DCGM_FR_PCIE_GENERATION` or `DCGM_FR_PCIE_WIDTH`; fail	Compare the result with the platform topology and expected link state.
A peer-to-peer correctness check reads corrupted data.	`DCGM_FR_BROKEN_P2P_MEMORY_DEVICE`, `DCGM_FR_BROKEN_P2P_WRITER_DEVICE`, or the corresponding PCIe- or NVLink-specific code; fail	The specific code distinguishes the reader or writer and, when known, whether the path used PCIe or NVLink.
PCIe replay or correctable-error activity exceeds its configured threshold.	`DCGM_FR_PCIE_REPLAY_THRESHOLD_VIOLATION` or `DCGM_FR_PCIE_H_REPLAY_VIOLATION`; fail	The link reported excessive recovery activity during the test.
Fewer GPU or NVSwitch NVLinks are up than an explicitly configured expectation.	`DCGM_FR_GPU_EXPECTED_NVLINKS_UP` or `DCGM_FR_NVSWITCH_EXPECTED_NVLINKS_UP`; fail	Inspect link and Fabric Manager state before rerunning a bandwidth workload.
A common monitored condition occurs, such as an uncorrectable memory error, XID, row-remap problem, thermal violation, or enabled error threshold.	Fail with the condition-specific `DCGM_FR_*` code	See Diagnostic Errors for the reported condition.
A concurrent subtest cannot be meaningfully formed from the selected GPUs.	`DCGM_FR_CONCURRENT_GPUS`; informational subtest omission	Other applicable PCIe subtests can still produce results.
DCGM field access, CUDA setup, NUMA setup, or another internal operation fails.	`DCGM_FR_DCGM_API` or `DCGM_FR_INTERNAL`; fail	This is an execution or environment failure, not a measured link threshold violation.
The plugin is disabled or the run is interrupted.	`DCGM_FR_TEST_DISABLED` or `DCGM_FR_ABORTED`; skip	The plugin did not complete and made no health determination.