dcgmproftester#

NAME#

dcgmproftester11, dcgmproftester12, dcgmproftester13 - generate CUDA loads for DCGM profiling fields and optionally compare the measured values

SYNOPSIS#

dcgmproftester<cuda-major> [-t <field-list>]
                              [-i <gpu-id-list>] [-d <seconds>]
                              [--no-dcgm-validation] [<options>]
dcgmproftester<cuda-major> --version
dcgmproftester<cuda-major> --help

<cuda-major> is 11, 12, or 13. <field-list> is a comma-separated list of implemented profiling field IDs, nonvlink, or nvlink. See WORKLOAD FIELD REFERENCE before using all.

DESCRIPTION#

dcgmproftester creates controlled CUDA activity for selected DCGM profiling fields. It can report the load it generated, read the corresponding field through an embedded DCGM host engine, and optionally compare the two. The three executable variants have the same command-line interface.

Warning

This is an active workload generator. It consumes GPU compute, memory, PCIe, or NVLink resources and can disturb applications and measurements on the selected devices. Do not rely on the default all-GPU selection on a shared system. Choose an idle physical GPU explicitly with -i and coordinate use of profiling counters with other users.

The process always initializes DCGM in embedded mode to enumerate physical GPUs, MIG instances, and GPU architecture. It has no --host option and does not send its test request to the configured nv-hostengine.

Without --no-dcgm-validation, the embedded host engine creates temporary entity and field groups and watches the requested profiling field. Profiling counter access commonly requires root or the corresponding driver permission. It can also fail with DCGM_ST_IN_USE when the system host engine, Nsight, or another process owns the profiling resource.

See the Linux process-credentials note when validation succeeds under sudo but fails for the invoking account.

To have the running host engine measure the workload, run dcgmproftester with --no-dcgm-validation and observe the load with dcgmi dmon. The option suppresses the embedded profiling watch and reads; it does not prevent embedded DCGM initialization.

BINARY SELECTION#

The number at the end of the executable name is the CUDA major version against which that executable was built and linked. It is not the dcgmproftester application version. In DCGM 4.6, all three --version commands report application version 2.0.0.

Choose the executable whose suffix matches the CUDA compatibility major version reported by the active NVIDIA driver. The program checks the driver through cuDriverGetVersion() before creating a workload and rejects an ordinary mismatch. The CUDA toolkit selected in an interactive shell does not override the executable’s linked major version.

The current selection rules are:

Use dcgmproftester11 when the driver reports CUDA major version 11.
Use dcgmproftester12 when the driver reports CUDA major version 12.
Use dcgmproftester13 when the driver reports CUDA major version 13.
If a future driver reports a major version newer than the newest variant shipped by this DCGM release, use the newest shipped variant. DCGM 4.6 therefore permits dcgmproftester13 in that case.
On a driver reporting CUDA 13.0, Maxwell, Pascal, and Volta GPUs (compute capability below 7.5) must use dcgmproftester12 because CUDA 13.0 removed those architectures. The program detects this case and directs the user to the CUDA 12 variant.

The CUDA version shown by nvidia-smi is the driver’s maximum supported CUDA compatibility version and is the relevant starting point for this choice. A version error from dcgmproftester reports both the expected and detected major versions.

WORKLOAD FIELD REFERENCE#

The generator implements workloads for the following twelve field IDs. The field tag is the name printed by dcgmi profile --list and used throughout the DCGM profiling documentation.

Implemented dcgmproftester workloads#
ID	Field tag	Metric driven	Workload and restrictions
1001	`gr_engine_active`	Graphics/compute engine activity	Varies engine activity. `--target-max-value` selects its maximum- target subtest.
1002	`sm_active`	SM activity	Varies activity across streaming multiprocessors. `--target-max-value` selects its maximum-target subtest.
1003	`sm_occupancy`	SM occupancy	Varies resident work and thread count. `--target-max-value` selects its maximum-target subtest.
1004	`tensor_active`	Tensor-pipe activity	Runs a tensor GEMM workload.
1005	`dram_active`	Device-memory activity	Drives device-memory transfers. On MIG, one compute instance per GPU instance is used for this field.
1006	`fp64_active`	FP64-pipe activity	Runs FP64 arithmetic; `--cublas` selects the cuBLAS implementation.
1007	`fp32_active`	FP32-pipe activity	Runs FP32 arithmetic; `--cublas` selects the cuBLAS implementation.
1008	`fp16_active`	FP16-pipe activity	Runs FP16 arithmetic; `--cublas` selects the cuBLAS implementation.
1009	`pcie_tx_bytes`	PCIe transmit bandwidth	Drives host/device transfers. Testing is serialized across physical GPUs regardless of `--max-processes`.
1010	`pcie_rx_bytes`	PCIe receive bandwidth	Drives host/device transfers. Testing is serialized across physical GPUs regardless of `--max-processes`.
1011	`nvlink_tx_bytes`	NVLink transmit bandwidth	Copies to the best connected GPU peer. Requires an NVLink peer, is not available in MIG mode, and is serialized across physical GPUs.
1012	`nvlink_rx_bytes`	NVLink receive bandwidth	Copies from the best connected GPU peer. Requires an NVLink peer, is not available in MIG mode, and is serialized across physical GPUs.

Implementation and hardware support are separate tests. First select an ID from this table, then list the profiling fields supported by the target GPU:

$ dcgmi profile --list --entity-id gpu:0

The usable set is the intersection of this table and that command’s output. See Profiling for full metric definitions.

The parser also accepts every profiling field ID from 1001 through 1096. Most IDs after 1012 do not have a generator implementation. Selecting one of them reports that the test is not implemented and fails the run.

nonvlink expands to IDs 1001 through 1010, all of which are implemented. nvlink expands to IDs 1011 and 1012. all expands to every profiling ID from 1001 through 1096; omitting -t has the same effect. Consequently, all and a no-argument invocation encounter unimplemented tests in DCGM 4.6 and are not useful defaults. Prefer explicit IDs.

OPTIONS#

Target and workload selection#

-t <fields>, --fieldId <fields>

Select a comma-separated list of numeric profiling field IDs, or one of nonvlink, nvlink, and all. The default is all. Option names are case-sensitive; the long spelling is --fieldId.

A comma-separated numeric list applies one set of options to every listed field. Fields run sequentially. See MULTIPLE FIELD SETS for repeated -t behavior.

-i <gpu-ids>, --gpuIds <gpu-ids>

Select a comma-separated list of physical DCGM GPU IDs. Omission or all selects every GPU returned by embedded DCGM. Obtain current IDs with dcgmi discovery --list. This option does not accept gpu:0 entity syntax, UUIDs, or individual MIG instance IDs.

-d <seconds>, --duration <seconds>

Set workload duration in seconds. The default is 30 and the minimum is 1. Multiple selected fields run sequentially, so their durations are additive; initialization, synchronization, and cleanup add elapsed time.

--max-processes <count>

Limit the number of physical GPUs tested concurrently. The default is 4; 0 means all selected GPUs, and useful nonzero values are 1 through 255. Supply a whole number. PCIe and NVLink fields force a limit of 1.

--target-max-value

For fields 1001, 1002, and 1003, run the maximum-target occupancy subtest instead of the normal stepped workload. It has no implemented effect for the other fields.

--cublas

Use the cuBLAS-based workload for fields 1006, 1007, and 1008 instead of the default CUDA-kernel implementation. Field 1004 already uses its GEMM workload and is not switched by this option.

DCGM measurement and validation#

--no-dcgm-validation

Do not create embedded DCGM profiling groups or watches, and do not read the selected profiling field. CUDA load generation still occurs, embedded DCGM is still initialized for inventory, and result files are still written. Use this mode when a standalone host engine observes the workload or when another process owns profiling counters.

--mode <tokens>

Set a comma-separated operational mode. Each token enables a behavior; a no prefix disables it. The recognized pairs are generateload/nogenerateload, report/noreport, validate/novalidate, and fast/nofast. Later tokens for the same behavior win. The default is generateload,report,novalidate,nofast.

report prints per-sample generated and DCGM values; noreport suppresses those reports but not final status or result files. validate compares DCGM samples with the generated target; the default novalidate merely reports samples. fast is meaningful with validate and stops a subtest after the first passing sample rather than completing the normal sequence.

In DCGM 4.6, the parsed generateload setting is not consulted by the workload implementation, so nogenerateload does not disable the load. Unknown mode tokens are logged but do not make parsing fail. Use only the exact tokens above and do not use this option as a load-disable control.

-w <percent>, --wait-to-check <percent>

With --mode validate, ignore comparison failures during the initial portion of a normal run. The default is 0. Supply a whole percentage from 0 through 99. Fast mode does not honor this waiting period.

--percent-tolerance <percent>

With --mode validate, allow this percentage below the lower generated target and above the upper generated target. The default validation tolerance is 10 percent. Supply a value greater than or equal to 0 and less than 100. Specifying this option replaces an explicit absolute tolerance; if both appear, the later one wins.

-a <value>, --absolute-tolerance <value>

With --mode validate, allow this absolute amount below and above the generated target interval. The value must be nonnegative. Specifying this option replaces an explicit percentage tolerance; if both appear, the later one wins.

--min-value <value>, -m <value>, --match-value <value>, --max-value <value>

Accepted by the DCGM 4.6 parser, which calculates an explicit comparison range from these values and the selected tolerance. A source defect leaves that range inactive, so the validator continues to compare against its generated target. Do not rely on these options in DCGM 4.6. On fields 1006 and 1008, specifying one also disables the built-in 0.5-through-1.0 range.

--sync-count <count>

Set the number of retries used while synchronizing generated activity with a measurement before advancing. The default is 1 and the accepted range is 0 through 10. Zero disables synchronous advancement.

Reporting and compatibility output#

-r <seconds>, --report <seconds>: Set the DCGM watch and report interval. The default is 1 second. Use a value greater than 0 and no greater than 10. This option is unrelated to the report/noreport mode token.
--dvs: Add DVS-style &&&& RUNNING, &&&& PASSED, and &&&& FAILED tags to standard output. Despite the generated help text, this switch enables rather than disables DVS output.

Logging and parser control#

--log-file <path>: Write diagnostic logging to this file. The default is dcgmproftester.log in the current directory. Logging is initialized once, for the first field set; place a nondefault option before the first -t when using repeated field sets.
--log-level <level>: Set file logging severity. Accepted values are NONE, FATAL, ERROR, WARN, INFO, DEBUG, and VERB, without regard to case. The default is INFO. DCGM 4.6 does not reject an unknown string; it silently uses INFO, so use a listed value.
--reset: Reset --target-max-value, --no-dcgm-validation, and --dvs to false for the current and subsequent repeated field sets. It does not reset the other switches or value options despite its general help text.
-c <path>, --config <path>: Accepted but not used by the DCGM 4.6 implementation. The path is not opened and no YAML or JSON settings are loaded. Express settings on the command line.
--version: Print the local dcgmproftester application version and exit before DCGM or CUDA initialization. The CUDA major remains part of the executable name, not this version string.
-h, --help: Print generated usage information and exit before DCGM or CUDA initialization.
--, --ignore_rest: Stop parsing labeled options. This is an internal TCLAP compatibility feature and has no normal workload use.

MULTIPLE FIELD SETS#

A comma-separated -t value gives all fields the same settings. This is the least surprising form and is recommended for most uses:

$ dcgmproftester13 -i 0 -t 1004,1005 -d 20 --no-dcgm-validation

Repeated -t options create stateful field sets. Options after one -t and before the next apply when the preceding field set is committed; the last set is committed at end of parsing. Settings persist into later sets until changed. For example, this runs field 1004 for 10 seconds and field 1005 for 30 seconds:

$ dcgmproftester13 -i 0 --no-dcgm-validation \
    -t 1004 -d 10 -t 1005 -d 30

Because logging is initialized from the first committed set and cannot move to another file during the process, put --log-file before the first -t. Prefer separate invocations when substantially different settings or independent exit status are required.

GPU, MIG, AND PEER SCOPE#

-i selects physical DCGM GPU IDs. On a selected GPU in MIG mode, the tool runs workers for all available compute instances; it has no command-line selector for one GPU instance or compute instance. The DRAM workload uses one compute instance from each GPU instance, and NVLink workloads reject MIG mode.

When embedded validation is enabled for several GPUs, the selected group must be homogeneous. Select same-SKU GPUs explicitly. --max-processes controls ordinary compute fields, while PCIe and NVLink fields are always serialized to avoid shared-fabric interference.

An NVLink workload chooses the peer with the greatest number of NVLink connections. A GPU with no NVLink-connected peer cannot run fields 1011 or 1012. Use dcgmi topo and dcgmi nvlink to inspect topology and link state before selecting those fields.

For an ordinary non-MIG, non-NVLink workload, a nonempty inherited CUDA_VISIBLE_DEVICES value is preserved in each CUDA worker. MIG and NVLink workers override it with the visibility set calculated from DCGM. Align or unset an inherited value when it does not contain the physical GPU selected with -i.

OUTPUT AND FILES#

With the default report mode, standard output contains periodic human- readable worker reports with the generated target and DCGM value. Diagnostic errors can appear on standard error, standard output, and in the log. There is no JSON output mode.

Every started subtest attempts to write a result file in the current working directory, even with --no-dcgm-validation or --mode noreport. Its name is <subtest-tag>_<gpu-id>.results, for example tensor_active_0.results. A later subtest with the same tag and GPU ID truncates and replaces that file. The file contains generated/DCGM columns and a final PASSED or FAILED line; a load-only run can have no DCGM samples.

./dcgmproftester.log: Default log path, relative to the invocation’s working directory.

EXIT STATUS#

Zero means that the top-level run returned DCGM_ST_OK. A nonzero status means parsing, initialization, a workload, validation, or cleanup failed. The program returns raw dcgmReturn_t values for many failures, so negative values wrap into the shell’s 0-through-255 range. Parser constraint failures can instead exit 1. Automation should distinguish only zero from nonzero and use the diagnostic text for the cause.

The shell conversion follows the conventions described in EXIT STATUS; the status is not a signed dcgmReturn_t after the shell has observed it.

DCGM 4.6 converts DCGM_ST_NOT_SUPPORTED from a field run to success. For example, an NVLink test on a GPU with no peer can print a skip message, All Tests Passed., and exit zero. Therefore, run one explicit field per automation invocation and require its field-specific pass/report output in addition to status zero. Do not treat All Tests Passed. alone as proof that the requested workload ran.

EXAMPLES#

Discover physical GPU IDs and the profiling fields supported by GPU 0:

$ dcgmi discovery --list
$ dcgmi profile --list --entity-id gpu:0

Generate a tensor workload on GPU 0 for 30 seconds while the system host engine owns measurement. Choose the executable suffix as described in BINARY SELECTION:

$ dcgmproftester13 --gpuIds 0 --fieldId 1004 --duration 30 \
    --no-dcgm-validation

In another terminal, observe that workload through the running host engine:

$ dcgmi dmon --field-id 1004 --entity-id gpu:0 --delay 1000

Run embedded comparison of tensor activity with a 5-percent tolerance. This form must be the sole profiling consumer and commonly requires elevated profiling permission:

$ sudo dcgmproftester13 -i 0 -t 1004 -d 30 \
    --mode generateload,report,validate,nofast \
    --percent-tolerance 5

Generate both PCIe directions on GPU 2 without acquiring profiling counters:

$ dcgmproftester12 -i 2 -t 1009,1010 -d 15 \
    --no-dcgm-validation