Is this page helpful?

Accounting Statistics#

Set of APIs designed to provide per process information about usage of GPU.

Note

All accounting statistics and accounting mode live in nvidia driver and reset to default (Disabled) when driver unloads. It is advised to run with persistence mode enabled.

Note

Enabling accounting mode has no negative impact on the GPU performance.

Functions#

nvmlReturn_t nvmlDeviceClearAccountingPids(nvmlDevice_t device)

Clears accounting information about all processes that have already terminated.

nvmlReturn_t nvmlDeviceGetAccountingBufferSize(nvmlDevice_t device, unsigned int *bufferSize)

Returns the number of processes that the circular buffer with accounting pids can hold.

nvmlReturn_t nvmlDeviceGetAccountingMode(nvmlDevice_t device, nvmlEnableState_t *mode)

Queries the state of per process accounting mode.

nvmlReturn_t nvmlDeviceGetAccountingPids(nvmlDevice_t device, unsigned int *count, unsigned int *pids)

Queries list of processes that can be queried for accounting stats.

nvmlReturn_t nvmlDeviceGetAccountingStats(nvmlDevice_t device, unsigned int pid, nvmlAccountingStats_t *stats)

Queries process's accounting stats.

nvmlReturn_t nvmlDeviceGetAccountingStats_v2(nvmlDevice_t device, nvmlAccountingStats_v2_t *stats)

Queries process's accounting stats (v2).

nvmlReturn_t nvmlDeviceSetAccountingMode(nvmlDevice_t device, nvmlEnableState_t mode)

Enables or disables per process accounting.

Structs#

nvmlAccountingStats_t

Describes accounting statistics of a process.

nvmlAccountingStats_v2_t

Describes accounting statistics (v2) of a process.

Functions#

nvmlReturn_t nvmlDeviceClearAccountingPids(nvmlDevice_t device)#

Clears accounting information about all processes that have already terminated.

For Kepler or newer fully supported devices. Requires root/admin permissions.

See nvmlDeviceGetAccountingMode See nvmlDeviceGetAccountingStats See nvmlDeviceSetAccountingMode

Parameters:

device – The identifier of the target device

Returns:

nvmlReturn_t nvmlDeviceGetAccountingBufferSize(
nvmlDevice_t device,
unsigned int *bufferSize,
)#

Returns the number of processes that the circular buffer with accounting pids can hold.

For Kepler or newer fully supported devices.

This is the maximum number of processes that accounting information will be stored for before information about oldest processes will get overwritten by information about new processes.

See also

nvmlDeviceGetAccountingStats

See also

nvmlDeviceGetAccountingPids

Parameters:

  • device – The identifier of the target device

  • bufferSize – Reference in which to provide the size (in number of elements) of the circular buffer for accounting stats.

Returns:

nvmlReturn_t nvmlDeviceGetAccountingMode(
nvmlDevice_t device,
nvmlEnableState_t *mode,
)#

Queries the state of per process accounting mode.

For Kepler or newer fully supported devices.

See nvmlDeviceGetAccountingStats for more details. See nvmlDeviceSetAccountingMode

Parameters:

  • device – The identifier of the target device

  • mode – Reference in which to return the current accounting mode

Returns:

nvmlReturn_t nvmlDeviceGetAccountingPids(
nvmlDevice_t device,
unsigned int *count,
unsigned int *pids,
)#

Queries list of processes that can be queried for accounting stats.

The list of processes returned can be in running or terminated state.

For Kepler or newer fully supported devices.

To query the number of processes under Accounting Mode, call this function with *count = 0 and pids=NULL. The return code will be NVML_ERROR_INSUFFICIENT_SIZE with an updated count value indicating the number of processes.

For more details see nvmlDeviceGetAccountingStats.

See also

nvmlDeviceGetAccountingBufferSize

Note

In case of PID collision some processes might not be accessible before the circular buffer is full.

Parameters:

  • device – The identifier of the target device

  • count – Reference in which to provide the pids array size, and to return the number of elements ready to be queried

  • pids – Reference in which to return list of process ids

Returns:

nvmlReturn_t nvmlDeviceGetAccountingStats(
nvmlDevice_t device,
unsigned int pid,
nvmlAccountingStats_t *stats,
)#

Queries process’s accounting stats.

For Kepler or newer fully supported devices.

Accounting stats capture GPU utilization and other statistics across the lifetime of a process. Accounting stats can be queried during life time of the process and after its termination. The time field in nvmlAccountingStats_t is reported as 0 during the lifetime of the process and updated to actual running time after its termination. Accounting stats are kept in a circular buffer, newly created processes overwrite information about old processes.

See nvmlAccountingStats_t for description of each returned metric. List of processes that can be queried can be retrieved from nvmlDeviceGetAccountingPids.

See also

nvmlDeviceGetAccountingBufferSize

Note

Accounting Mode needs to be on. See nvmlDeviceGetAccountingMode.

Note

Only compute and graphics applications stats can be queried. Monitoring applications stats can’t be queried since they don’t contribute to GPU utilization.

Note

In case of pid collision stats of only the latest process (that terminated last) will be reported

Warning

On Kepler devices per process statistics are accurate only if there’s one process running on a GPU.

Parameters:

  • device – The identifier of the target device

  • pid – Process Id of the target process to query stats for

  • stats – Reference in which to return the process’s accounting stats

Returns:

nvmlReturn_t nvmlDeviceGetAccountingStats_v2(
nvmlDevice_t device,
nvmlAccountingStats_v2_t *stats,
)#

Queries process’s accounting stats (v2).

For Kepler or newer fully supported devices.

Accounting stats (v2) capture GPU utilization and other statistics across the lifetime of a process. Accounting stats (v2) can be queried during life time of the process and after its termination. The time field in nvmlAccountingStats_v2_t is reported as 0 during the lifetime of the process and updated to actual running time after its termination. Accounting stats (v2) are kept in a circular buffer, newly created processes overwrite information about old processes.

See nvmlAccountingStats_v2_t for description of each returned metric. List of processes that can be queried can be retrieved from nvmlDeviceGetAccountingPids.

See also

nvmlDeviceGetAccountingBufferSize

Note

Accounting Mode needs to be on. See nvmlDeviceGetAccountingMode.

Note

Only compute and graphics applications stats can be queried. Monitoring applications stats can’t be queried since they don’t contribute to GPU utilization.

Note

In case of pid collision stats of only the latest process (that terminated last) will be reported

Warning

On Kepler devices per process statistics are accurate only if there’s one process running on a GPU.

Parameters:

  • device – The identifier of the target device

  • stats – Reference in which to return the process’s accounting stats (v2)

Returns:

nvmlReturn_t nvmlDeviceSetAccountingMode(
nvmlDevice_t device,
nvmlEnableState_t mode,
)#

Enables or disables per process accounting.

For Kepler or newer fully supported devices. Requires root/admin permissions.

See nvmlDeviceGetAccountingMode See nvmlDeviceGetAccountingStats See nvmlDeviceClearAccountingPids

Note

This setting is not persistent and will default to disabled after driver unloads. Enable persistence mode to be sure the setting doesn’t switch off to disabled.

Note

Enabling accounting mode has no negative impact on the GPU performance.

Note

Disabling accounting clears all accounting pids information.

Note

On MIG-enabled GPUs, accounting mode would be set to DISABLED and changing it is not supported.

Parameters:

  • device – The identifier of the target device

  • mode – The target accounting mode

Returns: