2.6. CUPTI Metric API

Functions, types, and enums that implement the CUPTI Metric API.

Classes

union 
A metric value.

Typedefs

typedef uint32_t  CUpti_MetricID
ID for a metric.

Enumerations

enum CUpti_MetricAttribute
Metric attributes.
enum CUpti_MetricCategory
A metric category.
enum CUpti_MetricEvaluationMode
A metric evaluation mode.
enum CUpti_MetricPropertyDeviceClass
Device class.
enum CUpti_MetricPropertyID
Metric device properties.
enum CUpti_MetricValueKind
Kinds of metric values.
enum CUpti_MetricValueUtilizationLevel
Enumeration of utilization levels for metrics values of kind CUPTI_METRIC_VALUE_KIND_UTILIZATION_LEVEL. Utilization values can vary from IDLE (0) to MAX (10) but the enumeration only provides specific names for a few values.

Functions

CUptiResult cuptiDeviceEnumMetrics ( CUdevice device, size_t* arraySizeBytes, CUpti_MetricID* metricArray )
Get the metrics for a device.
CUptiResult cuptiDeviceGetNumMetrics ( CUdevice device, uint32_t* numMetrics )
Get the number of metrics for a device.
CUptiResult cuptiEnumMetrics ( size_t* arraySizeBytes, CUpti_MetricID* metricArray )
Get all the metrics available on any device.
CUptiResult cuptiGetNumMetrics ( uint32_t* numMetrics )
Get the total number of metrics available on any device.
CUptiResult cuptiMetricCreateEventGroupSets ( CUcontext context, size_t metricIdArraySizeBytes, CUpti_MetricID* metricIdArray, CUpti_EventGroupSets** eventGroupPasses )
For a set of metrics, get the grouping that indicates the number of passes and the event groups necessary to collect the events required for those metrics.
CUptiResult cuptiMetricEnumEvents ( CUpti_MetricID metric, size_t* eventIdArraySizeBytes, CUpti_EventID* eventIdArray )
Get the events required to calculating a metric.
CUptiResult cuptiMetricEnumProperties ( CUpti_MetricID metric, size_t* propIdArraySizeBytes, CUpti_MetricPropertyID* propIdArray )
Get the properties required to calculating a metric.
CUptiResult cuptiMetricGetAttribute ( CUpti_MetricID metric, CUpti_MetricAttribute attrib, size_t* valueSize, void* value )
Get a metric attribute.
CUptiResult cuptiMetricGetIdFromName ( CUdevice device, const char* metricName, CUpti_MetricID* metric )
Find an metric by name.
CUptiResult cuptiMetricGetNumEvents ( CUpti_MetricID metric, uint32_t* numEvents )
Get number of events required to calculate a metric.
CUptiResult cuptiMetricGetNumProperties ( CUpti_MetricID metric, uint32_t* numProp )
Get number of properties required to calculate a metric.
CUptiResult cuptiMetricGetRequiredEventGroupSets ( CUcontext context, CUpti_MetricID metric, CUpti_EventGroupSets** eventGroupSets )
For a metric get the groups of events that must be collected in the same pass.
CUptiResult cuptiMetricGetValue ( CUdevice device, CUpti_MetricID metric, size_t eventIdArraySizeBytes, CUpti_EventID* eventIdArray, size_t eventValueArraySizeBytes, uint64_t* eventValueArray, uint64_t timeDuration, CUpti_MetricValue* metricValue )
Calculate the value for a metric.
CUptiResult cuptiMetricGetValue2 ( CUpti_MetricID metric, size_t eventIdArraySizeBytes, CUpti_EventID* eventIdArray, size_t eventValueArraySizeBytes, uint64_t* eventValueArray, size_t propIdArraySizeBytes, CUpti_MetricPropertyID* propIdArray, size_t propValueArraySizeBytes, uint64_t* propValueArray, CUpti_MetricValue* metricValue )
Calculate the value for a metric.

Typedefs

typedef uint32_t CUpti_MetricID

ID for a metric. A metric provides a measure of some aspect of the device.

Enumerations

enum CUpti_MetricAttribute

Metric attributes describe properties of a metric. These attributes can be read using cuptiMetricGetAttribute.

Values
CUPTI_METRIC_ATTR_NAME = 0
Metric name. Value is a null terminated const c-string.
CUPTI_METRIC_ATTR_SHORT_DESCRIPTION = 1
Short description of metric. Value is a null terminated const c-string.
CUPTI_METRIC_ATTR_LONG_DESCRIPTION = 2
Long description of metric. Value is a null terminated const c-string.
CUPTI_METRIC_ATTR_CATEGORY = 3
Category of the metric. Value is of type CUpti_MetricCategory.
CUPTI_METRIC_ATTR_VALUE_KIND = 4
Value type of the metric. Value is of type CUpti_MetricValueKind.
CUPTI_METRIC_ATTR_EVALUATION_MODE = 5
Metric evaluation mode. Value is of type CUpti_MetricEvaluationMode.
CUPTI_METRIC_ATTR_FORCE_INT = 0x7fffffff
enum CUpti_MetricCategory

Each metric is assigned to a category that represents the general type of the metric. A metric's category is accessed using cuptiMetricGetAttribute and the CUPTI_METRIC_ATTR_CATEGORY attribute.

Values
CUPTI_METRIC_CATEGORY_MEMORY = 0
A memory related metric.
CUPTI_METRIC_CATEGORY_INSTRUCTION = 1
An instruction related metric.
CUPTI_METRIC_CATEGORY_MULTIPROCESSOR = 2
A multiprocessor related metric.
CUPTI_METRIC_CATEGORY_CACHE = 3
A cache related metric.
CUPTI_METRIC_CATEGORY_TEXTURE = 4
A texture related metric.
CUPTI_METRIC_CATEGORY_NVLINK = 5
A Nvlink related metric.
CUPTI_METRIC_CATEGORY_FORCE_INT = 0x7fffffff
enum CUpti_MetricEvaluationMode

A metric can be evaluated per hardware instance to know the load balancing across instances of a domain or the metric can be evaluated in aggregate mode when the events involved in metric evaluation are from different event domains. It might be possible to evaluate some metrics in both modes for convenience. A metric's evaluation mode is accessed using CUpti_MetricEvaluationMode and the CUPTI_METRIC_ATTR_EVALUATION_MODE attribute.

Values
CUPTI_METRIC_EVALUATION_MODE_PER_INSTANCE = 1
If this bit is set, the metric can be profiled for each instance of the domain. The event values passed to cuptiMetricGetValue can contain values for one instance of the domain. And cuptiMetricGetValue can be called for each instance.
CUPTI_METRIC_EVALUATION_MODE_AGGREGATE = 1<<1
If this bit is set, the metric can be profiled over all instances. The event values passed to cuptiMetricGetValue can be aggregated values of events for all instances of the domain.
CUPTI_METRIC_EVALUATION_MODE_FORCE_INT = 0x7fffffff
enum CUpti_MetricPropertyDeviceClass

Enumeration of device classes for metric property CUPTI_METRIC_PROPERTY_DEVICE_CLASS.

Values
CUPTI_METRIC_PROPERTY_DEVICE_CLASS_TESLA = 0
CUPTI_METRIC_PROPERTY_DEVICE_CLASS_QUADRO = 1
CUPTI_METRIC_PROPERTY_DEVICE_CLASS_GEFORCE = 2
CUPTI_METRIC_PROPERTY_DEVICE_CLASS_TEGRA = 3
enum CUpti_MetricPropertyID

Metric device properties describe device properties which are needed for a metric. Some of these properties can be collected using cuDeviceGetAttribute.

Values
CUPTI_METRIC_PROPERTY_MULTIPROCESSOR_COUNT
CUPTI_METRIC_PROPERTY_WARPS_PER_MULTIPROCESSOR
CUPTI_METRIC_PROPERTY_KERNEL_GPU_TIME
CUPTI_METRIC_PROPERTY_CLOCK_RATE
CUPTI_METRIC_PROPERTY_FRAME_BUFFER_COUNT
CUPTI_METRIC_PROPERTY_GLOBAL_MEMORY_BANDWIDTH
CUPTI_METRIC_PROPERTY_PCIE_LINK_RATE
CUPTI_METRIC_PROPERTY_PCIE_LINK_WIDTH
CUPTI_METRIC_PROPERTY_PCIE_GEN
CUPTI_METRIC_PROPERTY_DEVICE_CLASS
CUPTI_METRIC_PROPERTY_FLOP_SP_PER_CYCLE
CUPTI_METRIC_PROPERTY_FLOP_DP_PER_CYCLE
CUPTI_METRIC_PROPERTY_L2_UNITS
CUPTI_METRIC_PROPERTY_ECC_ENABLED
CUPTI_METRIC_PROPERTY_FLOP_HP_PER_CYCLE
CUPTI_METRIC_PROPERTY_GPU_CPU_NVLINK_BANDWIDTH
enum CUpti_MetricValueKind

Metric values can be one of several different kinds. Corresponding to each kind is a member of the CUpti_MetricValue union. The metric value returned by cuptiMetricGetValue should be accessed using the appropriate member of that union based on its value kind.

Values
CUPTI_METRIC_VALUE_KIND_DOUBLE = 0
The metric value is a 64-bit double.
CUPTI_METRIC_VALUE_KIND_UINT64 = 1
The metric value is a 64-bit unsigned integer.
CUPTI_METRIC_VALUE_KIND_PERCENT = 2
The metric value is a percentage represented by a 64-bit double. For example, 57.5% is represented by the value 57.5.
CUPTI_METRIC_VALUE_KIND_THROUGHPUT = 3
The metric value is a throughput represented by a 64-bit integer. The unit for throughput values is bytes/second.
CUPTI_METRIC_VALUE_KIND_INT64 = 4
The metric value is a 64-bit signed integer.
CUPTI_METRIC_VALUE_KIND_UTILIZATION_LEVEL = 5
The metric value is a utilization level, as represented by CUpti_MetricValueUtilizationLevel.
CUPTI_METRIC_VALUE_KIND_FORCE_INT = 0x7fffffff
enum CUpti_MetricValueUtilizationLevel

Values
CUPTI_METRIC_VALUE_UTILIZATION_IDLE = 0
CUPTI_METRIC_VALUE_UTILIZATION_LOW = 2
CUPTI_METRIC_VALUE_UTILIZATION_MID = 5
CUPTI_METRIC_VALUE_UTILIZATION_HIGH = 8
CUPTI_METRIC_VALUE_UTILIZATION_MAX = 10
CUPTI_METRIC_VALUE_UTILIZATION_FORCE_INT = 0x7fffffff

Functions

CUptiResult cuptiDeviceEnumMetrics ( CUdevice device, size_t* arraySizeBytes, CUpti_MetricID* metricArray )
Get the metrics for a device.
Parameters
device
The CUDA device
arraySizeBytes
The size of metricArray in bytes, and returns the number of bytes written to metricArray
metricArray
Returns the IDs of the metrics for the device
Returns

  • CUPTI_SUCCESS

  • CUPTI_ERROR_NOT_INITIALIZED

  • CUPTI_ERROR_INVALID_DEVICE

  • CUPTI_ERROR_INVALID_PARAMETER

    if arraySizeBytes or metricArray are NULL

Description

Returns the metric IDs in metricArray for a device. The size of the metricArray buffer is given by *arraySizeBytes. The size of the metricArray buffer must be at least numMetrics * sizeof(CUpti_MetricID) or else all metric IDs will not be returned. The value returned in *arraySizeBytes contains the number of bytes returned in metricArray.

CUptiResult cuptiDeviceGetNumMetrics ( CUdevice device, uint32_t* numMetrics )
Get the number of metrics for a device.
Parameters
device
The CUDA device
numMetrics
Returns the number of metrics available for the device
Returns

  • CUPTI_SUCCESS

  • CUPTI_ERROR_NOT_INITIALIZED

  • CUPTI_ERROR_INVALID_DEVICE

  • CUPTI_ERROR_INVALID_PARAMETER

    if numMetrics is NULL

Description

Returns the number of metrics available for a device.

CUptiResult cuptiEnumMetrics ( size_t* arraySizeBytes, CUpti_MetricID* metricArray )
Get all the metrics available on any device.
Parameters
arraySizeBytes
The size of metricArray in bytes, and returns the number of bytes written to metricArray
metricArray
Returns the IDs of the metrics
Returns

  • CUPTI_SUCCESS

  • CUPTI_ERROR_INVALID_PARAMETER

    if arraySizeBytes or metricArray are NULL

Description

Returns the metric IDs in metricArray for all CUDA-capable devices. The size of the metricArray buffer is given by *arraySizeBytes. The size of the metricArray buffer must be at least numMetrics * sizeof(CUpti_MetricID) or all metric IDs will not be returned. The value returned in *arraySizeBytes contains the number of bytes returned in metricArray.

CUptiResult cuptiGetNumMetrics ( uint32_t* numMetrics )
Get the total number of metrics available on any device.
Parameters
numMetrics
Returns the number of metrics
Returns

  • CUPTI_SUCCESS

  • CUPTI_ERROR_INVALID_PARAMETER

    if numMetrics is NULL

Description

Returns the total number of metrics available on any CUDA-capable devices.

CUptiResult cuptiMetricCreateEventGroupSets ( CUcontext context, size_t metricIdArraySizeBytes, CUpti_MetricID* metricIdArray, CUpti_EventGroupSets** eventGroupPasses )
For a set of metrics, get the grouping that indicates the number of passes and the event groups necessary to collect the events required for those metrics.
Parameters
context
The context for event collection
metricIdArraySizeBytes
Size of the metricIdArray in bytes
metricIdArray
Array of metric IDs
eventGroupPasses
Returns a CUpti_EventGroupSets object that indicates the number of passes required to collect the events and the events to collect on each pass
Returns

  • CUPTI_SUCCESS

  • CUPTI_ERROR_NOT_INITIALIZED

  • CUPTI_ERROR_INVALID_CONTEXT

  • CUPTI_ERROR_INVALID_METRIC_ID

  • CUPTI_ERROR_INVALID_PARAMETER

    if metricIdArray or eventGroupPasses is NULL

Description

For a set of metrics, get the grouping that indicates the number of passes and the event groups necessary to collect the events required for those metrics.

See also:

cuptiEventGroupSetsCreate for details on event group set creation.

CUptiResult cuptiMetricEnumEvents ( CUpti_MetricID metric, size_t* eventIdArraySizeBytes, CUpti_EventID* eventIdArray )
Get the events required to calculating a metric.
Parameters
metric
ID of the metric
eventIdArraySizeBytes
The size of eventIdArray in bytes, and returns the number of bytes written to eventIdArray
eventIdArray
Returns the IDs of the events required to calculate metric
Returns

  • CUPTI_SUCCESS

  • CUPTI_ERROR_NOT_INITIALIZED

  • CUPTI_ERROR_INVALID_METRIC_ID

  • CUPTI_ERROR_INVALID_PARAMETER

    if eventIdArraySizeBytes or eventIdArray are NULL.

Description

Gets the event IDs in eventIdArray required to calculate a metric. The size of the eventIdArray buffer is given by *eventIdArraySizeBytes and must be at least numEvents * sizeof(CUpti_EventID) or all events will not be returned. The value returned in *eventIdArraySizeBytes contains the number of bytes returned in eventIdArray.

CUptiResult cuptiMetricEnumProperties ( CUpti_MetricID metric, size_t* propIdArraySizeBytes, CUpti_MetricPropertyID* propIdArray )
Get the properties required to calculating a metric.
Parameters
metric
ID of the metric
propIdArraySizeBytes
The size of propIdArray in bytes, and returns the number of bytes written to propIdArray
propIdArray
Returns the IDs of the properties required to calculate metric
Returns

  • CUPTI_SUCCESS

  • CUPTI_ERROR_NOT_INITIALIZED

  • CUPTI_ERROR_INVALID_METRIC_ID

  • CUPTI_ERROR_INVALID_PARAMETER

    if propIdArraySizeBytes or propIdArray are NULL.

Description

Gets the property IDs in propIdArray required to calculate a metric. The size of the propIdArray buffer is given by *propIdArraySizeBytes and must be at least numProp * sizeof(CUpti_DeviceAttribute) or all properties will not be returned. The value returned in *propIdArraySizeBytes contains the number of bytes returned in propIdArray.

CUptiResult cuptiMetricGetAttribute ( CUpti_MetricID metric, CUpti_MetricAttribute attrib, size_t* valueSize, void* value )
Get a metric attribute.
Parameters
metric
ID of the metric
attrib
The metric attribute to read
valueSize
The size of the value buffer in bytes, and returns the number of bytes written to value
value
Returns the attribute's value
Returns

  • CUPTI_SUCCESS

  • CUPTI_ERROR_NOT_INITIALIZED

  • CUPTI_ERROR_INVALID_METRIC_ID

  • CUPTI_ERROR_INVALID_PARAMETER

    if valueSize or value is NULL, or if attrib is not a metric attribute

  • CUPTI_ERROR_PARAMETER_SIZE_NOT_SUFFICIENT

    For non-c-string attribute values, indicates that the value buffer is too small to hold the attribute value.

Description

Returns a metric attribute in *value. The size of the value buffer is given by *valueSize. The value returned in *valueSize contains the number of bytes returned in value.

If the attribute value is a c-string that is longer than *valueSize, then only the first *valueSize characters will be returned and there will be no terminating null byte.

CUptiResult cuptiMetricGetIdFromName ( CUdevice device, const char* metricName, CUpti_MetricID* metric )
Find an metric by name.
Parameters
device
The CUDA device
metricName
The name of metric to find
metric
Returns the ID of the found metric or undefined if unable to find the metric
Returns

  • CUPTI_SUCCESS

  • CUPTI_ERROR_NOT_INITIALIZED

  • CUPTI_ERROR_INVALID_DEVICE

  • CUPTI_ERROR_INVALID_METRIC_NAME

    if unable to find a metric with name metricName. In this case *metric is undefined

  • CUPTI_ERROR_INVALID_PARAMETER

    if metricName or metric are NULL.

Description

Find a metric by name and return the metric ID in *metric.

CUptiResult cuptiMetricGetNumEvents ( CUpti_MetricID metric, uint32_t* numEvents )
Get number of events required to calculate a metric.
Parameters
metric
ID of the metric
numEvents
Returns the number of events required for the metric
Returns

  • CUPTI_SUCCESS

  • CUPTI_ERROR_NOT_INITIALIZED

  • CUPTI_ERROR_INVALID_METRIC_ID

  • CUPTI_ERROR_INVALID_PARAMETER

    if numEvents is NULL

Description

Returns the number of events in numEvents that are required to calculate a metric.

CUptiResult cuptiMetricGetNumProperties ( CUpti_MetricID metric, uint32_t* numProp )
Get number of properties required to calculate a metric.
Parameters
metric
ID of the metric
numProp
Returns the number of properties required for the metric
Returns

  • CUPTI_SUCCESS

  • CUPTI_ERROR_NOT_INITIALIZED

  • CUPTI_ERROR_INVALID_METRIC_ID

  • CUPTI_ERROR_INVALID_PARAMETER

    if numProp is NULL

Description

Returns the number of properties in numProp that are required to calculate a metric.

CUptiResult cuptiMetricGetRequiredEventGroupSets ( CUcontext context, CUpti_MetricID metric, CUpti_EventGroupSets** eventGroupSets )
For a metric get the groups of events that must be collected in the same pass.
Parameters
context
The context for event collection
metric
The metric ID
eventGroupSets
Returns a CUpti_EventGroupSets object that indicates the events that must be collected in the same pass to ensure the metric is calculated correctly. Returns NULL if no grouping is required for metric
Returns

  • CUPTI_SUCCESS

  • CUPTI_ERROR_NOT_INITIALIZED

  • CUPTI_ERROR_INVALID_METRIC_ID

Description

For a metric get the groups of events that must be collected in the same pass to ensure that the metric is calculated correctly. If the events are not collected as specified then the metric value may be inaccurate.

The function returns NULL if a metric does not have any required event group. In this case the events needed for the metric can be grouped in any manner for collection.

CUptiResult cuptiMetricGetValue ( CUdevice device, CUpti_MetricID metric, size_t eventIdArraySizeBytes, CUpti_EventID* eventIdArray, size_t eventValueArraySizeBytes, uint64_t* eventValueArray, uint64_t timeDuration, CUpti_MetricValue* metricValue )
Calculate the value for a metric.
Parameters
device
The CUDA device that the metric is being calculated for
metric
The metric ID
eventIdArraySizeBytes
The size of eventIdArray in bytes
eventIdArray
The event IDs required to calculate metric
eventValueArraySizeBytes
The size of eventValueArray in bytes
eventValueArray
The normalized event values required to calculate metric. The values must be order to match the order of events in eventIdArray
timeDuration
The duration over which the events were collected, in ns
metricValue
Returns the value for the metric
Returns

  • CUPTI_SUCCESS

  • CUPTI_ERROR_NOT_INITIALIZED

  • CUPTI_ERROR_INVALID_METRIC_ID

  • CUPTI_ERROR_INVALID_OPERATION

  • CUPTI_ERROR_PARAMETER_SIZE_NOT_SUFFICIENT

    if the eventIdArray does not contain all the events needed for metric

  • CUPTI_ERROR_INVALID_EVENT_VALUE

    if any of the event values required for the metric is CUPTI_EVENT_OVERFLOW

  • CUPTI_ERROR_INVALID_METRIC_VALUE

    if the computed metric value cannot be represented in the metric's value type. For example, if the metric value type is unsigned and the computed metric value is negative

  • CUPTI_ERROR_INVALID_PARAMETER

    if metricValue, eventIdArray or eventValueArray is NULL

Description

Use the events collected for a metric to calculate the metric value. Metric value evaluation depends on the evaluation mode CUpti_MetricEvaluationMode that the metric supports. If a metric has evaluation mode as CUPTI_METRIC_EVALUATION_MODE_PER_INSTANCE, then it assumes that the input event value is for one domain instance. If a metric has evaluation mode as CUPTI_METRIC_EVALUATION_MODE_AGGREGATE, it assumes that input event values are normalized to represent all domain instances on a device. For the most accurate metric collection, the events required for the metric should be collected for all profiled domain instances. For example, to collect all instances of an event, set the CUPTI_EVENT_GROUP_ATTR_PROFILE_ALL_DOMAIN_INSTANCES attribute on the group containing the event to 1. The normalized value for the event is then: (sum_event_values * totalInstanceCount) / instanceCount, where sum_event_values is the summation of the event values across all profiled domain instances, totalInstanceCount is obtained from querying CUPTI_EVENT_DOMAIN_ATTR_TOTAL_INSTANCE_COUNT and instanceCount is obtained from querying CUPTI_EVENT_GROUP_ATTR_INSTANCE_COUNT (or CUPTI_EVENT_DOMAIN_ATTR_INSTANCE_COUNT).

CUptiResult cuptiMetricGetValue2 ( CUpti_MetricID metric, size_t eventIdArraySizeBytes, CUpti_EventID* eventIdArray, size_t eventValueArraySizeBytes, uint64_t* eventValueArray, size_t propIdArraySizeBytes, CUpti_MetricPropertyID* propIdArray, size_t propValueArraySizeBytes, uint64_t* propValueArray, CUpti_MetricValue* metricValue )
Calculate the value for a metric.
Parameters
metric
The metric ID
eventIdArraySizeBytes
The size of eventIdArray in bytes
eventIdArray
The event IDs required to calculate metric
eventValueArraySizeBytes
The size of eventValueArray in bytes
eventValueArray
The normalized event values required to calculate metric. The values must be order to match the order of events in eventIdArray
propIdArraySizeBytes
The size of propIdArray in bytes
propIdArray
The metric property IDs required to calculate metric
propValueArraySizeBytes
The size of propValueArray in bytes
propValueArray
The metric property values required to calculate metric. The values must be order to match the order of metric properties in propIdArray
metricValue
Returns the value for the metric
Returns

  • CUPTI_SUCCESS

  • CUPTI_ERROR_NOT_INITIALIZED

  • CUPTI_ERROR_INVALID_METRIC_ID

  • CUPTI_ERROR_INVALID_OPERATION

  • CUPTI_ERROR_PARAMETER_SIZE_NOT_SUFFICIENT

    if the eventIdArray does not contain all the events needed for metric

  • CUPTI_ERROR_INVALID_EVENT_VALUE

    if any of the event values required for the metric is CUPTI_EVENT_OVERFLOW

  • CUPTI_ERROR_NOT_COMPATIBLE

    if the computed metric value cannot be represented in the metric's value type. For example, if the metric value type is unsigned and the computed metric value is negative

  • CUPTI_ERROR_INVALID_PARAMETER

    if metricValue, eventIdArray or eventValueArray is NULL

Description

Use the events and properties collected for a metric to calculate the metric value. Metric value evaluation depends on the evaluation mode CUpti_MetricEvaluationMode that the metric supports. If a metric has evaluation mode as CUPTI_METRIC_EVALUATION_MODE_PER_INSTANCE, then it assumes that the input event value is for one domain instance. If a metric has evaluation mode as CUPTI_METRIC_EVALUATION_MODE_AGGREGATE, it assumes that input event values are normalized to represent all domain instances on a device. For the most accurate metric collection, the events required for the metric should be collected for all profiled domain instances. For example, to collect all instances of an event, set the CUPTI_EVENT_GROUP_ATTR_PROFILE_ALL_DOMAIN_INSTANCES attribute on the group containing the event to 1. The normalized value for the event is then: (sum_event_values * totalInstanceCount) / instanceCount, where sum_event_values is the summation of the event values across all profiled domain instances, totalInstanceCount is obtained from querying CUPTI_EVENT_DOMAIN_ATTR_TOTAL_INSTANCE_COUNT and instanceCount is obtained from querying CUPTI_EVENT_GROUP_ATTR_INSTANCE_COUNT (or CUPTI_EVENT_DOMAIN_ATTR_INSTANCE_COUNT).