Post-Collection Analysis Guide#
NVIDIA Nsight Systems Analysis guide.
Overview#
Once you have profiled using Nsight Systems there are many options for analyzing the collected data, as well as to output it in various formats. These options are available from the CLI or the GUI.
Statistical Analysis#
Statistical Reports Shipped With Nsight Systems#
The Nsight Systems development team created and maintains a set of report scripts for some of the commonly requested statistical reports. These scripts will be updated to adapt to any changes in SQLite schema or internal data structures.
These scripts are located in the Nsight Systems package in the Target-<architecture>/reports directory. The following standard reports are available:
Note
The ability to display mangled names is a recent addition to the report file format, and requires that the profile data be captured with a recent version of Nsight Systems. Re-exporting an existing report file is not sufficient. If the raw, mangled kernel name data is not available, the default demangled names will be used.
Note
All time values given in nanoseconds by default. If you wish to output the results using a different time unit, use the --timeunit option when running the recipe.
cuda_api_gpu_sum[:nvtx-name][:base|:mangled] – CUDA Summary (API/Kernels/MemOps)#
Arguments
nvtx-name : Optional argument, if given, will prefix the kernel name with the name of the innermost enclosing NVTX range.
base - Optional argument, if given, will cause summary to be over the base name of the kernel, rather than the templated name.
mangled - Optional argument, if given, will cause summary to be over the raw mangled name of the kernel, rather than the templated name.
Note
The ability to display mangled names is a recent addition to the report file format, and requires that the profile data be captured with a recent version of Nsight Systems. Re-exporting an existing report file is not sufficient. If the raw, mangled kernel name data is not available, the default demangled names will be used.
Output: All time values default to nanoseconds
Time : Percentage of “Total Time”
Total Time : Total time used by all executions of this kernel
Instances : Number of executions of this kernel
Avg : Average execution time of this kernel
Med : Median execution time of this kernel
Min : Smallest execution time of this kernel
Max : Largest execution time of this kernel
StdDev : Standard deviation of execution time of this kernel
Category : Category of the operation
Operation : Name of the kernel
This report provides a summary of CUDA API calls, kernels and memory operations, and their execution times. Note that the “Time” column is calculated using a summation of the “Total Time” column, and represents that API call’s, kernel’s, or memory operation’s percent of the execution time of the APIs, kernels and memory operations listed, and not a percentage of the application wall or CPU execution time.
This report combines data from the cuda_api_sum, cuda_gpu_kern_sum, and
cuda_gpu_mem_size_sum reports. It is very similar to profile section of
nvprof --dependency-analysis.
cuda_api_sum – CUDA API Summary#
Arguments - None
Output: All time values default to nanoseconds
Time : Percentage of “Total Time”
Total Time : Total time used by all executions of this function
Num Calls : Number of calls to this function
Avg : Average execution time of this function
Med : Median execution time of this function
Min : Smallest execution time of this function
Max : Largest execution time of this function
StdDev : Standard deviation of the time of this function
Name : Name of the function
This report provides a summary of CUDA API functions and their execution times. Note that the “Time” column is calculated using a summation of the “Total Time” column, and represents that function’s percent of the execution time of the functions listed, and not a percentage of the application wall or CPU execution time.
cuda_api_trace – CUDA API Trace#
Arguments - None
Output: All time values default to nanoseconds
Start : Timestamp when API call was made
Duration : Length of API calls
Name : API function name
Result : Return value of API call
CorrID : Correlation used to map to other CUDA calls
Pid : Process ID that made the call
Tid : Thread ID that made the call
T-Pri : Run priority of call thread
Thread Name : Name of thread that called API function
This report provides a trace record of CUDA API function calls and their execution times.
cuda_gpu_kern_gb_sum[:nvtx-name][:base|:mangled] – CUDA GPU Kernel/Grid/Block Summary#
Arguments
nvtx-name - Optional argument, if given, will prefix the kernel name with the name of the innermost enclosing NVTX range.
base - Optional argument, if given, will cause summary to be over the base name of the kernel, rather than the templated name.
mangled - Optional argument, if given, will cause summary to be over the raw mangled name of the kernel, rather than the templated name.
Note
The ability to display mangled names is a recent addition to the report file format, and requires that the profile data be captured with a recent version of Nsight Systems. Re-exporting an existing report file is not sufficient. If the raw, mangled kernel name data is not available, the default demangled names will be used.
Output: All time values default to nanoseconds
Time : Percentage of “Total Time”
Total Time : Total time used by all executions of this kernel
Instances : Number of calls to this kernel
Avg : Average execution time of this kernel
Med : Median execution time of this kernel
Min : Smallest execution time of this kernel
Max : Largest execution time of this kernel
StdDev : Standard deviation of the time of this kernel
GridXYZ : Grid dimensions for kernel launch call
BlockXYZ : Block dimensions for kernel launch call
Name : Name of the kernel
This report provides a summary of CUDA kernels and their execution times. Kernels are sorted by grid dimensions, block dimensions, and kernel name. Note that the “Time” column is calculated using a summation of the “Total Time” column, and represents that kernel’s percent of the execution time of the kernels listed, and not a percentage of the application wall or CPU execution time.
cuda_gpu_kern_sum[:nvtx-name][:base|:mangled] – CUDA GPU Kernel Summary#
Note
In recent versions of Nsight Systems, this report was expanded to include and sort by CUDA grid and block dimensions. This change was made to accommodate developers doing a certain type of optimization work. Unfortunately, this change caused an unexpected burden for developers doing a different type of optimization work. In order to service both use-cases, this report has been returned to the original form, without grid or block information. A new report, called cuda_gpu_kern_gb_sum, has been created that retains the grid and block information.
Arguments
nvtx-name - Optional argument, if given, will prefix the kernel name with the name of the innermost enclosing NVTX range.
base - Optional argument, if given, will cause summary to be over the base name of the kernel, rather than the templated name.
mangled - Optional argument, if given, will cause summary to be over the raw mangled name of the kernel, rather than the templated name.
Note
The ability to display mangled names is a recent addition to the report file format, and requires that the profile data be captured with a recent version of Nsight Systems. Re-exporting an existing report file is not sufficient. If the raw, mangled kernel name data is not available, the default demangled names will be used.
Output: All time values default to nanoseconds
Time : Percentage of “Total Time”
Total Time : Total time used by all executions of this kernel
Instances : Number of calls to this kernel
Avg : Average execution time of this kernel
Med : Median execution time of this kernel
Min : Smallest execution time of this kernel
Max : Largest execution time of this kernel
StdDev : Standard deviation of the time of this kernel
Name : Name of the kernel
This report provides a summary of CUDA kernels and their execution times. Note that the “Time” column is calculated using a summation of the “Total Time” column, and represents that kernel’s percent of the execution time of the kernels listed, and not a percentage of the application wall or CPU execution time.
cuda_gpu_mem_size_sum – CUDA GPU MemOps Summary (by Size)#
Arguments - None
Output:
Total : Total memory utilized by this operation
Count : Number of executions of this operation
Avg : Average memory size of this operation
Med : Median memory size of this operation
Min : Smallest memory size of this operation
Max : Largest memory size of this operation
StdDev : Standard deviation of the memory size of this operation
Operation : Name of the operation
This report provides a summary of GPU memory operations and the amount of memory they utilize.
cuda_gpu_mem_time_sum – CUDA GPU MemOps Summary (by Time)#
Arguments - None
Output: All time values default to nanoseconds
Time : Percentage of “Total Time”
Total Time : Total time used by all executions of this operation
Count : Number of operations to this type
Avg : Average execution time of this operation
Med : Median execution time of this operation
Min : Smallest execution time of this operation
Max : Largest execution time of this operation
StdDev : Standard deviation of execution time of this operation
Operation : Name of the memory operation
This report provides a summary of GPU memory operations and their execution times. Note that the “Time” column is calculated using a summation of the “Total Time” column, and represents that operation’s percent of the execution time of the operations listed, and not a percentage of the application wall or CPU execution time.
cuda_gpu_sum[:nvtx-name][:base|:mangled] – CUDA GPU Summary (Kernels/MemOps)#
Arguments
nvtx-name - Optional argument, if given, will prefix the kernel name with the name of the innermost enclosing NVTX range.
base - Optional argument, if given, will cause summary to be over the base name of the kernel, rather than the templated name.
mangled - Optional argument, if given, will cause summary to be over the raw mangled name of the kernel, rather than the templated name.
Note
The ability to display mangled names is a recent addition to the report file format, and requires that the profile data be captured with a recent version of Nsight Systems. Re-exporting an existing report file is not sufficient. If the raw, mangled kernel name data is not available, the default demangled names will be used.
Output: All time values default to nanoseconds
Time : Percentage of “Total Time”
Total Time : Total time used by all executions of this kernel
Instances : Number of executions of this kernel
Avg : Average execution time of this kernel
Med : Median execution time of this kernel
Min : Smallest execution time of this kernel
Max : Largest execution time of this kernel
StdDev : Standard deviation of execution time of this kernel
Category : Category of the operation
Operation : Name of the kernel
This report provides a summary of CUDA kernels and memory operations, and their execution times. Note that the “Time” column is calculated using a summation of the “Total Time” column, and represents that kernel’s or memory operation’s percent of the execution time of the kernels and memory operations listed, and not a percentage of the application wall or CPU execution time.
This report combines data from the cuda_gpu_kern_sum and
cuda_gpu_mem_time_sum reports. This report is very similar to output of
the command nvprof --print-gpu-summary.
cuda_gpu_trace[:nvtx-name][:base|:mangled] – CUDA GPU Trace#
Arguments
nvtx-name - Optional argument, if given, will prefix the kernel name with the name of the innermost enclosing NVTX range.
base - Optional argument, if given, will display the base name of the kernel, rather than the templated name.
mangled - Optional argument, if given, will display the raw mangled name of the kernel, rather than the templated name.
Note
The ability to display mangled names is a recent addition to the report file format, and requires that the profile data be captured with a recent version of Nsight Systems. Re-exporting an existing report file is not sufficient. If the raw, mangled kernel name data is not available, the default demangled names will be used.
Output: All time values default to nanoseconds
Start : Timestamp of start time
Duration : Length of event
CorrId : Correlation ID
GrdX, GrdY, GrdZ : Grid values
BlkX, BlkY, BlkZ : Block values
Reg/Trd : Registers per thread
StcSMem : Size of Static Shared Memory
DymSMem : Size of Dynamic Shared Memory
Bytes : Size of memory operation
Throughput : Memory throughput
SrcMemKd : Memcpy source memory kind or memset memory kind
DstMemKd : Memcpy destination memory kind
Device : GPU device name and ID
Ctx : Context ID
GreenCtx: Green context ID
Strm : Stream ID
Name : Trace event name
This report displays a trace of CUDA kernels and memory operations. Items are sorted by start time.
cuda_kern_exec_sum[:nvtx-name][:base|:mangled] – CUDA Kernel Launch & Exec Time Summary#
Arguments
nvtx-name - Optional argument, if given, will prefix the kernel name with the name of the innermost enclosing NVTX range.
base - Optional argument, if given, will cause summary to be over the base name of the kernel, rather than the templated name.
mangled - Optional argument, if given, will cause summary to be over the raw mangled name of the kernel, rather than the templated name.
Note
The ability to display mangled names is a recent addition to the report file format, and requires that the profile data be captured with a recent version of Nsight Systems. Re-exporting an existing report file is not sufficient. If the raw, mangled kernel name data is not available, the default demangled names will be used.
Output: All time values default to nanoseconds
PID : Process ID that made kernel launch call
TID : Thread ID that made kernel launch call
DevId : CUDA Device ID that executed kernel (which GPU)
Count : Number of kernel records
QCount : Number of kernel records with positive queue time
Average, Median, Minimum, Maximum, and Standard Deviation for:
TAvg, TMed, TMin, TMax, TStdDev : Total time
AAvg, AMed, AMin, AMax, AStdDev : API time
QAvg, QMed, QMin, QMax, QStdDev : Queue time
KAvg, KMed, KMin, KMax, KStdDev : Kernel time
API Name : Name of CUDA API call used to launch kernel
Kernel Name : Name of CUDA Kernel
This report provides a summary of the launch and execution times of CUDA kernels. The launch and execution is broken down into three phases: “API time,” the execution time of the CUDA API call on the CPU used to launch the kernel; “Queue time,” the time between the launch call and the kernel execution; and “Kernel time,” the kernel execution time on the GPU. The “total time” is not a just sum of the other times, as the phases sometimes overlap. Rather, the total time runs from the start of the API call to end of the API call or the end of the kernel execution, whichever is later.
The reported queue time is measured from the end of the API call to the start of the kernel execution. The actual queue time is slightly longer, as the kernel is enqueue somewhere in the middle of the API call, and not in the final nanosecond of function execution. Due to this delay, it is possible for kernel execution to start before the CUDA launch call returns. In these cases, no queue time will be reported. Only kernel launches with positive queue times are included in the queue average, minimum, maximum, and standard deviation calculations. The “QCount” column indicates how many launches had positive queue times (and how many launches were involved in calculating the queue time statistics). Subtracting “QCount” from “Count” will indicate how many kernels had no queue time.
Be aware that having a queue time is not inherently bad. Queue times indicate that the GPU was busy running other tasks when the new kernel was scheduled for launch. If every kernel launch is immediate, without any queue time, that _may_ indicate an idle GPU with poor utilization. In terms of performance optimization, it should not necessarily be a goal to eliminate queue time.
cuda_kern_exec_trace[:nvtx-name][:base|:mangled] – CUDA Kernel Launch & Exec Time Trace#
Arguments
nvtx-name - Optional argument, if given, will prefix the kernel name with the name of the innermost enclosing NVTX range.
base - Optional argument, if given, will cause summary to be over the base name of the kernel, rather than the templated name.
mangled - Optional argument, if given, will cause summary to be over the raw mangled name of the kernel, rather than the templated name.
Note: the ability to display mangled names is a recent addition to the report file format, and requires that the profile data be captured with a recent version of Nsight Systems. Re-exporting an existing report file is not sufficient. If the raw, mangled kernel name data is not available, the default demangled names will be used.
Output: All time values default to nanoseconds
API Start : Start timestamp of CUDA API launch call
API Dur : Duration of CUDA API launch call
Queue Start : Start timestamp of queue wait time, if it exists
Queue Dur : Duration of queue wait time, if it exists
Kernel Start : Start timestamp of CUDA kernel
Kernel Dur : Duration of CUDA kernel
Total Dur : Duration from API start to kernel end
PID : Process ID that made kernel launch call
TID : Thread ID that made kernel launch call
DevId : CUDA Device ID that executed kernel (which GPU)
API Function : Name of CUDA API call used to launch kernel
GridXYZ : Grid dimensions for kernel launch call
BlockXYZ : Block dimensions for kernel launch call
Kernel Name : Name of CUDA Kernel
This report provides a trace of the launch and execution time of each CUDA kernel. The launch and execution is broken down into three phases: “API time,” the execution time of the CUDA API call on the CPU used to launch the kernel; “Queue time,” the time between the launch call and the kernel execution; and “Kernel time,” the kernel execution time on the GPU. The “total time” is not a just sum of the other times, as the phases sometimes overlap. Rather, the total time runs from the start of the API call to end of the API call or the end of the kernel execution, whichever is later.
The reported queue time is measured from the end of the API call to the start of the kernel execution. The actual queue time is slightly longer, as the kernel is enqueue somewhere in the middle of the API call, and not in the final nanosecond of function execution. Due to this delay, it is possible for kernel execution to start before the CUDA launch call returns. In these cases, no queue times will be reported.
Be aware that having a queue time is not inherently bad. Queue times indicate that the GPU was busy running other tasks when the new kernel was scheduled for launch. If every kernel launch is immediate, without any queue time, that _may_ indicate an idle GPU with poor utilization. In terms of performance optimization, it should not necessarily be a goal to eliminate queue time.
dx11_pix_sum – DX11 PIX Range Summary#
Arguments - None
Output: All time values default to nanoseconds
Time : Percentage of “Total Time”
Total Time : Total time used by all instances of this range
Instances : Number of instances of this range
Avg : Average execution time of this range
Med : Median execution time of this rage
Min : Smallest execution time of this range
Max : Largest execution time of this range
StdDev : Standard deviation of execution time of this range
Range : Name of the range
This report provides a summary of D3D11 PIX CPU debug markers, and their execution times. Note that the “Time” column is calculated using a summation of the “Total Time” column, and represents that range’s percent of the execution time of the ranges listed, and not a percentage of the application wall or CPU execution time.
dx12_gpu_marker_sum – DX12 GPU Command List PIX Ranges Summary#
Arguments - None
Output: All time values default to nanoseconds
Time : Percentage of “Total Time”
Total Time : Total time used by all instances of this range
Instances : Number of instances of this range
Avg : Average execution time of this range
Med : Median execution time of this range
Min : Smallest execution time of this range
Max : Largest execution time of this range
StdDev : Standard deviation of execution time of this range
Range : Name of the range
This report provides a summary of DX12 PIX GPU command list debug markers, and their execution times. Note that the “Time” column is calculated using a summation of the “Total Time” column, and represents that range’s percent of the execution time of the ranges listed, and not a percentage of the application wall or CPU execution time.
dx12_pix_sum – DX12 PIX Range Summary#
Arguments - None
Output: All time values default to nanoseconds
Time : Percentage of “Total Time”
Total Time : Total time used by all instances of this range
Instances : Number of instances of this range
Avg : Average execution time of this range
Med : Median execution time of this range
Min : Smallest execution time of this range
Max : Largest execution time of this range
StdDev : Standard deviation of execution time of this range
Range : Name of the range
This report provides a summary of D3D12 PIX CPU debug markers, and their execution times. Note that the “Time” column is calculated using a summation of the “Total Time” column, and represents that range’s percent of the execution time of the ranges listed, and not a percentage of the application wall or CPU execution time.
mpi_event_sum – MPI Event Summary#
Arguments - None
Output: All time values default to nanoseconds
Time : Percentage of “Total Time”
Total Time : Total time used by all instances of this event
Instances : Number of instances of this event
Avg : Average execution time of this event
Med : Median execution time of this event
Min : Smallest execution time of this event
Max : Largest execution time of this event
StdDev : Standard deviation of execution time of this event
Source: Original source class of event data
Name : Name of MPI event
This report provides a summary of all recorded MPI events. Note that the “Time” column is calculated using a summation of the “Total Time” column, and represents that event’s percent of the total execution time of the listed events, and not a percentage of the application wall or CPU execution time.
mpi_event_trace – MPI Event Trace#
Arguments - None
Output: All time values default to nanoseconds
Start : Start timestamp of event
End : End timestamp of event
Duration : Duration of event
Event : Name of event type
Pid : Process Id that generated the event
Tid : Thread Id that generated the event
Tag : MPI message tag
Rank : MPI Rank that generated event
PeerRank : Other MPI rank of send or receive type events
RootRank : Root MPI rank for broadcast type events
Size : Size of message for uni-directional operations (send & recv)
CollSendSize : Size of sent message for collective operations
CollRecvSize : Size of received message for collective operations
This report provides a trace record of all recorded MPI events.
Note that MPI_Sendrecv events with different rank, tag, or size values are broken up into two separate report rows, one reporting the send, and one reporting the receive. If only one row exists, the rank, tag, and size can assumed to be the same.
mpi_msg_size_sum – MPI Message Size Summary#
Arguments - None
Output: Message size values are in bytes
Total Message Volume : Aggregated message size from all instances of this API function
Instances : Number of instances of this API function
Avg : Average message size of this API function
Med : Median message size of this API function
Min : Smallest message size of this API function
Max : Largest message size of this API function
StdDev : Standard deviation of message size for this API function
Source : Message source (p2p, coll_send, coll_recv)
Name : Name of the MPI API function
This report provides a message size summary of all collective and point-to-point MPI calls.
Note that for MPI collectives the report presents the sent message with Source
equal to coll_send and the received message with Source equal to coll_recv.
network_congestion[:ticks_threshold=<ticks_per_ms>] – Network Devices Congestion#
Arguments
ticks_threshold=<ticks_per_ms> - Threshold in ticks/ms above which we report congestion. Default is 10000.
Output: All time values default to nanoseconds
Start : Start timestamp of congestion interval
End : End timestamp of congestion interval
Duration : Duration of congestion interval
Send wait rate: Rate of congestion during the interval
GUID : The device GUID
Name : The device name
This report displays congestion events with a high send wait rate. By default, only events with a send wait rate above 10000 ticks/ms are shown, but a custom threshold value can be set.
Each event defines a period of time when the device experienced some level of congestion. The level of congestion is defined by the send wait rate, given in time ticks per millisecond (ticks/ms). The specific duration of a tick is device specific, but can be assumed to be nanoseconds in scale. Congestion is measured by counting the number of ticks during which the port had data to transmit, but no data was sent because of insufficient credits or because of lack of arbitration. The presented value of send wait rate is the amount of ticks counted during an event, normalized over the event’s duration. Higher send wait rate values indicate more congestion.
Because the specific duration of a tick is device dependent, analysis should focus on the relative send wait rates of events generated by the same device. Comparing absolute send wait rates across devices is only meaningful if the time tick duration is known to be similar.
For IB Switch metrics, we do not present the device name, only the GUID.
nvtx_gpu_proj_sum – NVTX GPU Projection Summary#
Arguments - None
Output: All time values default to nanoseconds
Range : Name of the NVTX range
Style : Range style; Start/End or Push/Pop
Total Proj Time: Total projected time used by all instances of this range name
Total Range Time: Total original NVTX range time used by all instances of this range name
Range Instances : Number of instances of this range
Proj Avg : Average projected time for this range
Proj Med : Median projected time for this range
Proj Min : Minimum projected time for this range
Proj Max : Maximum projected time for this range
Proj StdDev : Standard deviation of projected times for this range
Total GPU Ops : Total number of GPU ops
Avg GPU Ops : Average number of GPU ops
Avg Range Lvl : Average range stack depth
Avg Num Child : Average number of children ranges
This report provides a summary of NVTX time ranges projected from the CPU to the GPU. Each NVTX range contains one or more GPU operations. A GPU operation is considered to be “contained” by the NVTX range if the CUDA API call used to launch the operation is within the NVTX range. Only ranges that start and end on the same thread are taken into account.
The projected range will have the start timestamp of the start of the first enclosed GPU operation and the end timestamp of the end of the last enclosed GPU operation. This report then summarizes all the range instances by name and style. Note that in cases when one NVTX range might enclose another, the time of the child(ren) range(s) is not subtracted from the parent range. This is because the projected times may not strictly overlap like the original NVTX range times do. As such, the total projected time of all ranges might exceed the total sampling duration.
nvtx_gpu_proj_trace – NVTX GPU Projection Trace#
Arguments - None
Output: All time values default to nanoseconds
Name : Name of the NVTX range
Projected Start : Projected range start timestamp
Projected Duration : Projected range duration
Orig Start : Original NVTX range start timestamp
Orig Duration : Original NVTX range duration
Style : Range style; Start/End or Push/Pop
PID : Process ID
TID : Thread ID
NumGPUOps : Number of enclosed GPU operations
Lvl : Stack level, starts at 0
NumChild : Number of children ranges
RangeId : Arbitrary ID for range
ParentId : Range ID of the enclosing range
RangeStack : Range IDs that make up the push/pop stack
This report provides a trace of NVTX time ranges projected from the CPU onto the GPU. Each NVTX range contains one or more GPU operations. A GPU operation is considered to be “contained” by an NVTX range if the CUDA API call used to launch the operation is within the NVTX range. Only ranges that start and end on the same thread are taken into account.
The projected range will have the start timestamp of the first enclosed GPU operation and the end timestamp of the last enclosed GPU operation, as well as the stack state and relationship to other NVTX ranges.
nvtx_kern_sum[:base|:mangled] – NVTX Range Kernel Summary#
Arguments
base - Optional argument, if given, will cause summary to be over the base name of the CUDA kernel, rather than the templated name.
mangled - Optional argument, if given, will cause summary to be over the raw mangled name of the kernel, rather than the templated name.
Note
The ability to display mangled names is a recent addition to the report file format, and requires that the profile data be captured with a recent version of Nsight Systems. Re-exporting an existing report file is not sufficient. If the raw, mangled kernel name data is not available, the default demangled names will be used.
Output: All time values default to nanoseconds
NVTX Range : Name of the range
Style : Range style; Start/End or Push/Pop
PID : Process ID for this set of ranges and kernels
TID : Thread ID for this set of ranges and kernels
NVTX Inst : Number of NVTX range instances
Kern Inst : Number of CUDA kernel instances
Total Time : Total time used by all kernel instances of this range
Avg : Average execution time of the kernel
Med : Median execution time of the kernel
Min : Smallest execution time of the kernel
Max : Largest execution time of the kernel
StdDev : Standard deviation of the execution time of the kernel
Kernel Name : Name of the kernel
This report provides a summary of CUDA kernels, grouped by NVTX ranges. To compute this summary, each kernel is matched to one or more containing NVTX range in the same process and thread ID. A kernel is considered to be “contained” by an NVTX range if the CUDA API call used to launch the kernel is within the NVTX range. The actual execution of the kernel may last longer than the NVTX range. A specific kernel instance may be associated with more than one NVTX range if the ranges overlap. For example, if a kernel is launched inside a stack of push/pop ranges, the kernel is considered to be “contained” by all of the ranges on the stack, not just the deepest range. This becomes very confusing if NVTX ranges appear inside other NVTX ranges of the same name.
Once each kernel is associated to one or more NVTX range(s), the list of ranges and kernels grouped by range name, kernel name, and PID/TID. A summary of the kernel instances and their execution times is then computed. The “NVTX Inst” column indicates how many NVTX range instances contained this kernel, while the “Kern Inst” column indicates the number of kernel instances in the summary line.
nvtx_pushpop_sum – NVTX Push/Pop Range Summary#
Arguments - None
Output: All time values given in nanoseconds
Time : Percentage of “Total Time”
Total Time : Total time used by all instances of this range
Instances : Number of instances of this range
Avg : Average execution time of this range
Med : Median execution time of this range
Min : Smallest execution time of this range
Max : Largest execution time of this range
StdDev : Standard deviation of execution time of this range
Range : Name of the range
This report provides a summary of NV Tools Extensions Push/Pop Ranges and their execution times. Note that the “Time” column is calculated using a summation of the “Total Time” column, and represents that range’s percent of the execution time of the ranges listed, and not a percentage of the application wall or CPU execution time.
nvtx_pushpop_trace – NVTX Push/Pop Range Trace#
Arguments - None
Output: All time values default to nanoseconds
Start : Range start timestamp
End : Range end timestamp
Duration : Range duration
DurChild : Duration of all child ranges
DurNonChild : Duration of this range minus child ranges
Name : Name of the NVTX range
PID : Process ID
TID : Thread ID
Lvl : Stack level, starts at 0
NumChild : Number of children ranges
RangeId : Arbitrary ID for range
ParentId : Range ID of the enclosing range
RangeStack : Range IDs that make up the push/pop stack
NameTree : Range name prefixed with level indicator
This report provides a trace of NV Tools Extensions Push/Pop Ranges, their execution time, stack state, and relationship to other push/pop ranges.
nvtx_startend_sum – NVTX Start/End Range Summary#
Arguments - None
Output: All time values default to nanoseconds
Time : Percentage of “Total Time”
Total Time : Total time used by all instances of this range
Instances : Number of instances of this range
Avg : Average execution time of this range
Med : Median execution time of this range
Min : Smallest execution time of this range
Max : Largest execution time of this range
StdDev : Standard deviation of execution time of this range
Range : Name of the range
This report provides a summary of NV Tools Extensions Start/End Ranges and their execution times. Note that the “Time” column is calculated using a summation of the “Total Time” column, and represents that range’s percent of the execution time of the ranges listed, and not a percentage of the application wall or CPU execution time.
nvtx_sum – NVTX Range Summary#
Arguments - None
Output: All time values default to nanoseconds
Time : Percentage of “Total Time”
Total Time : Total time used by all instances of this range
Instances : Number of instances of this range
Avg : Average execution time of this range
Med : Median execution time of this range
Min : Smallest execution time of this range
Max : Largest execution time of this range
StdDev : Standard deviation of execution time of this range
Style : Range style; Start/End or Push/Pop
Range : Name of the range
This report provides a summary of NV Tools Extensions Start/End and Push/Pop Ranges, and their execution times. Note that the “Time” column is calculated using a summation of the “Total Time” column, and represents that range’s percent of the execution time of the ranges listed, and not a percentage of the application wall or CPU execution time.
nvvideo_api_sum – NvVideo API Summary#
Arguments - None
Output: All time values default to nanoseconds
Time : Percentage of “Total Time”
Total Time : Total time used by all executions of this function
Num Calls : Number of calls to this function
Avg : Average execution time of this function
Med : Median execution time of this function
Min : Smallest execution time of this function
Max : Largest execution time of this function
StdDev : Standard deviation of the time of this function
Event Type : Which API this function belongs to
Name : Name of the function
This report provides a summary of NvVideo API functions and their execution times. Note that the “Time” column is calculated using a summation of the “Total Time” column, and represents that function’s percent of the execution time of the functions listed, and not a percentage of the application wall or CPU execution time.
openacc_sum – OpenACC Summary#
Arguments - None
Output: All time values default to nanoseconds
Time : Percentage of “Total Time”
Total Time : Total time used by all executions of event type
Count : Number of event type
Avg : Average execution time of event type
Med : Median execution time of event type
Min : Smallest execution time of event type
Max : Largest execution time of event type
StdDev : Standard deviation of execution time of event type
Name : Name of the event
This report provides a summary of OpenACC events and their execution times. Note that the “Time” column is calculated using a summation of the “Total Time” column, and represents that event type’s percent of the execution time of the events listed, and not a percentage of the application wall or CPU execution time.
opengl_khr_gpu_range_sum – OpenGL KHR_debug GPU Range Summary#
Arguments - None
Output: All time values default to nanoseconds
Time : Percentage of “Total Time”
Total Time : Total time used by all instances of this range
Instances : Number of instances of this range
Avg : Average execution time of this range
Med : Median execution time of this range
Min : Smallest execution time of this range
Max : Largest execution time of this range
StdDev : Standard deviation of execution time of this range
Range : Name of the range
This report provides a summary of OpenGL KHR_debug GPU PUSH/POP debug Ranges, and their execution times. Note that the “Time” column is calculated using a summation of the “Total Time” column, and represents that range’s percent of the execution time of the ranges listed, and not a percentage of the application wall or CPU execution time.
opengl_khr_range_sum – OpenGL KHR_debug Range Summary#
Arguments - None
Output: All time values default to nanoseconds
Time : Percentage of “Total Time”
Total Time : Total time used by all instances of this range
Instances : Number of instances of this range
Avg : Average execution time of this range
Med : Median execution time of this range
Min : Smallest execution time of this range
Max : Largest execution time of this range
StdDev : Standard deviation of execution time of this range
Range : Name of the range
This report provides a summary of OpenGL KHR_debug CPU PUSH/POP debug Ranges, and their execution times. Note that the “Time” column is calculated using a summation of the “Total Time” column, and represents that range’s percent of the execution time of the ranges listed, and not a percentage of the application wall or CPU execution time.
openmp_sum – OpenMP Summary#
Arguments - None
Output: All time values default to nanoseconds
Time : Percentage of “Total Time”
Total Time : Total time used by all executions of event type
Count : Number of event type
Avg : Average execution time of event type
Med : Median execution time of event type
Min : Smallest execution time of event type
Max : Largest execution time of event type
StdDev : Standard deviation of execution time of event type
Name : Name of the event
This report provides a summary of OpenMP events and their execution times. Note that the “Time” column is calculated using a summation of the “Total Time” column, and represents that event type’s percent of the execution time of the events listed, and not a percentage of the application wall or CPU execution time.
osrt_sum – OS Runtime Summary#
Arguments - None
Output: All time values default to nanoseconds
Time : Percentage of “Total Time”
Total Time : Total time used by all executions of this function
Num Calls : Number of calls to this function
Avg : Average execution time of this function
Med : Median execution time of this function
Min : Smallest execution time of this function
Max : Largest execution time of this function
StdDev : Standard deviation of execution time of this function
Name : Name of the function
This report provides a summary of operating system functions and their execution times. Note that the “Time” column is calculated using a summation of the “Total Time” column, and represents that function’s percent of the execution time of the functions listed, and not a percentage of the application wall or CPU execution time.
syscall_sum – Syscall Summary#
Arguments - None
Output: All time values default to nanoseconds
Time : Percentage of “Total Time”
Total Time : Total time used by all executions of this syscall
Num Calls : Number of calls to this syscall
Avg : Average execution time of this syscall
Med : Median execution time of this syscall
Min : Smallest execution time of this syscall
Max : Largest execution time of this syscall
StdDev : Standard deviation of execution time of this syscall
Name : Name of the syscall
This report provides a summary of syscalls and their execution times. Note that the “Time” column is calculated using a summation of the “Total Time” column, and represents that syscall’s percent of the execution time of the syscalls listed, and not a percentage of the application wall or CPU execution time.
um_cpu_page_faults_sum – Unified Memory CPU Page Faults Summary#
Arguments - None
Output:
CPU Page Faults : Number of CPU page faults that occurred CPU Instruction Address : Address of the CPU instruction that caused the CPU page faults
This report provides a summary of CPU page faults for unified memory.
um_sum[:rows=<limit>] – Unified Memory Analysis Summary#
Arguments
rows=<limit> - Maximum number of rows returned by the query. Default is 10.
Output:
Virtual Address : Virtual base address of the page(s) being transferred
HtoD Migration Size : Bytes transferred from Host to Device
DtoH Migration Size : Bytes transferred from Device to Host
CPU Page Faults : Number of CPU page faults that occurred for the virtual base address
GPU Page Faults : Number of GPU page faults that occurred for the virtual base address
Migration Throughput : Bytes transferred per second
This report provides a summary of data migrations for unified memory.
um_total_sum – Unified Memory Totals Summary#
Arguments - None
Output:
Total HtoD Migration Size : Total bytes transferred from host to device
Total DtoH Migration Size : Total bytes transferred from device to host
Total CPU Page Faults : Total number of CPU page faults that occurred
Total GPU Page Faults : Total number of GPU page faults that occurred
Minimum Virtual Address : Minimum value of the virtual address range for the pages transferred
Maximum Virtual Address : Maximum value of the virtual address range for the pages transferred
This report provides a summary of all the page faults for unified memory.
vulkan_api_sum – Vulkan API Summary#
Arguments - None
Output: All time values default to nanoseconds
Time : Percentage of “Total Time”
Total Time : Total time used by all executions of this function
Num Calls: Number of calls to this function
Avg : Average execution time of this function
Med : Median execution time of this function
Min : Smallest execution time of this function
Max : Largest execution time of this function
StdDev : Standard deviation of the time of this function
Name : Name of the function
This report provides a summary of Vulkan API functions and their execution times. Note that the “Time” column is calculated using a summation of the “Total Time” column, and represents that function’s percent of the execution time of the functions listed, and not a percentage of the application wall or CPU execution time.
vulkan_api_trace – Vulkan API Trace#
Arguments - None
Output: All time values default to nanoseconds
Start : Timestamp when API call was made
Duration : Length of API calls
Name : API function name
Event Class : Vulkan trace event type
Context : Trace context ID
CorrID : Correlation used to map to other Vulkan calls
Pid : Process ID that made the call
Tid : Thread ID that made the call
T-Pri : Run priority of call thread
Thread Name : Name of thread that called API function
This report provides a trace record of Vulkan API function calls and their execution times.
vulkan_gpu_marker_sum – Vulkan GPU Range Summary#
Arguments - None
Output: All time values default to nanoseconds
Time : Percentage of “Total Time”
Total Time : Total time used by all instances of this range
Instances : Number of instances of this range
Avg : Average execution time of this range
Med : Median execution time of this range
Min : Smallest execution time of this range
Max : Largest execution time of this range
StdDev : Standard deviation of execution time of this range
Range : Name of the range
This report provides a summary of Vulkan GPU debug markers, and their execution times. Note that the “Time” column is calculated using a summation of the “Total Time” column, and represents that range’s percent of the execution time of the ranges listed, and not a percentage of the application wall or CPU execution time.
vulkan_marker_sum – Vulkan Range Summary#
Arguments - None
Output: All time values default to nanoseconds
Time : Percentage of “Total Time”
Total Time : Total time used by all instances of this range
Instances : Number of instances of this range
Avg : Average execution time of this range
Med : Median execution time of this range
Min : Smallest execution time of this range
Max : Largest execution time of this range
StdDev : Standard deviation of execution time of this range
Range : Name of the range
This report provides a summary of Vulkan debug markers on the CPU, and their execution times. Note that the “Time” column is calculated using a summation of the “Total Time” column, and represents that range’s percent of the execution time of the ranges listed, and not a percentage of the application wall or CPU execution time.
wddm_queue_sum – WDDM Queue Utilization Summary#
Arguments - None
Output: All time values default to nanoseconds
Utilization : Percent of time when queue was not empty
Instances : Number of events
Avg : Average event duration
Med : Median event duration
Min : Minimum event duration
Max : Maximum event duration
StdDev : Standard deviation of event durations
Name : Event name
Q Type : Queue type ID
Q Name : Queue type name
PID : Process ID associated with event
GPU ID : GPU index
Context : WDDM context of queue
Engine : Engine type ID
Node Ord : WDDM node ordinal ID
This report provides a summary of the WDDM queue utilization. The utilization is calculated by comparing the amount of time when the queue had one or more active events to total duration, as defined by the minimum and maximum event time for a given Process ID (regardless of the queue context).
Report Formatters Shipped With Nsight Systems#
The following formats are available in Nsight Systems
Column#
Usage:
column[:nohdr][:nolimit][:nofmt][:<width>[:<width>]...]
Arguments
nohdr: Do not display the header.nolimit: Remove 100 character limit from auto-width columns Note: This can result in extremely wide columns.nofmt: Do not reformat numbers.<width>...: Define the explicit width of one or more columns. If the value.is given, the column will auto-adjust. If a width of 0 is given, the column will not be displayed.
The column formatter presents data in vertical text columns. It is primarily designed to be a human-readable format for displaying data on a console display.
Text data will be left-justified, while numeric data will be right-justified. If the data overflows the available column width, it will be marked with a “…” character, to indicate the data values were clipped. Clipping always occurs on the right-hand side, even for numeric data.
Numbers will be reformatted to make easier to visually scan and understand. This
includes adding thousands-separators. This process requires that the string
representation of the number is converted into its native representation
(integer or floating point) and then converted back into a string representation
to print. This conversion process attempts to preserve elements of number
presentation, such as the number of decimal places, or the use of scientific
notation, but the conversion is not always perfect (the number should always be
the same, but the presentation may not be). To disable the reformatting process,
use the argument nofmt.
If no explicit width is given, the columns auto-adjust their width based off the header size and the first 100 lines of data. This auto-adjustment is limited to a maximum width of 100 characters. To allow larger auto-width columns, pass the initial argument nolimit. If the first 100 lines do not calculate the correct column width, it is suggested that explicit column widths be provided.
Table#
Usage:
table[:nohdr][:nolimit][:nofmt][:<width>[:<width>]...]
Arguments
nohdr: Do not display the header.nolimit: Remove 100 character limit from auto-width columns Note: This can result in extremely wide columns.nofmt: Do not reformat numbers.<width>...: Define the explicit width of one or more columns. If the value.is given, the column will auto-adjust. If a width of 0 is given, the column will not be displayed.
The table formatter presents data in vertical text columns inside text boxes. Other than the lines between columns, it is identical to the column formatter.
CSV#
Usage:
csv[:nohdr]
Arguments
nohdr: Do not display the header.
The csv formatter outputs data as comma-separated values. This format is commonly used for import into other data applications, such as spread-sheets and databases.
There are many different standards for CSV files. Most differences are in how escapes are handled, meaning data values that contain a comma or space.
This CSV formatter will escape commas by surrounding the whole value in double-quotes.
TSV#
Usage:
tsv[:nohdr][:esc]
Arguments
nohdr: Do not display the header.esc: escape tab characters, rather than removing them.
The TSV formatter outputs data as tab-separated values. This format is sometimes used for import into other data applications, such as spreadsheets and databases.
Most TSV import/export systems disallow the tab character in data values. The formatter will normally replace any tab characters with a single space. If the esc argument has been provided, any tab characters will be replaced with the literal characters “t”.
JSON#
Usage:
json
Arguments: no arguments
The JSON formatter outputs data as an array of JSON objects. Each object represents one line of data, and uses the column names as field labels. All objects have the same fields. The formatter attempts to recognize numeric values, as well as JSON keywords, and converts them. Empty values are passed as an empty string (and not nil, or as a missing field).
At this time the formatter does not escape quotes, so if a data value includes double-quotation marks, it will corrupt the JSON file.
HDoc#
hdoc[:title=<title>][:css=<URL>]
Arguments:
title: string for HTML document title.css: URL of CSS document to include.
The HDoc formatter generates a complete, verifiable (mostly), standalone HTML
document. It is designed to be opened in a web browser, or included in a larger
document via an <iframe>.
HTable#
Usage:
htable
Arguments: no arguments
The HTable formatter outputs a raw HTML <table> without any of the surrounding
HTML document. It is designed to be included into a larger HTML document.
Although most web browsers will open and display the document, it is better to
use the HDoc format for this type of use.
Expert Systems Analysis#
The Nsight Systems expert system is a feature aimed at automatic detection of performance optimization opportunities in an application’s profile. It uses a set of predefined rules to determine if the application has known bad patterns.
Using Expert System from the CLI#
usage:
nsys [global-options] analyze [options]
[nsys-rep-or-sqlite-file]
If a .nsys-rep file is given as the input file and there is no .sqlite file with the same name in the same directory, it will be generated.
Note
The Expert System view in the GUI will give you the equivalent command line.
Using Expert System from the GUI#
The Expert System View can be found in the same drop-down as the Events View. If there is no .sqlite file with the same name as the .nsys-rep file in the same directory, it will be generated.
The Expert System View has the following components:
Drop-down to select the rule to be run.
Rule description and advice summary.
CLI command that will give the same result.
Table containing results of running the rule.
Settings button that allows users to specify the rule’s arguments.
A context menu is available to correlate the table entry with the timeline. The options are the same as the Events View:
Zoom to Selected on Timeline (ctrl+double-click)
The highlighting is not supported for rules that do not return an event but rather an arbitrary time range (e.g., GPU utilization rules).
The CLI and GUI share the same rule scripts and messages. There might be some formatting differences between the output table in GUI and CLI.
Expert System Rules#
Rules are scripts that run on the SQLite DB output from Nsight Systems to find common improvable usage patterns.
Each rule has an advice summary with explanation of the problem found and suggestions to address it. Only the top 50 results are displayed by default.
There are currently six rules in the expert system. They are described below. Additional rules will be made available in a future version of Nsight Systems.
CUDA Synchronous Operation Rules#
Asynchronous memcpy with pageable memory
This rule identifies asynchronous memory transfers that end up becoming synchronous if the memory is pageable. This rule is not applicable for Nsight Systems Embedded Platforms Edition
Suggestion: If applicable, use pinned memory instead
Synchronous Memcpy
This rule identifies synchronous memory transfers that block the host.
Suggestion: Use cudaMemcpy*Async APIs instead.
Synchronous Memset
This rule identifies synchronous memset operations that block the host.
Suggestion: Use cudaMemset*Async APIs instead.
Synchronization APIs
This rule identifies synchronization APIs that block the host until all issued CUDA calls are complete.
Suggestions: Avoid excessive use of synchronization. Use asynchronous CUDA event calls, such as cudaStreamWaitEvent and cudaEventSynchronize, to prevent host synchronization.
GPU Low Utilization Rules#
Nsight Systems determines GPU utilization based on API trace data in the collection. Current rules consider CUDA, Vulkan, DX12, and OpenGL API use of the GPU.
GPU Starvation
This rule identifies time ranges where a GPU is idle for longer than 500ms. The threshold is adjustable.
Suggestions: Use CPU sampling data, OS Runtime blocked state backtraces, and/or OS Runtime APIs related to thread synchronization to understand if a sluggish or blocked CPU is causing the gaps. Add NVTX annotations to CPU code to understand the reason behind the gaps.
Notes: For each process, each GPU is examined, and gaps are found within the time range that starts with the beginning of the first GPU operation on that device and ends with the end of the last GPU operation on that device. GPU gaps that cannot be addressed by the user are excluded. This includes:
Profiling overhead in the middle of a GPU gap.
The initial gap in the report that is seen before the first GPU operation.
The final gap that is seen after the last GPU operation.
GPU Low Utilization
This rule identifies time regions with low utilization.
Suggestions: Use CPU sampling data, OS Runtime blocked state backtraces, and/or OS Runtime APIs related to thread synchronization to understand if a sluggish or blocked CPU is causing the gaps. Add NVTX annotations to CPU code to understand the reason behind the gaps.
Notes: For each process, each GPU is examined, and gaps are found within the time range that starts with the beginning of the first GPU operation on that device and ends with the end of the last GPU operation on that device. This time range is then divided into equal chunks, and the GPU utilization is calculated for each chunk. The utilization includes all GPU operations as well as profiling overheads that the user cannot address.
The utilization refers to the “time” utilization and not the “resource” utilization. This rule attempts to find time gaps when the GPU is or isn’t being used, but does not take into account how many GPU resources are being used. Therefore, a single running memcpy is considered the same amount of “utilization” as a huge kernel that takes over all the cores. If multiple operations run concurrently in the same chunk, their utilization will be added up and may exceed 100%.
Chunks with an in-use percentage less than the threshold value are displayed. If consecutive chunks have a low in-use percentage, the individual chunks are coalesced into a single display record, keeping the weighted average of percentages. This is why returned chunks may have different durations.
AI-Assisted Analysis#
Note
AI-Assisted Analysis is a Preview Feature.
Nsight Systems includes an AI skill that helps you use the CLI, run post-collection analysis recipes, and analyze profiling reports. The skill is provided in a bring-your-own (BYO) pack for use with different AI agents.
The skill guides the agent to use product documentation, help from the installed CLI, and profiling data from your reports.
Installing the Skill Pack#
See Installing the AI Skill in the Installation Guide to install the
nsight-systems skill in your agent’s personal skills folder. After installing
the skill, start a new agent session. For example, ask:
In my nsys-rep file, which is the longest CUDA kernel execution?
The Nsight Systems GUI report loading pane shows installation commands with the
resolved skill directory for your platform. nsys --help refers to the
Installation Guide.
What the Skill Pack Covers#
The skill helps with questions about Nsight Systems, its CLI, recipes, and report analysis. It covers CPU and GPU activity, CUDA APIs and kernels, graphics APIs, frame and stutter analysis, OS runtime, NVTX, MPI, and NCCL.
Disclaimer
Agentic AI is inherently non-deterministic: the same prompt can produce different results across runs, and outputs may be incomplete or inaccurate. Additionally, this skill is tested against a limited set of AI models, and results may vary between them and across versions. Treat results as drafts, and verify important findings against the Nsight Systems documentation, live CLI or recipe help, and the profiling report data before relying on them.
Troubleshooting AI Assisted Profiling#
When you use an AI agent to help optimize application performance, it can fall into familiar patterns that lead to misleading measurements or conclusions, even when you use Nsight Systems correctly. The checks below can help you recognize them and guide the agent toward reliable profiling, analysis, and optimization results.
For setup and use of the bundled skill pack, see AI-Assisted Analysis. The checks below can also be used with other agent workflows.
In general it is a good idea to verify important findings against the product documentation, live CLI or recipe help, and profiling report data. Agent output may be incomplete or inaccurate, and results can vary between runs, models, and versions.
Common Analysis Failures#
AI agents can operate profiling tools successfully and still reach an unsupported conclusion. Use the following checks when reviewing an agent’s analysis or defining its workflow. These failures can also occur in human analysis; their frequency depends on the workload, available data, model, and agent environment.
Failure |
How to recognize it |
Prevention and recovery |
|---|---|---|
Unsupported commands or query assumptions |
A proposed option, report, table, or column is absent from the installed version, or a query uses the wrong timestamp or ID model. |
Check live CLI and recipe help. Inspect the exported schema before writing queries. Use documented scripts where appropriate, and resolve errors before interpreting their output. |
Incomplete capture treated as complete evidence |
A required trace source, process, device, or interval is missing. An empty result is interpreted as proof of no activity. |
Check collection completion, diagnostics, enabled sources, and captured scope. Distinguish unavailable data from an observed zero. Recapture the missing activity or state the resulting limitation. |
Unrepresentative workload region |
The analysis describes startup, compilation, or warmup while the stated objective concerns steady-state performance, or vice versa. |
Identify the intended phase, iteration, frame, or NVTX range before analyzing it. Verify that the capture includes that region and enough representative work; recapture when it does not. |
Incorrect units, percentages, or aggregation |
A kernel summary percentage is treated as application elapsed time, or overlapping CPU, GPU, or nested-range durations are added. |
State units, scope, denominator, and aggregation for each result. CUDA kernel-summary percentages refer to summed kernel durations, not application elapsed time. Account for overlap when estimating impact on elapsed time. |
Hotspot mistaken for the limiting bottleneck |
The agent targets the largest summary entry without checking whether that work limits the application’s latency or throughput. |
Examine the relevant timeline, dependencies, CPU work, transfers, synchronization, and communication. Use a focused follow-up measurement to test the hypothesis, then validate application-level performance after a change. |
Correlation mistaken for causation |
A long wait, API call, or GPU gap is named as the cause of a slowdown solely because it occurs during the slow interval. |
Distinguish active execution from waiting and scheduling delay. Trace dependencies back to the work or event that releases the wait. Treat temporal coincidence as a hypothesis until supported by additional evidence. |
Incomparable baseline and candidate |
Reports use different input shapes, workload phases, environments, or collection settings, but the difference is attributed to one optimization. |
Record build and workload provenance for both reports. Identify intentional differences and hold other relevant conditions constant. Repeat the comparison or qualify conclusions when conditions differ. |
Measurement perturbation or noise |
A claimed regression appears only with detailed collection enabled, or a small speedup is comparable to run-to-run variation. |
Validate performance with repeated unprofiled measurements. Reduce collection detail when diagnosing perturbation. When comparing with Nsight Compute, account for replay, serialization, cache control, and clock settings. |
Performance improvement mistaken for correctness |
Faster execution is accepted despite changed outputs, skipped work, an intermittent failure, or validation limited to a few inputs. |
Run workload correctness tests separately from profiling. Use appropriate correctness tools for suspected defects, including supported race checks where applicable. Mark invalid results as unusable evidence and revalidate affected conclusions. |
See Common Analysis Pitfalls and SQLite Schema Pitfalls for additional interpretation and query guidance. The Statistical Reports reference defines report fields and timing semantics.
Failures in Repeated Optimization Workflows#
An agent that repeatedly profiles, changes code, and evaluates results also needs reliable experiment management. The following practices belong in the surrounding agent or benchmark workflow. They are not automatic guarantees provided by the skill pack or profiling report.
An unsuccessful experiment can be a useful result. Distinguish an idea that did not improve performance from a measurement that is invalid or a change that fails correctness checks.
Failure |
How to recognize it |
Prevention and recovery |
|---|---|---|
Bias toward small changes and abandonment of failed ideas |
The search repeatedly tunes parameters while avoiding structural changes, or discards an idea after one unsuccessful implementation. |
Separate idea evaluation from implementation attempts. Preserve unsuccessful experiments and their reasons. Allow incremental work on isolated branches while keeping accepted results subject to correctness and performance validation. |
Serialized exploration or resource contention |
Compute resources sit idle while an agent plans, or simultaneous workers interfere with each other’s benchmark measurements. |
Schedule independent work where useful, but isolate or serialize measurements that compete for shared resources. Record whether contention is part of the intended workload. |
Context decay and premature quitting |
The agent repeats completed work, loses validation requirements, or ends the campaign before its stopping condition is satisfied. |
Keep a concise experiment record outside conversational context. Record the current objective, evidence, pending work, and completion condition. Recheck that record after compaction or agent restart. |
Local maxima and stalls |
Repeated trials improve only the current approach, with little progress toward the application objective or exploration of alternatives. |
Review progress against the stated objective. Preserve starting points for alternative approaches and use explicit restart or backtracking policies. Investigate whether validation or scoring excludes useful intermediate changes. |
Poor time estimates and unreliable task coordination |
Collection or benchmark jobs outlive the agent’s estimate, completion is missed, or background processes remain unaccounted for. |
Track job or session IDs, process ownership, status, and artifact paths. Use supported waiting, timeout, and cancellation mechanisms. Reconcile active jobs before launching replacements; inspect partial artifacts before using them. |
Instruction drift and hidden failures |
The agent relaxes validation to obtain progress, omits failed trials, or disables failing code while presenting the experiment as successful. |
Keep acceptance checks explicit and apply them independently where possible. Record test failures and disabled work. Separate exploratory code from the validated candidate and reject unsupported success claims. |
Reward hacking: output replay |
Repeated benchmark calls appear faster because a previous result is returned instead of performing the required work. |
Test with varied inputs and verify outputs. Define whether caching is allowed by the workload contract. Use changes in trace activity as a reason to investigate, not as proof of cheating or correctness. |
Reward hacking: input specialization or overfitting |
A candidate recognizes benchmark-specific inputs, or improves a subset of shapes while hiding regressions in an aggregate score. |
Evaluate representative and adversarial inputs across required shapes. Review per-case results alongside the aggregate and check that specializations satisfy the intended input contract. |
Reward hacking: contamination |
A campaign intended to evaluate independent discovery reuses existing solutions without documenting their origin. |
Record solution provenance and follow the campaign’s rules on reuse. Use held-out problems or controlled access when independent discovery is required. Reuse may be appropriate in production optimization. |
Reward hacking: measurement tampering |
A candidate changes timers, skips validation, or modifies the scoring path instead of improving the required application behavior. |
Keep benchmark and validation components under independent control where appropriate. Verify that the intended workload actually ran and repeat affected measurements with the trusted evaluation path. |
Polluted experiment records or persistent memory |
Invalid trials retain high scores, later trials build on them, or unsupported conclusions are saved as lessons for future campaigns. |
Mark invalid results in the experiment record and exclude them from selection. Track dependencies and revalidate affected descendants. Retain evidence and scope for reusable conclusions, and withdraw them when their supporting evidence is invalidated. |
Validating an AI-Assisted Analysis#
Use the following procedure to make an analysis reproducible and reviewable:
Define the objective and workload. State the application metric, relevant input cases, measured phase, and correctness requirements. Record the build or source revision and execution environment.
Verify the capture. Confirm completion and examine collection diagnostics. Verify the intended processes, devices, trace sources, and workload region. Record missing data and capture limitations.
Ground each observation. Use the skill pack’s analysis workflow and supported reports or recipes. Retain commands, filters, units, denominators, and references to the supporting trace events or results. Verify options with live help for the installed version.
Test the bottleneck hypothesis. Examine overlap and dependencies. State alternative explanations and identify a targeted measurement that can distinguish them. Collect additional data when required evidence is unavailable.
Validate the change. Run correctness checks and compare application performance under matched conditions without profiling instrumentation. Repeat measurements when variability could explain the result and review per-case behavior as well as aggregate scores.
Preserve the outcome. Record successful and unsuccessful trials. Label invalid measurements, state the reason, and revalidate conclusions or candidates that depend on them. Keep active-job ownership and completion state available across agent restarts.
Note
A profiling trace describes recorded execution. It does not prove application correctness, establish behavior for uncaptured inputs, or by itself distinguish legitimate caching from an invalid shortcut. Use workload tests and independent validation for those questions.
Recording Findings#
Ask the agent to separate observations from hypotheses and verified results. A compact record can use the following fields:
Objective: Application metric, workload cases, and correctness requirements.
Provenance: Build/revision, environment, report path, and collection settings.
Observation: Measured behavior, units, scope, and denominator.
Evidence: Analysis command, filters, and event or result references.
Hypothesis: Suspected cause and plausible alternative explanations.
Next measurement: Evidence needed to distinguish those explanations.
Validation: Correctness outcome and repeated application-level measurements.
Outcome: Improvement, no improvement, correctness failure, or invalid measurement.
Limitations: Missing data, untested inputs, and unresolved questions.
Record an unresolved conclusion as unresolved. A supported statement about missing evidence is more useful than an unsupported diagnosis that later experiments treat as fact.
A few checks for these pitfalls can go a long way toward more successful AI-driven analysis. By helping the agent recognize missing evidence, validate measurements, and learn from each experiment, you can build confidence in its findings and keep your optimization work moving toward your performance goals.
Investigation Methodology#
This section describes an approach for investigating Nsight Systems traces, with or without an AI agent. It explains how to start from a reported symptom, use report data to check what happened, measure how much time each cause adds, and stop when the evidence is strong enough to answer. It is guidance only. Exact facts must come from the report data and the Nsight Systems documentation.
Provenance tagging#
Base every finding on queried data and cite the source: table name, column, thread, metric, and time window, with enough detail for a reader to reproduce the query.
Do not label plain observations. “RenderThread ran for 37 ms” is a direct observation from query data and needs no tag.
Do label causal and explanatory claims, because those are the claims most likely to be wrong:
[INFERRED]: a conclusion derived from data but not directly stated. Use on every “therefore”, “because”, “this caused”, or “this explains” claim.[DOC: <page>]: an interpretation based on a documentation page. Name the page, for example[DOC: WDDM Queues].[BACKGROUND]: general platform knowledge that this trace does not show. It lowers confidence. Use it when you explain why a mechanism works, for example “DxgKrnl serializes GPU context switches [BACKGROUND]”.[ESTIMATED]: a value calculated from incomplete data; state the inputs and the uncertainty.
For important derived values, record each step from the raw data to the value you cite. For a compact record of each finding, see the “Recording Findings” section of Troubleshooting AI Assisted Profiling.
Top-down investigation#
Work from symptoms to causes, never the reverse. If an AI agent produced the analysis, also apply the checks in the “Validating an AI-Assisted Analysis” section of Troubleshooting AI Assisted Profiling.
Classify: establish severity. Quote exact values from data.
Locate: find where in the trace the problem occurs. Pick a time window around the problem, such as the slow frames, and widen it only when the comparison needs it.
Scan: before forming hypotheses, compare the problem window with a clean one across all available data. List every signal that changed, not just the first one you find. For graphics frames, check for long-short frame patterns first; see Stutter Analysis.
Hypothesize: form two or three candidate causes from the scan results. Use domain knowledge to guide queries, not as evidence.
Verify: query data to test each hypothesis. Run at least one check per hypothesis that could prove it wrong. Compare against a clean reference window of equal duration when one is available.
Converge: build the full causal chain: trigger event -> mechanism -> pipeline effect -> user-visible symptom. Back every link with data, or label it with a provenance tag.
Ask “why?” until you reach a cause#
Every observation raises the question “why?” Each answer leads to the next question. Follow the chain until you reach either a generator you can name (a specific function, a lock holder, a wait target, a change in the input data, or an OS scheduling decision) or the limit of the trace data (unresolved symbols, missing CPU samples, or no GPU metrics).
The answers pass through these layers. The examples follow one slow frame on a machine with 32 CPU cores:
Layer |
Type |
Example |
Acceptable to stop here? |
|---|---|---|---|
0 |
Symptom |
“Frame 101 takes 110 ms, about 7x its baseline frame (16 ms)” |
No |
1 |
Phase |
“The Simulation NVTX range takes 95 ms of the 110 ms” |
No |
2 |
Activity |
“GameThread never blocks in the Simulation range; it is either running or waiting for a CPU” |
No |
3 |
Evidence |
“In the Simulation range, GameThread waits 65 ms for a CPU, against 1 ms in the baseline frame” |
Only with a magnitude match |
4 |
Mechanism |
“32 worker threads keep all 32 cores busy and preempt GameThread about 100 times” |
Yes, the target |
5 |
Generator |
“The thread pool starts one worker per core and leaves no core for GameThread” |
Yes, the ideal stopping point |
Do not conclude at Layer 0 (symptom), Layer 1 (phase), or Layer 2 (activity). Layer 3 (evidence) is acceptable only with a magnitude match. Aim for Layer 4 (mechanism), and go on to Layer 5 (generator) when the data allows. If the data runs out before Layer 3, report the cause as unresolved and name the data you need.
Keep notes of the current layer, so it is obvious when you are not yet ready to conclude:
Current layer: 2 (activity), not ready to conclude
Finding: GameThread never blocks in the Simulation range of frame 101
Next question: How long does it wait for a CPU there, and in the baseline frame?
Data needed: GameThread's scheduling events in that range, in both frames
Update the notes each time you move to a new layer.
Drill-down checklist for each signal#
For each signal that changed in the scan, work through these steps before you move on. Check which tables and columns the report has before you write queries.
Which thread? Use per-thread on-CPU time from scheduling events, or per-thread event counts, to find the threads that own the activity. In graphics captures, the Thread Scheduling and Thread Callstacks sections of the gfx_frame_analysis Recipe, and the per-function summaries of the gfx_api_analysis Recipe, answer most of these questions per frame; select the frame with
--pid <PID> --frame <N>.What is that thread doing? Use whichever data source exists in the trace:
CUDA API tables: which runtime or driver calls are longest for that thread?
Graphics API calls (D3D12, Vulkan, OpenGL): which calls dominate, and do calls to one function run back to back?
Sampling callchains: which modules or functions dominate the leaf frames?
GPU metrics: is the GPU busy or starved (GR Active, SMs Active)?
NVTX ranges: which application phase takes longer than usual?
ETW events: paging, shader compilation, or device events?
Scheduling states and block reasons: is the thread running, waiting for a CPU, waiting on a resource, or voluntarily blocked (
UserRequest)?
Which thread is blocked? Find the thread that stalls. Compare its on-CPU, waiting-for-CPU, and blocked time in your window with the clean reference. Read its top block reasons and the process’s OS runtime wait calls. Then find the call it waits in from the callstacks taken as it left the CPU.
The Thread Scheduling times of the gfx_frame_analysis Recipe are upper bounds. A context switch event marks a change of state, not a duration, so the recipe counts each state until the thread’s next event. When a sched-in or sched-out event is missing, an interval runs on to the next event, or to the end of the window, and can include time the thread spent in another state. The recipe’s
thread_schedtable also hasonCpuConfirmedPct,readyConfirmedPct, andblockedConfirmedPctcolumns. Each column gives the percentage of that state’s time that the recorded events confirm. When the percentage is low, use the times to rank threads, not as exact durations. The notebook does not show these columns: read them fromthread_sched.parquetnext to the notebook, or from the--format csvor--format jsonloutput.Per-call or per-count scaling? Compare the average duration per event in your window with the clean reference. If the duration per call grows with concurrency, suspect a serialized resource. If it stays flat, the cost comes from the number of calls.
What triggers it? Name the operation that starts the chain. Candidates include a blocking API call, a synchronous shader compilation, an allocation that pushes VRAM over budget, a CPU overrun, GPU-bound work, or an external process visible in scheduling events.
Magnitude match in milliseconds. Measure how much time the cause added compared with the clean reference, and compare that extra time with the effect delta. Take durations from API call intervals, scheduling intervals, or NVTX ranges; event counts and rates are not a magnitude match. For the durations you can compare, see Magnitude match.
Convergence checks before concluding#
All of these must hold before you treat a finding as complete:
Magnitude match: measure the time the cause added, as in step 6 of the drill-down checklist, and state its share of the effect delta. For example, “In the Simulation range, which is on the critical path, GameThread waited 65 ms for a CPU, against 1 ms in the same range of the baseline frame. The 64 ms excess is 68% of the 94 ms frame delta.”
Temporal ordering: the cause must start before the effect. State timestamps.
Entity attribution: name the specific thread (name and TID) and what it is doing (API call, module from callstack, scheduling state).
Drill past the symptom: the cause you name is not itself a consequence of the problem; see Reversed causation.
When the data allows, also name the generator: an operation a developer can change, such as “PhysX cooking on GameThread during the frame” or “synchronous shader compilation on the render path”. You do not need the generator to conclude. Once the checks above hold, conclude at the deepest layer the data supports, and state what deeper data would add. A good finding with honest uncertainty is more useful than an investigation that never concludes.
Rigor and units#
State the sample size and flag N=1. Present a cause found in one report as likely, not proven, and offer fixes as options.
Rate each cause by the share of the effect delta it explains: more than 50% is a high-confidence cause, 25-50% is a partial cause, and less than 25% is a contributing factor. Check that a part does not exceed the whole: a ratio above 100% needs an explanation (see Magnitude match). Check data quality against a clean reference where one exists. For alternative explanations and other common mistakes, see the “Common Analysis Failures” section of Troubleshooting AI Assisted Profiling.
Report frame time in ms and FPS, durations in ms and percent of frame, and normalize to per-second rates when comparing windows of different duration. State the value and the ratio, not words such as “huge” or “massive”.
Use measured values, and mark estimates as [ESTIMATED]. Cite explicit window boundaries for windowed comparisons, and check that breakdowns sum to their total.
A performance problem often has a primary cause and other factors that make it worse. State the primary cause, then measure each other factor separately.
Stutter Analysis#
This section is a reference for stutter in graphics traces. It covers game stutter (CPU present-to-present timing), display stutter (flip-to-flip timing), how the two relate, how Nsight Systems detects stutter, and the common causes with the checks for each. It applies whether or not you use an AI agent. The step-by-step investigation procedure is in the Investigation Methodology section. For a short definition, see the stutter glossary entry.
When analyzing stutters, always distinguish between game stutters and display stutters. They have different causes, different impacts, and require different data to diagnose.
Data needed#
Many checks on this page need capture options beyond the defaults. On Windows, the WDDM trace has three modes, and each mode records a different set of DxgKrnl events:
Switches |
Records |
Needed for |
|---|---|---|
|
GPU memory allocations, memory transfers, VidMm budget, usage, and commitment changes, and DxgKrnl |
GPU memory activity |
|
Queue, DMA, and paging packets, evictions, paging operations, hardware queues, and the WDDM HW Scheduler rows on a machine that uses hardware scheduling. No DxgKrnl |
GPU busy or idle, GPU context contention, paging and evictions, and the WDDM event rates of the gfx_frame_analysis Recipe |
|
Everything in the previous two rows, plus display flips, VSync interrupts, more VidMm events, and DxgKrnl Profiler events (see Capture overhead in Common stutter causes) |
Display flip timing, and the |
nsys rejects --wddm-additional-events=true unless --wddm-memory-trace=false is also set.
Without Streamline or Reflex markers, the graphics recipes take frames from the graphics API present calls, and without a graphics API trace from DxgKrnl Present events. A capture with --wddm-memory-trace=false has no DxgKrnl Present events, so the recipes use the DXGI Present events instead, which every WDDM trace mode records.
The CPU-side checks need their own switches. Thread Scheduling and CPU Concurrency need CPU context switch tracing. A process-tree trace covers the application’s own threads, which is enough for Thread Scheduling. The threads of other processes, and the whole-machine part of CPU Concurrency, need a system-wide trace. On Windows, use --sample=system-wide, because --sample sets the --cpuctxsw scope. The default process-tree sampling also does this when nsys launches the application, so --cpuctxsw=system-wide alone is not enough; --sample=none with --cpuctxsw=system-wide also works. On Linux, use --cpuctxsw=system-wide.
OS wait calls on Windows need --trace=dx12 --dx12-wait-calls=true. Thread Callstacks need CPU sampling (--sample=process-tree or --sample=system-wide), and on Windows also --resolve-symbols=true to name the symbols. GPU metrics such as GR Active need --gpu-metrics-devices (see GPU Metrics). For all switches, see CLI Profile Command Switch Options.
Game stutters (CPU present-to-present)#
What it is: Variation in the interval between consecutive present calls on the CPU. This reflects the game engine’s frame production cadence: how evenly the engine processes input, simulates, renders, and submits frames.
In the GUI, CPU frames come from the graphics API present calls; for OpenXR applications, they run from xrBeginFrame to xrEndFrame. For a process with no graphics API trace, Nsight Systems uses ETW Present events from the trace instead. When Streamline reports frames, or when Reflex markers were recorded, Nsight Systems builds application frames from those markers instead, and Streamline markers take priority over Reflex markers (see FPS Overview). On Windows, Reflex markers are recorded only with --reflex-events=true. The graphics recipes use the same markers and the same graphics API present calls, but not OpenXR frames, and fall back to ETW Present events. NVTX ranges and marks that the application adds itself are not used as frame boundaries.
Why it matters even if the display absorbs it:
Input latency: Uneven frame production means uneven input sampling intervals. Even if the display shows smooth output, the game may poll input at 13 ms, then 26 ms, then 13 ms, making controls feel inconsistent.
Simulation consistency: Variable frame times mean a variable simulation time step, which can affect physics, animation, and gameplay feel.
Reflex end-to-end latency: If Reflex is active, game-side frame pacing directly affects the input-to-render latency pipeline. A long game frame delays the entire chain (see Reflex render latency and FPS Overview).
Stutter detection: rolling baseline method
The Stutter row of the GUI marks a frame as a stutter when it is much longer than the median of the frames around it; FPS Overview gives the window, the thresholds, and how to change them. A local median adapts to scene changes and works at any frame rate, so it is a better baseline than one median for the whole capture. Only frames longer than the local median are marked. A very short frame right after a long one is usually the catch-up half of a long-short pair, not a separate problem.
The Process Summary of the gfx_frame_analysis Recipe also reports timing patterns: long-short oscillation, and a difference between the median durations of even-numbered and odd-numbered frames. The recipe marks three frames in a row as part of an oscillation when the step into the middle frame and the step out of it go in opposite directions, and each step differs by more than 5 ms and by more than 100% of the shorter frame. It reports the share of frames that take part and flags oscillation when that share is above 25%. The even-odd test uses the same 5 ms and 100% thresholds. With these thresholds, ordinary jitter of a few percent does not count as oscillation.
With --frame, the gfx_frame_analysis recipe repeats the oscillation and even-odd checks on the selected frame and up to five frames on each side of it. When the selected frame is part of an oscillation, the recipe also shows whether it is the long or the short half, and which adjacent frame it pairs with.
In this window of up to 11 frames, one isolated long frame can flag oscillation. When the step up to it and the step down from it both pass the thresholds, they mark three frames, which is more than 25% of the window. If you selected the long frame, it gets the long-half role; if you selected a neighbor, that frame gets the short-half role. Treat the result as a cadence only when the durations in the Nearby Frames table alternate more than once.
The GUI runs a different, process-wide oscillation check (see FPS Overview). Do not expect the GUI and the recipe to agree.
The recipe also reports slow-frame episodes. An episode is three or more frames in a row that each take at least 1.8 times a baseline: the larger of the process’s median frame time and the median of up to 240 earlier frames that were not slow. Slow frames never enter that median, so a long plateau cannot become its own baseline, as it can in the Stutter row’s window of about 19 frames. Treat an episode as a sustained slowdown rather than a per-frame stutter, and do not expect its frame counts to match the Stutter row.
CPU-GPU swap-chain desynchronization (render queue flip-flop)#
A render queue flip-flop, also called CPU-GPU swap-chain desynchronization, is a repeating wait and catch-up cycle in CPU present timing. On the long half, the CPU finds the present queue full, or waits on a GPU fence, before it can submit. Once an earlier present leaves the queue, the next submit returns quickly (the short half). Tight frame-latency settings, uneven GPU frame cost, or a submit path that serializes on Present make this pattern more likely. For background on present queues and frames in flight, see CPU-GPU frame pipelining.
To detect the cadence, run the gfx_frame_analysis Recipe with --pid <PID> --frame <N> on a frame of the pattern, and read the local timing pattern, the frame’s role (long or short half), and its paired frame. Confirm the long half with the recipe’s Thread Scheduling and Thread Callstacks sections on the submit or render thread: the thread should spend most of the frame blocked in a fence, Present, or swap-chain wait, not running on a CPU.
Cadence alone does not name a cause. A detected oscillation describes pacing; it is not by itself the root cause of a hitch. Name swap-chain desynchronization as the cause only when the long-short alternation is the main source of frame-time variance. Use the slow-frame episodes of the gfx_frame_analysis recipe to tell a recurring plateau from isolated outliers. When an episode exists, start from a representative frame inside its range, and treat a much slower frame outside that range as a possibly separate cause. When no episode exists, investigate the slowest isolated frame. Rule out DLSS Frame Generation present bunching (see DLSS Frame Generation), and do not confuse this CPU present pattern with a display flip miss or with the Reflex OS Render Queue latency stage (see FPS Overview).
Serialized frame production or intermittent swap-chain desynchronization? An application that never overlaps CPU submission with GPU execution produces the same long-short cadence, because a submit path that blocks on GPU completion releases its queued presents in a burst and then stalls. Separate the two before naming a cause:
Is the wait on both halves or only on the long one? A wait in every iteration means that the pipeline has no frame in flight, but only when the wait targets the frame that its own iteration submitted. To check this, compare the time of the submit thread’s next call after the wait with the end of that frame’s GPU workload row. Intermittent desynchronization waits only on the long half. To find the blocking wait in each half:
On D3D12, the wait shows as a Win32 wait call on the thread’s OS runtime libraries row, recorded with
--dx12-wait-calls=true(see D3D12 API Trace).ID3D12Fence::SetEventOnCompletionusually returns almost at once, so its duration does not include the wait.On Vulkan,
vkWaitForFencesandvkWaitForPresentKHRinclude the wait in their own duration.The gfx_api_analysis Recipe, run with
--pid <PID> --frame <N>on each half, shows how many calls were made and how long they took. It does not show whether the thread was blocked or running in a call, and it shows no fence values. The recipe also does not show D3D12 Present calls: they are stored inDXGI_API, which the recipe does not read.
Is the GPU busy or idle across the long half? Serialized submission leaves a GPU idle gap only as long as the CPU’s recording time, so GR Active in
GPU_METRICSstays high and the GPU workload rows run back to back. Intermittent desynchronization leaves the GPU mostly idle through the long frame, because the CPU waits on queue capacity, not on GPU work. Presents that land within a fraction of a millisecond of each other most likely cannot all be displayed.
When these checks show that submission never overlaps GPU execution, the cause is a fence wait on the submit thread: report the missing pipeline as the cause, and the cadence as its consequence. Name swap-chain desynchronization only for a pipeline that keeps CPU and GPU work overlapped and loses that overlap from time to time.
Submitting all D3D12 work from one thread is a design fact, not a cause by itself. Name the wait it exposes: usually a fence wait, swap-chain desynchronization when the cadence matches, or more GPU work per frame.
Report the cycle, not the short half. When presents bunch, consecutive present intervals are not independent frames, so the median frame time mixes long and short halves and describes neither. Take the effective frame interval from the long half (the even or the odd frames, whichever are the long ones), state how many presents each cycle contains, and do not assume that the extra presents of a cycle reached the display as separate frames.
Display stutters (flip-to-flip)#
What it is: Variation in the interval between consecutive display updates (hardware flips or scanouts). This is what the user actually sees.
Why it matters:
Directly visible as judder, hitching, or held frames
A single long flip (2x normal) is noticeable even if the frames around it are smooth
Data needed: Flip-to-flip timing comes from the display flip events of the DxgKrnl provider. Of the WDDM trace modes, only the extensive trace records them (see Data needed): the MMIOFlipMultiPlaneOverlay, MMIOFlipMultiPlaneOverlay3, IndependentFlip, and PresentHistory events, together with the VSyncInterrupt and VSyncDPCMultiPlane events. These events have no dedicated table. They are stored as generic ETW events, in the ETW_EVENTS table and on the ETW rows of the timeline, and Nsight Systems does not compute flip intervals from them. A custom ETW trace of the DxgKrnl provider can also record them (see Custom ETW Trace).
When flip data is available, compute the intervals between consecutive flips; the same rolling-median idea applies to them. With VRR, the local median follows changes in refresh timing. Two patterns point to display-side problems: flip intervals at exact 2x multiples of the normal interval (a missed flip), and fewer flips than GPU presents (frames rendered but never shown).
The relationship between game and display stutters#
Game stutters and display stutters are not the same thing and do not always correlate:
Scenario |
Game stutter? |
Display stutter? |
User impact |
|---|---|---|---|
GPU finishes on time, flip scheduling misses a beat |
No |
Yes |
Visual hitch, but input latency unaffected |
CPU present timing oscillates, flip queue absorbs it |
Yes |
No |
Input feels inconsistent, but visually smooth |
GPU overloaded, frame takes 2x normal |
Yes |
Yes |
Both visual and input impact |
Frame queue drains, GPU catches up |
Compensating short frame |
No (queue absorbs) |
Short frame may feel like a “snap” |
Analysis approach#
When investigating stutters, measure every layer the capture contains and investigate each with equal depth:
CPU present-to-present (game frame time): for engine and input health
GPU present completion-to-completion (the stop times of WDDM queue packets marked as present packets, which need
--wddm-memory-trace=false): to separate GPU causes from CPU causesDisplay flip-to-flip, when the capture includes display flip events: for visual smoothness
Then compare: if game stutters exist but the display does not show them, the flip queue is absorbing them (good for visuals, but input latency may still be affected). If display stutters exist but game timing is clean, the issue is in the flip scheduling path (driver, display controller, compositor).
CPU-side present intervals show when frames were submitted, not when they were shown. Without display flip data, you cannot tell whether a stutter was visible to the user. Claim a visible stutter only when flip-to-flip data confirms it.
Locating a specific frame: the gfx_frame_analysis Recipe lists the frames of each process with their durations, and --frame <N> selects one. Each frame is the interval between two consecutive present timestamps from the sources described under Game stutters (CPU present-to-present). When the recipe uses present calls or ETW Present events and DLSS Frame Generation is active, the presents of generated frames usually also start frames, so frame numbers do not match application frames.
Scan the frame with nsys recipe gfx_frame_analysis --input report.nsys-rep --pid <PID> --frame <N>. It compares the ETW and WDDM event rates in the frame’s time window, counted across all processes, with those of a baseline frame. The WDDM rates count evictions, paging packets, and DMA packets, so they need --wddm-memory-trace=false. That mode records no DxgKrnl Present events, so when the report has no Streamline or Reflex markers and no graphics API trace, the recipe takes frames from DXGI Present events (see Data needed). With hardware-accelerated GPU scheduling, the DMA packet rate counts mostly paging and copy work. Prefer --baseline session_median when the frame is inside a slow-frame episode, because every neighbor is slow for the same reason and a local baseline hides the difference. --baseline previous gives no baseline for frame 1, and --baseline next gives none for the last frame; the comparisons are then not available.
Use the event rates only to support a finding: they show that activity changed, not how many milliseconds it cost (see the convergence checks).
Per-API call breakdown: to see which D3D12, Vulkan, or OpenGL calls dominate a frame, use the gfx_api_analysis Recipe with --pid <PID> --frame <N>. To find bursts of calls to one function, use --api <api> --function <name>. Bursts of UpdateTileMappings or CreateCommittedResource calls point to tiled-resource updates or allocation churn.
Thread blocking and callstacks: to tell whether a thread is parked in a per-frame wait or busy on a CPU, use the Thread Scheduling and Thread Callstacks sections of the gfx_frame_analysis Recipe. Thread Scheduling times are upper bounds (see the drill-down checklist). Read the block reasons as follows:
UserRequest: a wait that user-mode code asked for, such as a fence, event, mutex, or present wait.Resource,KeyedEvent, andAlertByThreadId: usually lock contention.CPU oversubscription is not a block reason. A preempted thread is still ready to run, so the recipe counts that time as waiting for a CPU.
Thread Scheduling also lists the OS runtime calls in the window, totaled per process and function. On Windows, these are the Win32 wait calls, such as WaitForSingleObjectEx, that --dx12-wait-calls=true records, and they appear under the name Win32 Wait API. On Linux, they are all the calls that --trace=osrt records, such as pthread_cond_wait, and also calls that are not waits, such as read. The recipe also lists the threads that ran in the selected frame but not in its baseline frame, sorted by on-CPU time. If Thread Callstacks shows no stacks for a thread, that thread was not sampled in the frame; check the next thread in the list.
When Thread Scheduling shows one application thread on a CPU for about the whole long frame with little blocked time (for example, the main thread on a CPU for about 98% of the frame while the render, RHI, and worker threads wait for it in UserRequest), the frame is CPU-bound on that thread. When that thread is the main or simulation thread, this is the Main thread heavy work cause. One such thread is enough to call the frame CPU-bound. Name the subsystem from its callstacks.
When many application workers appear among the threads that ran in the selected frame but not in its baseline frame, and they form a wide phase, the cause is different: the Worker pool join stall (see Common stutter causes). In that case the workers own the subsystem, so read the Thread Callstacks modules of a leading worker from that list. The busiest single thread is usually the application’s main or simulation thread, and its modules do not name the worker subsystem, so do not settle the question from that thread alone. Do not dismiss a third-party module as minor because it is rare in one thread’s samples; check its share in a worker’s samples first. Unresolved leaf symbols do not end the search, because the modules still name the binary.
An on-CPU thread is running, not necessarily working. A thread can hold a core while it polls for something it does not own: a worker’s completion flag, a lock, a GPU fence, or an I/O result. A spin-wait uses up its time slice and is preempted rather than blocked, so its Thread Scheduling times look like a real CPU overrun. The fix is the opposite one: reducing that thread’s own work changes nothing, so remove the dependency or replace the poll with a blocking wait. When one thread is on a CPU across the hitch but its samples do not name work, check for a poll loop before you conclude that the thread is simply busy:
Do the leaf symbols name a wait primitive? Thread Callstacks leaf symbols concentrated in a yield or spin primitive (
SwitchToThread,Sleep, anRtl*critical-section spin, or an engine or middleware spin mutex) identify the poll loop directly. This is the only signal that names the cause. It needs resolved symbols, so it is not available when the leaves are raw addresses.Does the thread sleep more often than its peers? A poll loop that calls
Sleep(1)or longer blocks with theDelayExecutionreason on each call. ASleep(0)loop does not block: the thread stays ready to run, so that time shows as waiting for a CPU. Heartbeat, async-writer, and HTTP threads also sleep from time to time, so compare the suspect thread’sDelayExecutioncount with the other named threads in the same capture, not with zero.Does the on-CPU run end when a producer finishes? A poll ends when the awaited work completes, not at a boundary of the polling thread’s own work. Compare the end of the on-CPU run with the end of the awaited GPU workload, worker phase, or I/O completion. Real CPU work does not line up with another entity’s completion.
This check has two limits. Leaf concentration alone does not separate a poll loop from a hot compute loop, and per-frame sampling of a single thread often gives only a few tens of samples, so do not conclude that a thread spins from a concentration ratio alone. Also, a thread that yields stays ready to run, so the gfx_frame_analysis recipe counts the time until it runs again as waiting for a CPU and lists no block reason for it. A missing YieldExecution block reason is not evidence against spinning.
Investigating game stutters#
Display stutters are user-visible, but game frame stutters are where the actionable root causes live. A display stutter caused by the flip scheduling path is often outside the game developer’s control. A game frame stutter caused by DirectStorage, PSO compilation, or asset streaming is directly actionable.
Both layers need a full root cause investigation. Do not stop because the Stutter row marks no frames or does not appear. The row appears only when it marks at least one frame. It does not analyze a process with fewer than 20 CPU frames, and it never tests the first 10 and the last 10 frames of a process. Also investigate:
Tail analysis: What causes the P95-P99 tail? Even when no frame is marked as a stutter, a P99 well above the median with a clear cause (for example, streaming bursts) is a meaningful finding.
Short frame analysis: Frames well below the median often pair with long frames that follow them. Check the timing patterns of the gfx_frame_analysis Recipe before drilling into local events. Decide whether the pattern comes from DLSS Frame Generation (see DLSS Frame Generation) or is a render queue flip-flop with a fence or Present wait in the frame.
Per-thread differential: Compare the periodic CPU samples of each thread in tail frames and in median frames. Look for threads whose activity differs by more than 2x, and follow those threads to their callstacks.
Known causes: Check the common stutter causes in the tail frames, starting with asset streaming and PSO compilation. Asset streaming is one of the most common causes of game frame variance. Compare counts and totals only between windows of equal length, or convert them to rates first (see rigor and units).
GPU memory activity: Check the DxgKrnl GPU memory events in every investigation, whether or not streaming is suspected. Compare event counts between stutter and clean windows of equal length. Which events a report contains depends on the WDDM trace mode (see Data needed). For what each event means, see DxgKrnl events.
Allocations and budgets (default and extensive WDDM trace):
DeviceAllocation,AdapterAllocation,VidMmProcessBudgetChange,VidMmProcessUsageChange,VidMmProcessCommitmentChange. With--wddm-memory-trace=falsealone, the report has onlyAdapterAllocationand the budget and usage changes.Memory transfers (default and extensive WDDM trace):
MemoryTransferResidency and paging (
--wddm-memory-trace=false):AllocationFault,PageInAllocation,MarkAllocation, and the paging queue packetsPage tables (
--wddm-memory-trace=false):PagingOpUpdatePageTable,PagingOpFlushTlbEvictions (
--wddm-memory-trace=false):EvictAllocationOther VidMm events (extensive WDDM trace only):
VidMmMakeResident,ProcessAllocationDetails,SetAllocationPriority
A burst (5-10x above baseline) during stutter frames points to video memory pressure or heavy resource churn. This can come from streaming, but also from scene transitions, LOD changes, or any workload that creates or destroys GPU resources.
Demotion: Demotion events do not show which allocations were demoted or whether that hurt performance. Report the activity without assuming how severe it is.
Cross-tabulation: For each possible cause (streaming, PSO compilation, memory allocation, and so on), build a 2x2 table: cause present or absent against tail frame or normal frame. This separates true contributors from activity that only happens at the same time.
The Stutter row decides which frames to mark, not whether to investigate. Any frame time variance with an identifiable cause is worth reporting, because:
The flip queue may not always absorb it (different GPU, different resolution, different scene)
It affects input sampling consistency
It may get worse under load in more demanding scenes
It is actionable feedback for the game developer
CPU-bound vs GPU-bound: GPU idle decomposition#
To classify whether a stutter frame is CPU-bound or GPU-bound, split its frame time on the GPU timeline as cpu_frametime ~= gpu_busy_ms + gpu_idle_ms. gpu_idle_ms is the largest contiguous GPU-idle gap that overlaps the stutter window: wall-clock time where GR Active, Compute Warps in Flight, and Sync Copy Engine Active or Async Copy Engine Active in GPU_METRICS are all below about 5%. When GPU metrics are missing, use the largest gap in the union of the DMA packets of all engines of the game’s GPU instead. This needs --wddm-memory-trace=false, and it does not work with hardware-accelerated GPU scheduling. Measuring a contiguous wall-clock gap, not a single logical frame, avoids frame alignment errors under DLSS Frame Generation. Counting only time when graphics, compute, and copy work are all idle avoids a common trap: looking at 3D packets alone counts async compute and frame generation work as idle time.
Classify against what is usual for this trace, not against a fixed percentage. Take the baseline from a clean reference window (the median over clean frames), then split the stutter excess:
excess_idle_ms = frame_gpu_idle_ms - baseline_idle_msexcess_busy_ms = frame_gpu_busy_ms - baseline_busy_ms
Which term dominates the stutter delta |
Classification |
Action |
|---|---|---|
|
CPU-bound (GPU starved for submission) |
Find what blocked submission; see the three outcomes after the GPU context check |
|
GPU-bound (more GPU work) |
Compare GPU work durations with clean frames; the GPU was not starved |
Neither dominates |
Other (display, scheduling, noise) |
Investigate elsewhere |
This test adjusts itself to the trace: a trace that normally runs at about 0% idle and rises to 5% on a stutter frame is unusual (the 5% is almost the entire excess), while a trace that idles at about 8% everywhere tells you nothing when one frame reaches 9%. Judge by the deviation from the clean baseline. The raw gpu_idle_ms / cpu_frametime fraction is only a sanity check on top of the baseline test: above about 0.10 the frame is clearly idle-driven, 0.05-0.10 is an unclear band where you must run the GPU context check before you call the frame CPU-bound, and below the noise floor of the trace the frame is not idle-driven.
gpu_idle_ms is a symptom, never the cause in a magnitude match. It shows that the frame is CPU-bound, not why submission stopped. The cause duration you cite must be the wall-clock duration of a single entity on the critical path: one thread’s extra blocked time inside its critical-path phase, the on-CPU time of a thread that compiles shaders synchronously, or a single fence wait. If the value you cite is the idle gap, the present gap, or the frame delta itself, the ratio is about 100% by construction and proves nothing (see Magnitude match).
GPU context check (required in the 0.05-0.10 band, good practice always). gpu_idle_ms counts the GPU as busy while any context runs work on it, so a long frame with low idle time does not by itself prove that the game kept the GPU busy: the GPU work may belong to another context while the game’s own work starved. List the contexts whose DMA packets ran on the game’s GPU in the window, and find the process that owns each context that is not the game’s: in WDDM_QUEUE_PACKET_START_EVENTS, the globalTid of that context’s packets is the thread that submitted them. Decode the process ID from it and look the process up in PROCESSES (see Serialized process and thread identifiers). The packet tables need --wddm-memory-trace=false. With hardware-accelerated GPU scheduling, the DMA packets miss most application work, so use the WDDM HW Scheduler rows instead.
When idle time is the dominant excess, the investigation ends in one of three outcomes:
The game’s context is idle and a blocker is found -> CPU-bound. Name what blocked submission: a blocked thread, a lock, a fence, or synchronous shader compilation. Do not rule this out only because DMA packets stay long elsewhere in the window; that tests whether the GPU stayed busy, which is a different question.
Another context filled the gap -> GPU context contention. This is a real finding, not a dead end. Name the process that owns the competing context (the compositor
dwm.exe, an overlay, a capture tool, or a second 3D application) and report that it delayed the game’s GPU work.The game’s context is idle, with no blocker and no competing context -> report the cause as unattributed (low confidence). After you have tested every hypothesis, saying that the cause is unknown is the correct result.
GPU busy during a CPU wait: fence-wait vs CPU-side barrier#
When a thread shows UserRequest blocks and the wait target is unknown, what the GPU did during the wait tells a CPU-side barrier from a GPU fence:
GPU idle during the wait -> the GPU was starved. The cause is on the CPU side (a worker or task barrier, or late command recording), so reduce the CPU barrier.
GPU idle, and the wait is in Present or on the frame-latency waitable object -> this is a present-queue wait, not a CPU barrier (see render queue flip-flop).
GPU busy through the wait -> the thread waited on a GPU fence. The GPU work of the frame takes longer than the target frame interval, so the frame is GPU-bound.
A CPU barrier and a GPU fence wait look the same in Thread Scheduling, where both show UserRequest blocks; only GPU activity tells them apart. Use DMA packets for this check, because they show when a GPU engine ran the work. Queue packets do not: a queue packet starts when the work is queued and stops when it retires, so back-to-back queue packets do not prove that the GPU was busy. A gap with no queue packet on any context does mean that nothing was queued, so the GPU was idle. When the capture has GPU metrics, confirm the result with GR Active.
Pair each DMA packet start with its stop on the context and the submission ID, as the wddm_queue_sum statistical report does, and keep the packets that overlap the wait window. Both tables need --wddm-memory-trace=false:
SELECT s.gpu, s.engineType, s.context,
s.start AS run_start, e.start AS run_end,
(e.start - s.start) / 1e6 AS run_ms
FROM WDDM_DMA_PACKET_START_EVENTS s
JOIN WDDM_DMA_PACKET_STOP_EVENTS e
ON e.context = s.context
AND e.uliCompletionId = s.uliSubmissionId
AND e.start >= s.start
WHERE s.start < <win_end_ns>
AND e.start > <win_start_ns>
ORDER BY s.start
Gaps between the packets, across all engines of the game’s GPU, are GPU idle time. Engine type 1 is the 3D engine and engine type 6 is a copy engine (see SQLite Schema Event Values). To find the wait window in SCHED_EVENTS, see SQLite Schema Pitfalls.
Hardware-accelerated GPU scheduling. These DMA packet checks work only without hardware-accelerated GPU scheduling (HWS). On a GPU that uses HWS, DxgKrnl reports the application’s work as HwSchedDmaPacket events, which Nsight Systems does not show or export. The DMA packet tables then hold mostly paging and copy work for that GPU, so it looks idle even while it runs the application’s frames. In the queue packet tables, the context column of work on a hardware queue holds the hardware queue handle, not the context: the parentDxgHwQueue value of WDDM_HW_QUEUE_EVENTS, whose context column gives the real context. A join on context with the DMA packet tables finds nothing for this work. If the timeline shows WDDM HW Scheduler rows for a GPU, that GPU uses HWS. In that case, use GR Active in GPU_METRICS to see whether the GPU was busy, and the WDDM HW Scheduler rows to see which process ran on each hardware queue (see WDDM HW Scheduler).
Magnitude match: single-thread wall-clock only#
When you compute the magnitude of a claimed cause and compare it with the frame delta, use the wall-clock excess of the single thread or phase on the critical path, never a sum of durations from several threads, several phases, or concurrent blocks. Concurrent work overcounts: if thread A takes 3 ms and thread B takes 4 ms in parallel, the wall-clock impact is max(3, 4) = 4 ms, not 7 ms. Summing them gives a ratio above 100%, which is wrong, not just imprecise.
Valid comparisons:
A single thread’s start-to-end duration in the stutter frame (NVTX phase start and end, or the thread’s first sched-in to last sched-out in
SCHED_EVENTS) minus the same thread’s median duration over clean frames.The excess of one thread’s time in one scheduling state (waiting for a CPU, or blocked) inside its phase on the critical path, against the same thread in the baseline frame.
A single discrete blocking event (one off-CPU wait), compared directly with the frame excess.
Invalid comparisons:
Summing the off-CPU blocks of a thread that fall outside its phase on the critical path. Those blocks run alongside other work; only the blocks inside that phase can delay the frame.
Summing durations across several threads. Only the thread on the critical path decides when the frame is delivered.
Summing WDDM packet durations into an “aggregate GPU work” span. Packets run on parallel engines and overlap, so a summed GPU work span longer than the frame itself is a measurement artifact, not a workload spike. Measure GPU load by the contiguous GPU-busy wall-clock span and GR Active, never by the sum of packet durations.
Any comparison that gives a ratio above 100% without a clear explanation of why the cause can exceed the effect, for example because the cause was measured against a different baseline, or in a window wider than the frame. A ratio above about 130% almost always means a measurement error.
If you cannot isolate the single phase on the critical path, treat the ratio as an estimate: the block durations are measured, but which of them were on the critical path needs callstack confirmation.
Common stutter causes#
A non-exhaustive reference of causes seen in practice, with the key check for each.
Game engine / application#
Cause |
What to look for |
|---|---|
Shader compilation (PSO) |
PSO compile threads or D3D background threads continuously on a CPU across the long frames and absent in clean windows. Runtime shader compilation can cause frame-time spikes for the whole session. It is often a consequence of a pipeline stall rather than the root cause, so check which came first. |
Asset streaming / loading |
The gfx_frame_analysis list of threads that ran in the selected frame but not in its baseline frame includes |
Main thread heavy work |
The main or render thread changes from short on-CPU runs with waits to continuous on-CPU time during the stutter frame. Callstacks show the subsystem (scene traversal, physics, script). |
Spin-wait on a dependency |
A thread on a CPU for about the whole frame delta, preempted rather than blocked, plus one positive spin signal from the poll-loop checks in Analysis approach. On-CPU time alone fits heavy work just as well. |
Worker pool join stall |
The CPU Concurrency section of the gfx_frame_analysis recipe shows much more busy core time and more threads running at once than the clean baseline, and the delayed application or render thread is blocked until the worker phase completes. Identify which thread on the critical path the phase delays. |
Thread contention / lock convoy |
Several threads alternate between blocked and running on the same lock. |
Scene complexity spike |
GPU work per frame increases (longer GPU frame time), so the CPU waits longer at the present fence or swap chain. GR Active and SMs Active in |
GPU / driver#
Cause |
What to look for |
|---|---|
GPU starvation |
GR Active drops to near zero during the stutter while the CPU is late submitting work, so the cause is on the CPU side. Classify the frame with the GPU idle decomposition, which also lists the possible outcomes. |
GPU context contention |
The “idle” gap of the 3D engine is actually filled by another GPU context. Map the competing context to its owning process with the GPU context check of the GPU idle decomposition, and name the application (compositor, overlay, capture tool, or a second application). |
VRAM pressure / paging |
Spikes in |
DirectStorage decompression |
Activity of the DirectStorage Worker, IOSubmitThread, or decompression threads at the same time as the stutter, often with a copy engine spike. |
Clock frequency drop |
GPC Clock Frequency or SYS Clock Frequency in the GPU metrics drops during the stutter window (thermal throttling, a power limit, or a performance state change). |
Flip scheduling |
Display flips are late or missed even though the GPU delivers frames on time. This check needs flip intervals (see Display stutters (flip-to-flip)). |
OS / system#
Cause |
What to look for |
|---|---|
DPC / ISR interference |
Long DPCs or ISRs on the core that runs the game’s render thread. Collect them with |
Core contention from other processes |
Another process (antivirus, overlay, telemetry) runs on the same core as the render thread during the stutter. Check |
CPU oversubscription |
More threads are ready to run than there are cores. In the gfx_frame_analysis recipe, Thread Scheduling shows the delayed thread waiting for a CPU for much of the frame, and CPU Concurrency shows a mean number of threads running at once for the whole machine close to the core count. The whole-machine numbers need a system-wide trace (see Data needed). On Linux, the recipe cannot separate waiting for a CPU from blocked time. |
NUMA / CCX / CCD migration |
The thread moves across NUMA nodes or AMD CCX or CCD boundaries and pays a cold-cache penalty. Check for changes of the CPU number in scheduling events. |
Power management |
A CPU core enters a deep C-state and takes too long to wake up, or P-state changes cause a frequency dip. More common on laptops and power-saving profiles. |
Disk I/O stall |
Blocking file I/O on the render thread (rare but severe). On Linux, OS runtime trace ( |
Memory commit / page fault storm |
A large allocation or working set growth triggers hard page faults. Check for |
Capture overhead |
The extensive WDDM trace also records DxgKrnl Profiler events (IDs 105 and 106). These events can be a large share of all events and can make the report much larger. If a stutter lines up with bursts of these events, capture again without the extensive trace before you blame the application. |
DLSS Frame Generation#
With DLSS Frame Generation, each frame that the application renders reaches the display as one or more generated frames plus the application frame, and each of them usually has its own present call (see FPS Overview). Whether this inflates stutter counts depends on where the frames come from:
When Streamline reports frames, the CPU Frame Duration and Stutter rows and the graphics recipes use application frames, one for each frame the application renders. Generated frames do not inflate the stutter count.
When frames are built from raw present calls, every present starts a new frame. This happens with Streamline releases older than 2.12 when the report has no Reflex markers, and when the recipes fall back to ETW Present events. Present intervals then bunch into long-short patterns, and stutter and oscillation counts are inflated.
Before you blame frame generation for present bunching, confirm that it actually generated frames. When Streamline reports frames, the Presented Frames row shows generated frames between the application frames only when frame generation ran. Without Streamline frames the row does not appear, and a loaded Streamline module alone does not prove that frames were generated.
Advanced Report Analysis#
Nsight Systems Advanced Report Analysis is functionality to better support complex
statistical analysis across multiple result files. You can run recipes through
Analysis Library in the GUI or with the nsys recipe CLI command.
Note
Advanced Report Analysis is not supported on macOS with Intel (x86_64) processors.
Possible use cases for this functionality include:
Multi-Node Analysis - When you run Nsight Systems across a cluster, it typically generates one result file per rank on the cluster. While you can load multiple result files into the GUI for visualization, this analysis system allows you to run statistical analysis across all of the result files.
Multi-Pass Analysis - Some features in Nsight Systems cannot be run together due to overhead or hardware considerations. For example, there are frequently more CPU performance counters available than your CPU has registers. Using this analysis, you could run multiple runs with different sets of counters and then analyze the results together.
Multi-Run Analysis - Sometimes you want to compare two runs that were not taken at the same time together. Perhaps you ran the tool on two different hardware configurations and want to see what changed. Perhaps you are doing regression testing or performance improvement analysis and want to check your status. Comparing those result files statistically can show patterns.
Complex/multi-phase analysis - Sometimes you may want to perform a complicated, or multi-phase analysis on one or more results files. The helper functionality available in the Advanced Analysis system can simplify common steps.
Complex data output - Sometimes you want to be able to build complex visualizations from your analysis, rather than just tabular data from bare statistics.
Analysis Steps
Note
Required recipe dependencies are installed automatically when needed at run time. See Installing Advanced Analysis System in the Installation Guide for more information.
Generate the reports - Generate the reports as you always have, in fact, you can use reports that you have generated previously.
Set up - Choose a recipe (see Available Advanced Analysis Recipes) and provide its required inputs and options.
Launch Analysis - Run the recipe. For CLI execution modes, including local processing and Dask, see Running Advanced Analysis Recipes from the CLI.
Output - the recipe produces an output directory. Depending on the recipe, this can include tables, notebooks, and an
.nsys-analysisfile for opening the results in the Nsight Systems GUI.View the data - depending on your recipe, you can have any number of visualizations, from simple tabular information to Jupyter notebooks which can be opened inside the GUI.
Analysis Library#
Recipes are available through Analysis Library in the Nsight Systems GUI. Open a report and select Analysis Library (preview) from the View selector drop-down list in the report tab.
Use Analysis Library to browse and run recipes for post-collection analysis of one or more reports. The UI guides you through selecting a recipe, configuring its inputs and options, and viewing the results.
Available Advanced Analysis Recipes#
Browse recipes in Analysis Library in the GUI or list installed recipes
with nsys recipe --help in the CLI.
Nsight Systems provides several initial analysis recipes, mostly based around making our existing statistics and expert systems rules run multi-report.
These recipes can be found at
<install-dir>/target-<os>-<arch>/python/packages/nsys_recipe/recipes.
Please note that all recipes are in the form of python scripts. You may alter
the given recipes or write your own to meet your needs. Refer to
Tutorial: Create a User-Defined Recipe for an example of how to do this.
However, be advised that the APIs may change for the next few versions. Additional
recipes will be added on an ongoing basis.
For a recipe’s inputs and options, use Analysis Library or
nsys recipe <recipe-name> --help. See
Running Advanced Analysis Recipes from the CLI for CLI input, output, execution,
and troubleshooting guidance.
List of recipes
Each recipe will be tagged with one or more keywords to help understand its purpose.
Keywords |
Description |
|
|---|---|---|
Expert System |
The recipe originated from the Expert System. A script
with the same name is also available via |
|
Stats System |
The recipe originated from the Stats System. A script
with the same name is also available via |
|
Trace |
The recipe provides a trace record of individual events that are observable in the GUI timeline. |
|
Summary |
The recipe provides a summarized view of events, often representing aggregated data. |
|
Pace |
The recipe provides a detailed analysis of how a specific event progresses across the application. |
|
Heatmap |
The recipe provides a heatmap that visualizes patterns across the application. |
|
- cuda_api_sumCUDA API Summary
This recipe provides a summary of CUDA API functions and their execution times.
Keywords: CUDA, Summary, Stats System
- cuda_api_syncCUDA Synchronization APIs
This recipe identifies synchronization APIs that block the host until the issued CUDA calls are complete.
Keywords: CUDA, Synchronization, Trace, Expert System
- cuda_gpu_kern_histCUDA GPU Kernel Duration Histogram
This recipe represents the probability of the duration of a CUDA kernel among all its instances or all kernels in the program.
Keywords: CUDA, Kernel, Histogram, Duration
- cuda_gpu_kern_paceCUDA GPU Kernel Pacing
This recipe investigates the progress and consistency of a particular CUDA kernel throughout the application.
Keywords: CUDA, Kernel, Pace
- cuda_gpu_kern_sumCUDA GPU Kernel Summary
This recipe provides a summary of CUDA kernels and their execution times.
Keywords: CUDA, Kernel, Summary, Stats System
- cuda_gpu_mem_size_sumCUDA GPU MemOps Summary (by Size)
This recipe provides a summary of GPU memory operations and the amount of memory they utilize.
Keywords: CUDA, Memory, Summary, Stats System
- cuda_gpu_mem_time_sumCUDA GPU MemOps Summary (by Time)
This recipe provides a summary of GPU memory operations and their execution times.
Keywords: CUDA, Memory, Summary, Stats System
- cuda_gpu_time_util_mapCUDA GPU Time Utilization Heatmap
This recipe calculates the percentage of time that CUDA kernels were running.
Keywords: CUDA, Kernel, Heatmap
- cuda_memcpy_asyncCUDA Async Memcpy with Pageable Memory
This recipe identifies asynchronous memory transfers that end up becoming synchronous if the memory is pageable.
Keywords: CUDA, Memcpy, Trace, Expert System
- cuda_memcpy_syncCUDA Synchronous Memcpy
This recipe identifies memory transfers that are synchronous.
Keywords: CUDA, Memcpy, Trace, Expert System
- cuda_memset_syncCUDA Synchronous Memset
This recipe identifies synchronous memset operations with pinned host memory or Unified Memory region.
Keywords: CUDA, Memset, Trace, Expert System
- diffStatistics Diff
This script compares outputs from two runs of the same statistical recipe.
Keywords: Diff, Summary
- dx12_mem_opsDX12 Memory Operations
This recipe flags problematic memory operations with warnings.
Keywords: DX12, Memory, Trace, Expert System
- file_access_sumOS Runtime File Access Summary
This recipe provides a summary of file access functions, including high-level overview of file access patterns across the system.
For details and use cases of this recipe, see file_access_sum Recipe.
Keywords: OSRT, Summary
- gfx_api_analysisGraphics API Analysis
This recipe summarizes graphics API durations, compares session percentiles, and finds chains of consecutive calls to one function.
For details and use cases of this recipe, see gfx_api_analysis Recipe.
Keywords: Graphics, DX12, Vulkan, OpenGL, Summary, Trace
- gfx_frame_analysisGraphics Frame Analysis
This recipe summarizes CPU frame timing and compares a selected frame’s ETW and WDDM activity with a baseline frame.
For details and use cases of this recipe, see gfx_frame_analysis Recipe.
Keywords: Graphics, Frames, Summary, ETW, WDDM
- gfx_hotspotGraphics Hotspot Analysis
This recipe generates a report of CPU hotspots for graphics applications.
The output format for this recipe is different than other recipes. See gfx_hotspot Recipe.
Keywords: DX12, Vulkan, Summary, Trace
- gpu_gapsGPU Gaps
This recipe identifies time regions where a GPU is idle for longer than a set threshold.
Keywords: CUDA, Utilization, Expert System
- gpu_metric_util_mapGPU Metric Utilization Heatmap
This recipe calculates the percentage of SM Active, SM Issue, and Tensor Active metrics.
Keywords: GPU Metrics, Heatmap
- gpu_metric_util_sumGPU Metrics Utilization Summary
This recipe provides a summary of different GPU metrics. GPU metrics are based on binary inclusion. Any ranges that do not include at least one sampling point are excluded from the output.
Keywords: GPU Metrics, Summary
- gpu_time_utilGPU Time Utilization
This recipe identifies time regions with low GPU utilization.
Keywords: CUDA, Utilization, Expert System
- gpu_vram_usage_traceGPU VRAM Usage Trace
This recipe traces the VRAM usage of GPU workloads, allowing comparison of changes between CPU frames, and identifying issues in resource migration between VRAM and SYSMEM, and with resource allocation & deallocation.
For details and use cases of this recipe, see gpu_vram_usage_trace Recipe - Preview Feature.
Keywords: VRAM, Trace
- mpi_gpu_time_util_mapMPI and GPU Time Utilization Heatmap
This recipe calculates the percentage of time that CUDA kernels were running and MPI communication was active, as well as their overlap.
Keywords: MPI, CUDA, Kernel, Utilization, Heatmap
- mpi_sumMPI Summary
This recipe provides a summary of MPI functions and their execution times.
Keywords: MPI, Summary
- nccl_gpu_overlap_traceNCCL GPU Overlap Trace
This recipe calculates the percentage of overlap for communication and compute kernels. Communication kernels are identified by the ‘nccl’ prefix.
Keywords: NCCL, CUDA, Kernel, Overlap, Trace
- nccl_gpu_proj_sumNCCL GPU Projection Summary
This recipe provides a summary of NCCL functions projected from the CPU onto the GPU. For advanced NCCL tracing, use nccl_sum instead because GPU projection is unnecessary.
Keywords: NCCL, CUDA, GPU Projection, Summary
- nccl_gpu_time_util_mapNCCL GPU Time Utilization Heatmap
This recipe calculates the percentage of time that communication and compute kernels were running, as well as their overlap. Communication kernels are identified by the ‘nccl’ prefix.
Keywords: NCCL, CUDA, Kernel, Utilization, Overlap, Heatmap
- nccl_paceNCCL Pacing
This recipe investigates the progress and consistency of repeated NCCL collective communication contexts. It requires advanced NCCL tracing.
Keywords: NCCL, Pace
- nccl_stragglerNCCL Straggler
This recipe analyzes NCCL collective timing to identify ranks that repeatedly delay communicator progress. It requires advanced NCCL tracing.
Keywords: NCCL, Straggler, Collective, Heatmap
- nccl_sumNCCL Summary
This recipe provides a full-run NCCL communication summary over communicators, operations, message sizes, datatypes, and ranks.
Keywords: NCCL, Summary
- network_map_awsAWS Metrics Heatmap
This recipe displays heatmaps of AWS EFA metrics.
Keywords: Network, AWS, EFA, Heatmap
- network_sumNetwork Traffic Summary
This recipe provides a summary of the network traffic over NICs and InfiniBand Switches.
Keywords: Network, Summary
- network_traffic_mapNetwork Devices Traffic Heatmap
This recipe displays heatmaps of sent traffic, received traffic, and congestion events for network devices.
Keywords: Network, Heatmap
- nvtx_cpu_topdownCPU Topdown methodology metrics correlated to NVTX ranges
This recipe calculates CPU Topdown methodology metrics for NVTX push/pop ranges based on collected PMU core events for NVIDIA CPUs featuring Arm cores.
For details and use cases of this recipe, see nvtx_cpu_topdown Recipe.
Keywords: NVTX, CPU Topdown, Metrics, Summary
- nvlink_sumNVLink Network Throughput Summary
This recipe provides a summary of the NVLink network throughput.
Keywords: NVLink, Summary
- nvtx_gpu_proj_paceNVTX GPU Projection Pacing
This recipe investigates the progress and consistency of a particular NVTX range projected from the CPU onto the GPU, throughout the application.
Keywords: NVTX, GPU Projection, Pace
- nvtx_gpu_proj_sumNVTX GPU Projection Summary
This recipe provides a summary of NVTX time ranges projected from the CPU onto the GPU, and their execution times.
Keywords: NVTX, GPU Projection, Summary, Stats System
- nvtx_gpu_proj_traceNVTX GPU Projection Trace
This recipe provides a trace of NVTX time ranges projected from the CPU onto the GPU.
Keywords: NVTX, GPU Projection, Trace, Stats System
- nvtx_paceNVTX Pacing
This recipe investigates the progress and consistency of a particular NVTX range throughout the application.
Keywords: NVTX, Pace
- nvtx_sumNVTX Range Summary
This recipe provides a summary of NVTX Start/End and Push/Pop Ranges, and their execution times.
Keywords: NVTX, Summary, Stats System
- osrt_sumOS Runtime Summary
This recipe provides a summary of C library functions and their execution times.
Keywords: OSRT, Summary, Stats System
- s3_access_sumS3 Access Summary
This recipe provides a summary of S3 object store access across traced S3 client libraries.
For details and use cases of this recipe, see s3_access_sum Recipe.
Keywords: S3, Storage, Summary
- storage_util_mapStorage Metrics Heatmap
This recipe displays heatmaps of storage devices metrics.
Keywords: Storage, Heatmap
- ucx_gpu_time_util_mapUCX and GPU Time Utilization Heatmap
This recipe calculates the percentage of time that CUDA kernels were running and UCX communication was active, as well as their overlap.
Keywords: UCX, CUDA, Kernel, Heatmap
Running Advanced Analysis Recipes from the CLI#
Advanced Analysis recipes process one or more Nsight Systems reports and produce tables, Jupyter notebooks, or other analysis output. The options and examples in this section apply to the CLI. For GUI access, see Analysis Library.
Use the recipe command to run a recipe:
nsys recipe [args] <recipe-name> [recipe args]
Use nsys recipe --help to list installed recipes and
nsys recipe <recipe-name> --help for a recipe’s inputs and options.
Recipe inputs#
Recipes commonly accept one or more .nsys-rep files with --input.
You can also pass a directory to process its report files in filename order.
Subdirectories are not searched. Append :n to process only the first
n reports. For example:
nsys recipe cuda_api_sum --input reports/:8
This example processes up to eight reports in reports/, in filename order.
Output locations#
--output selects the recipe results directory, while --export-dir
selects the intermediate report-export location. If the output directory
already exists, the recipe warns and uses a new temporary directory unless
--force-overwrite is specified. Check the output path printed after the
run. --force-overwrite deletes the existing result directory.
Execution modes#
The default concurrent mode processes reports in parallel on the local
system. Use --mode none for sequential processing and to check whether a
failure is specific to parallel execution. The dask-futures mode uses Dask
for distributed processing; see Configuring Dask before using it.
Comparing recipe runs#
Use diff to compare the output directories from two runs of the same
statistical recipe. It reads recipe output, not .nsys-rep files. List
compatible recipes with nsys recipe diff --print-compat-recipes.
Run the source recipe with --format jl to produce the Parquet files that
diff reads. For example:
nsys recipe cuda_api_sum --input baseline.nsys-rep --output baseline_output --format jl
nsys recipe cuda_api_sum --input comparison.nsys-rep --output comparison_output --format jl
nsys recipe diff --input baseline_output comparison_output
The recipe subtracts the second input from the first. With the baseline first, as in this example, a positive duration difference means the comparison took less time. Rows found only in the comparison are omitted. Rows found only in the baseline have no calculated difference, rather than zero.
Columns and units come from the source recipe. Use the same recipe version and analysis options for both runs.
Large inputs and failures#
Processing more or larger reports can increase memory and disk use. Process
fewer reports or use :n on a directory to reduce the input set. Time and
NVTX filters narrow the analysis but may not reduce memory needed to export
and load the data. For disk-space failures, check the output and export locations.
If a recipe reports that no data is available, confirm that the report contains the trace source required by that recipe and that filters did not exclude the relevant interval. Check capture diagnostics for incomplete collection. An empty result does not necessarily mean that no activity occurred.
Recipe Output Examples#
Recipe output can include data tables and Jupyter notebooks in a result directory, as shown below.
Output directory contents#
The exact files and columns depend on the recipe and selected options. Common recipe outputs include:
files.parquetorfiles.csvfor the input-file and rank mapping.rank_stats.parquetfor per-input or per-rank statistics.all_stats.parquetfor aggregate statistics across inputs.A Jupyter notebook such as
stats.ipynbfor interactive analysis.An
.nsys-analysisfile for opening the recipe results in the Nsight Systems GUI.
Ranks assigned by a recipe identify its inputs and are not necessarily MPI ranks. Use the file mapping to identify the source report.
Keep the notebook and data files in the output directory when opening an
.nsys-analysis file, because the files refer to one another.
Table
Trace or summary data will be stored in data storage formats such as CSV, Parquet, or Arrow. Typically, you can also access the same data within the output Jupyter notebook.
Summary table:
Trace table:
Overlap table:
Visualization
Some recipes include data visualization in the output Jupyter notebooks. These graphs use Plotly, which provides interactivity.
Summary graph:
Box plot:
Line graph:
Top N graph:
Pace graph:
Heatmap:
Opening in Jupyter Notebook#
Recipes that produce Jupyter notebooks can also create an .nsys-analysis
file for opening the results in the Nsight Systems GUI. You can run these
recipes from Analysis Library or the CLI. To open an existing result, select
File->Open in the GUI and choose the .nsys-analysis file.
Open the folder icon and click on the notebook icon to open the Jupyter notebook.
Run the Jupyter notebook:
And the output appears on-screen. In this case a heat map of activity running a Jacobi solver.
Configuring Dask#
The multi-report analysis system does not offer options to configure the Dask environment. However, you could achieve this by modifying the recipe script directly or using one of the following from Dask’s configuration system:
YAML files: By default, Dask searches for all YAML files in
~/.config/dask/or/etc/dask/. This search path can be changed using the environment variableDASK_ROOT_CONFIGorDASK_CONFIG. See the Dask documentation for the complete list of locations and the lookup order. Example:$ cat example.yaml 'Distributed': 'scheduler': 'allowed-failures': 5Environment variables: Dask searches for all environment variables that start with
DASK_, then transforms keys by converting to lower-case and changing double-underscores to nested structures. See Dask documentation for the complete list of variables. Example:DASK_DISTRIBUTED__SCHEDULER__ALLOWED_FAILURES=5
Dask Client
With no configuration set, the dask-futures mode option initializes the Dask Client with the default arguments, which results in creating a LocalCluster in the background. The following are the YAML/environment variables that could be set to change the default behavior:
distributed.comm.timeouts.connect / DASK_DISTRIBUTED__COMM__TIMEOUTS__CONNECT
client-name / DASK_CLIENT_NAME
scheduler-address / DASK_SCHEDULER_ADDRESS
distributed.client.heartbeat / DASK_DISTRIBUTED__CLIENT__HEARTBEAT
distributed.client.scheduler-info-interval / DASK_DISTRIBUTED__CLIENT__SCHEDULER_INFO_INTERVAL
distributed.client.preload / DASK_DISTRIBUTED__CLIENT__PRELOAD
distributed.client.preload-argv / DASK_DISTRIBUTED__CLIENT__PRELOAD_ARGV
Recipe’s environment variables
Recipe has its own list of environment variables to make the configuration more complete and flexible. These environment variables are either missing from Dask’s configuration system or specific to the recipe system:
NSYS_DASK_SCHEDULER_FILE: Path to a file with scheduler information. It will be used to initialize the Dask Client.
NSYS_DIR: Path to the directory of Nsight Systems containing the target and host directories. The nsys executable and the recipe dependencies will be searched in this directory instead of the one deduced from the currently running recipe file path.
gfx_hotspot Recipe#
This recipe’s output is different from other recipes and is presented as a web application.
The output can be viewed by passing the --run-viewer argument to the recipe -
along with the further --show-viewer which will automatically open a web
browser to the report view.
Alternatively, a previously-executed gfx_hotspot recipe’s output can be
viewed by executing the run_viewer.py script from the recipe output folder.
For the best results, run the recipe on a report with resolved symbols.
Threading Analysis
In this tab, an overview of the multi-threading behavior of the target (most active) process is presented.
- Application Statistics:
This table shows the CPU and thread statistics for the target process.
- CPU Info:
This table shows information about the CPU hardware.
- Top 5 Processes CPU Utilisation:
This table shows the most active processes during the sample, to help detect situations where another process is interfering with the target process’s execution.
- Threading Health Check:
This table contains a list of very common CPU-bound application performance indicators. If the target application is GPU bound, the entire table will be shown in green. If it is CPU bound, then each row will be highlighted in green if the value is healthy, in yellow if it requires attention, and in red if it potentially indicates of a threading issue. For unhealthy metrics, the “warning” column will also show steps or investigation angles that may be considered in order to improve the result.
- Thread Utilisation:
This graph shows the process threads, ordered from most busy to least.
- Thread Concurrency:
This graph shows the percentage and amount of time an average graphic frame is running each number of threads concurrently. High percentage of low thread counts could indicate excessive serialization in the algorithm, where CPU work could be better parallelized by improving the use of multi-threading.
Hotspot Analysis
- In this tab, frames are selected in one of four methods:
Longest Frame time (Slow Frames)
Periodic time-based selection (Periodic Frames)
Frames with highest transfer activity (Bar1 Reads)
Frames with least GPU activity (GR Idle)
The report view then allows comparing the selected frames to each other and to the median frame in the same metric, helping identify the main differences and possible problem areas in each one.
- Overview:
These tables show the report overview as well as the frame selection method and other capture-wide statistics and general information. A shorthand list of the “Performance Issues” table for each frame is also shown.
- Frame Times:
This graph shows a sequence of the graphical frames (CPU time and GPU time derived from GPU Utilisation percentage per CPU frame time) ordered by their index. The selected frames are indicated and labelled. Clicking any of the indicated frames will set it as the left frame for comparison.
- Region / Compare to:
These controls allow selecting the two frames to be shown for comparison. “Periodic Frames” shows 10 sampled frames (with equally distributed indices), while the other three modes show the 5 frames with the highest value in the chosen metric and the median frame in the same metric. All information from this point onwards is shown per selected frame in each of the two columns, allowing for 1-to-1 comparison. Selecting the same frame for both controls will show just the single frame as the entire width of the view.
- Frame Info:
This table shows the frame duration and start time, the number of threads that were active during the frame, and the thread IDs of key threads in the frame processing operation which are important for determining likely performance issues.
- Performance Issues:
This table shows the key performance limiters and hotspots for the selected frame. Each indicator will have a breakdown of what indicators were present to call out the performance issue during this frame. These indicators are not necessarily the root cause of the problems in the region, but have been flagged for consideration.
- GPU Metrics:
This table shows the average or total (respectively) values of the GPU metrics collected during the frame time. If GPU Metrics were not collected, this table will not appear.
- System ETW Events (Windows only):
This chart shows a breakdown of the system process-reported ETW events during the frame. If WDDM trace and Custom ETW trace were not collected, this chart will not appear.
- DxgKrnl Events (Windows only):
This chart shows a breakdown of the DxgKrnl ETW provider events during the frame. If WDDM trace was not collected, this chart will not appear.
- CPU Thread Utilisation Time:
This graph shows the time spent inside each thread during the frame. The bars match the two selected frames, and the matching-colored line shows the total frame time. Clicking any of the columns in the graph will select that thread for the following elements in the report.
- Thread:
This control allows selecting the thread to be shown in the following views.
- Call stacks:
This control shows the sampled call stacks during the frame. Clicking a call stack frame will filter the view to only show call stacks containing this call stack frame, allowing to drill down into potential problem areas. The title of the control indicates the two modes selected for display, which can be switched with the two toggles in the top right of the control:
- Call stacks - Merged:
Merges all similar call stacks logically, regardless of when in the frame time the functions appeared. This is useful to see where the cumulative time is spent.
- Call stacks - Over Time:
Keeps call stacks ordered chronologically, so that repeated calls to the same function appear separately.
- Periodic Sampled Call stacks:
Only shows call stacks acquired by periodic sampling (matching the orange marks in Nsight Systems’s timeline view). This view provides a better statistical overview of where the frame time was spent.
- All Call stacks:
Shows periodic sampled call stacks as well as call stacks acquired from other sources such as call stacks from ETW events (Windows) and event-based sampling (Linux) (matching both the orange AND the grey marks in Nsight Systems’s timeline view).
- Modules in Sampled Call Stacks:
This graph shows the number of call stacks in the frame that include at least one call stack frame in a function belonging to each module. This helps identify which modules were the most active during the frame.
- ETW Events (Windows only):
This chart shows a breakdown of the thread-reported ETW events during the frame. If WDDM trace and Custom ETW trace were not collected, this chart will not appear.
- Context Switch Call Stacks:
This table shows a breakdown of the call stacks that led to context switches for the thread during the frame, indicating where the thread may have stalled. Hovering the mouse cursor over the “Name” column will show the full call stack for each entry.
- DX12 API / Vulkan API:
These tables show a breakdown of the graphical API functions that appeared in sampled call stacks. If DX12 / Vulkan trace were not collected, these tables will not appear.
- Known Symbols From Sampled Call Stacks:
This table shows a breakdown of known symbols that often cause performance issues, such as DX12’s CreateCommittedResource. If symbols were not resolved for the nsys-rep file, this table will not appear.
- PIX Markers (Windows only):
This table shows a breakdown of PIX marker ranges that contained sampled call stacks. If WDDM trace and DX11 / DX12 trace were not collected or the target application does not use PIX markers, this table will not appear.
gfx_api_analysis Recipe#
This recipe shows which D3D12, Vulkan, and OpenGL API functions take the most time. It compares typical call durations with slow calls and can locate chains of consecutive calls to one function.
Prerequisites
Collect the graphics API trace for the application. GPU workload data is optional. Frame selection uses the same markers and frame IDs as gfx_frame_analysis Recipe. Session analysis does not require frame markers.
For example, capture D3D12 calls with:
nsys profile --trace=dx12 <application>
Use vulkan or opengl instead for those API families.
Usage
Summarize all recorded API families and processes:
nsys recipe gfx_api_analysis --input report.nsys-rep
Summarize a CPU frame from a specific process:
nsys recipe gfx_api_analysis --input report.nsys-rep --pid 1234 --frame 42
Inspect the calls and chains of one function:
nsys recipe gfx_api_analysis --input report.nsys-rep --api dx12 --function UpdateTileMappings
Use --api to select dx12, vulkan, or opengl. Function selection
accepts an exact name or a substring that matches one function. An exact
match takes priority. CPU function selection does not filter GPU workload
names.
Output
The default notebook, analysis.ipynb, lets you select a report, API family,
and process by name and PID. CSV and JSON Lines output are also available
through --format. The notebook contains these sections:
- CPU API Summary:
Shows which functions consume the most accumulated call time, with counts and total, average, and maximum durations.
- CPU API Session Distribution:
Shows minimum, median, P95, P99, and maximum call durations to separate typical costs from occasional slow calls.
- GPU Workload Summary and GPU Workload Session Distribution:
Show separate duration statistics when GPU workload data is available.
- Selected Frame:
Shows the selected CPU frame’s process, ID, and time boundaries when
--frameis used.
- Call Chains by Thread:
Shows the longest chain and active-window occupancy for each thread when a function is selected.
- Ordered Calls:
Lists the selected function’s calls by time, with durations, thread IDs, gaps, and chain IDs.
For a selected frame, CPU and GPU summaries include events that overlap it. Full call durations describe latency. Time inside the frame is clipped to the frame boundaries and determines the frame percentage. Summed time across threads can exceed 100%. CPU and GPU distributions always cover the session for the selected process and API family. Function selection applies only to CPU results. Call timelines also cover the session.
Chains contain consecutive calls on the same thread, with a gap of up to
1 microsecond by default. Set --gap-threshold-ns to change this threshold.
Overlapping calls break a chain. The longest chain is selected by call count.
On each thread, the active window runs from the function’s first call start
to its latest call end. Occupancy is the fraction of this window covered by
calls, counting overlaps once. These CPU timings alone do not establish GPU
queue serialization.
GPU results do not match each CPU call to its GPU work. Calls without thread IDs contribute to session statistics but cannot be assigned to a process or included in a thread timeline.
For shared recipe instructions, see Advanced Report Analysis.
gfx_frame_analysis Recipe#
This recipe summarizes CPU frame timing and helps identify slow frames and timing patterns. It can also compare a selected frame’s ETW and WDDM activity with a baseline frame, and report how the application’s threads were scheduled.
Prerequisites
The recipe prioritizes Application Present Request NVTX markers and considers Reflex Present markers that extend the frame coverage before and after those markers. If neither is available, it uses the D3D11, D3D12, Vulkan, or OpenGL present calls, then DxgKrnl Present events, and then DXGI Present events.
Reports with ETW or WDDM data can also provide event comparisons for a selected frame. On Windows, collect this data with:
nsys profile --trace=wddm <application>
Thread scheduling requires CPU context switch data, and the OS wait tables also require OS wait calls. On Windows, these come from DX12 wait call tracing:
nsys profile --cpuctxsw=system-wide --trace=dx12,wddm --dx12-wait-calls=true <application>
On Linux, they come from OS runtime tracing:
nsys profile --cpuctxsw=system-wide --trace=osrt <application>
Thread callstacks require CPU sampling, and naming their symbols also requires symbol resolution, which is off by default on Windows. Enable it with:
nsys profile --sample=process-tree --resolve-symbols=true <application>
Usage
Summarize all processes with detected frames:
nsys recipe gfx_frame_analysis --input report.nsys-rep
Scan a frame from a specific process and compare it with a baseline frame:
nsys recipe gfx_frame_analysis --input report.nsys-rep --pid 1234 --frame 42
Use --pid to select a process and --frame to select a frame to scan.
The baseline is another frame from the same process used to compare ETW and
WDDM activity with the selected frame. Choose it with --baseline:
neighborhood_median(default): select the frame with the closest duration to the median of up to five frames before and five after the selected frame.session_median: select another frame with the closest duration to the median across the entire session for the selected process.previousornext: select the preceding or following frame.
For an even number of frames, baseline selection uses the higher of the two middle durations.
Output
By default, the recipe generates the Jupyter notebook frames.ipynb.
CSV and JSON Lines output are also available through --format.
The notebook contains the following sections. Select a report and process to explore the results.
- Process Summary:
Shows frame duration statistics, timing patterns, and slow-frame episode counts for the selected process.
- Frame Durations:
Plots each frame’s duration, with median and P95 reference lines. Use the chart and frame table to identify frames for further analysis.
- Slow-Frame Episodes:
Shows periods of sustained slow frames.
- Thread Scheduling:
Shows how long each thread of the selected process ran on a CPU, waited for a CPU, and was blocked, the reasons it was blocked, and its OS runtime wait calls. Covers the whole session, or with
--frame, the selected frame next to its baseline frame. Select a thread to narrow the tables down.
- CPU Concurrency:
Reports the busy core time, and the mean and maximum number of threads running at once, first for the selected process alone and then for the whole machine, over the same window as the thread scheduling tables.
- Thread Callstacks:
Shows the modules each thread’s callstacks ran through and the symbols they ended in, for periodic samples, for the stacks taken as a thread left a CPU, and on Windows for the stacks taken as a thread woke another one, over the same window as the thread scheduling tables.
Runs with --frame also show the selected frame, nearby frames, ETW and WDDM
event-rate comparisons with the baseline frame, and the threads that ran during
the selected frame but not during its baseline frame.
gpu_vram_usage_trace Recipe - Preview Feature#
This recipe analyzes VRAM usage patterns and statistics from Nsight Systems reports, helping identify and troubleshoot potential issues in GPU memory management.
Overview
The gpu_vram_usage_trace recipe generates an interactive Jupyter notebook that analyzes VRAM usage patterns, such as resource migrations between VRAM and system memory, and resource allocation and deallocation timing. This analysis helps identify memory management issues, optimization opportunities, and potential causes of performance degradation related to GPU memory usage.
Key Capabilities
The recipe provides insights into:
VRAM Usage Tracking: Per-frame monitoring of VRAM’s and SYSMEM’s usage, commitment, and budget across all GPU resources.
Memory Resource Details: Comprehensive information for all allocated GPU memory resources.
Resource Migrations: Analysis of resource migration patterns between VRAM and system memory.
Frame-by-Frame Analysis & Comparison: Detailed view of available resources at a specific point in time and comparison to another.
Debugging Context Integration: Correlation of memory resource usage with user-provided performance markers, resource debug names, and callstack information.
Use Cases
The recipe is particularly valuable for identifying and addressing the following scenarios (but not limited to these):
VRAM Exhaustion: Detecting when applications approach or exceed available VRAM limits, and identifying which resources are consuming the available VRAM.
Memory Thrashing: Identifying excessive resource migrations between VRAM and system memory.
Frame Spikes: Analyzing frames with abnormal performance due to VRAM usage and/or resource transitions.
Important Notes
Preview Feature: This recipe is a preview feature and may be subject to change in the near future.
Single Report Only: This recipe is intended for use with a single report. Using it with multiple reports may cause unexpected behavior.
Prerequisites
Windows (DirectX 12 and Vulkan): This recipe currently supports reports recorded on Windows, using either DirectX 12 or Vulkan.
Usage
This recipe requires that Nsight Systems reports be collected with WDDM tracing enabled.
Steps:
Collect an nsys-rep report with WDDM tracing enabled
With `nsys.exe` (CLI): Use the parameters –trace=wddm together with either –wddm-memory-trace=true or –wddm-additional-events=true.
With `nsys-ui.exe` (GUI): Enable the WDDM Trace collector, using either the “Collect WDDM memory trace” or “Extensive trace including Hardware Scheduling queues…” option.
Note
Optional: To collect additional debug name information on resources (affects the Resident Resources Details section), enable tracing of debug markers as follows:
For DirectX 12:
With `nsys-ui.exe` (GUI): Enable the DX12 Trace collector and the Trace Debug Markers option.
With `nsys.exe` (CLI): Adjust the trace argument:
--trace=wddm,dx12-annotations.
For Vulkan:
With `nsys-ui.exe` (GUI): Enable the Vulkan Trace collector and the Trace Debug Markers option.
With `nsys.exe` (CLI): Adjust the trace argument:
--trace=wddm,vulkan-annotations.
Run the recipe
nsys recipe gpu_vram_usage_trace --input [report file path]Open the generated notebook Open the produced stats.ipynb Jupyter notebook to view the interactive analysis.
Output
As the main output, the recipe generates an interactive Jupyter notebook
stats.ipynb with the following sections:
- Global Process and GPU Selectors:
- Interactive Timeline Charts:
- Resident Resources Diff Tables:
- Resident Resources Details Section:
- All Allocations Table:
Recommended Workflow
Start by selecting the process and GPU of interest using the global selectors at the top.
Identify and select suspicious frames, such as frames with unusual memory usage or with a high volume of memory transition events.
- Use the Resident Resources table to learn more about allocated resources during the selected frames:
Identify resources with significant memory usage.
Identify resources that have changed between the two points in time (transitioned between VRAM and SYSMEM, or were allocated/deallocated).
View resources’ allocation details, performance markers, and callstacks to help recognize the specific resources and possible problematic settings.
Use the “All Allocations” table to find suspicious resources, such as resources with excessive residency changes or other unusual characteristics.
nvtx_cpu_topdown Recipe#
This recipe calculates CPU Topdown methodology metrics for NVTX push/pop ranges based on collected PMU core events for NVIDIA CPUs featuring Arm cores. It can process multiple Nsight Systems reports.
Currently, the recipe supports NVIDIA Grace (TM) CPUs and NVIDIA DGX Spark (TM) CPUs.
We recommend using this recipe after running the collect_cpu_topdown.sh
script, which simplifies collecting all PMU core event and metric data needed to
perform a CPU Topdown analysis of the workload’s CPU performance. For more
details on this script, refer to the Arm Topdown Analysis section.
If PMU core events other than those required by Topdown are collected, the recipe will calculate available CPU metrics based on them and display those metrics in the output.
Use Case
The recipe is most useful when the following conditions are met:
The application runs on CPU cores supported by the recipe.
The application is instrumented with NVTX push/pop ranges.
NVTX range spans a specific CPU algorithm / code section that does not make syscalls or calls to other libraries whose functions take significant time to execute.
NVTX ranges with the same name are used to represent the same workload across all threads and all repetitions.
The duration of the NVTX range is 5 ms or longer to obtain more accurate results.
In systems with heterogeneous CPU cores, the NVTX range executed on a given core type remains consistent across runs, ensuring a reliable view of CPU metrics for that range on that core type. For example, to achieve stable runs, you can pin the process to specific CPU cores using
taskset.
Note
For the case of NVTX ranges from multiple threads, only the NVTX ranges from
either the main thread (default) or the thread specified via --thread-name
will be processed.
Usage
[1] mkdir reports && cd reports
[2] <path to target-linux-sbsa-armv8>/CpuProfiling/collect_cpu_topdown.sh ./myApp
[3] nsys recipe nvtx_cpu_topdown --input .
This step creates a new directory to store the reports. We recommend using an empty new directory, because the
collect_cpu_topdown.shscript overwrites the output files and does not currently allow customization of file names.This step creates several report files: cpu-td1.nsys-rep, cpu-branch-ipc.nsys-rep, etc.
Note
Note that since multiple reports are created, this step can take significant time to complete.
This step runs the recipe, uses all reports in the current directory as the input, and produces a
.ipynbJupyter notebook,.parquetand.csv(ifcsvis included in--format) files as the output.
Output
As the main output, the recipe generates the Jupyter notebook
nvtx_cpu_topdown.ipynb with the following sections:
- NVTX Summary for Heterogeneous CPU Cores:
Displays a summary of NVTX ranges compiled from Nsight Systems reports provided to the recipe.
For a report selected from the drop-down menu, the section shows each NVTX range (in call stack order) with its instance count, median* instance duration, and CPU time aggregated across heterogeneous cores, as well as CPU time per core type - both related to the NVTX instance with the median* duration.
Note: median* is defined as the middle value in the sorted list. For an even number of elements, it is the second of the two middle values.
Note
This section is available only for data collected from heterogeneous CPU cores. For these cores, the remaining sections apply to each CPU core type individually and can be toggled using the
Select CPU Coredrop-down menu.
- Warnings:
Displays warnings generated during recipe execution and related to the entire recipe output (or to the portion of it specific to a given CPU core type). If there are no warnings, this section is not displayed.
- NVTX Summary:
Displays a summary of NVTX ranges compiled from Nsight Systems reports provided to the recipe.
For a report selected from the drop-down menu, the section shows each NVTX range (in call stack order) with its instance count, median duration with median absolute deviation, median CPU time with median absolute deviation, and relevant notes.
If NVTX ranges are filtered out, they are grayed out in the table, and a note is displayed in the Notes column for the corresponding range. The following ranges are candidates to be filtered out:
Ranges that contain fewer than 3 PMU samples in at least one Nsight Systems report provided to the recipe.
Ranges that are not present in at least one Nsight Systems report provided to the recipe.
If NVTX ranges are not stable across some of the reports, the section will display a warning next to the unstable data and a note in the Notes column for the corresponding range.
- CPU Topdown Methodology Metrics:
Presents the metric results of the CPU Topdown methodology for the selected NVTX range.
For the range name selected from the drop-down menu, the most appropriate NVTX range instance is identified from the Nsight Systems reports as follows:
For data collected from heterogeneous CPU cores: The NVTX range instance with the median* CPU time is selected from each report.
Otherwise: The NVTX range instance with the median* duration is selected from the first report (displayed by default in the NVTX Summary section). The corresponding instance index is then used to extract data from subsequent reports.
Note: median* is defined as the middle value in a sorted list. For an even number of elements, it is the second of the two middle values.
The section shows the following tables:
Topdown Level 1 metrics
Frontend Bound metrics
Backend Bound metrics
Bad Speculation metrics
Retiring metrics
Miscellaneous metrics
Each table is displayed only when the required data is available.
- Report Summary:
Displays information about the Nsight Systems report files given to the recipe for input, as well as: the PMU core events collected in each specific report, and the CPU core metrics computed for each specific report.
file_access_sum Recipe#
This recipe analyzes file access patterns and I/O performance statistics from one or more Nsight Systems reports, aggregating data across processes and machines.
Overview
The file_access_sum recipe generates an interactive Jupyter notebook that analyzes POSIX VFS (Virtual File System) function calls captured during profiling sessions. This analysis helps identify I/O bottlenecks, optimization opportunities, and file access patterns that could impact application performance.
Key Capabilities
The recipe provides insights into:
File Access Patterns: Breakdown of read-only, write-only, and read-write file access patterns.
Performance Metrics: Total bytes transferred, operation counts, and average I/O sizes per operation.
Cross-Process Analysis: File access patterns across multiple hosts, processes, and threads.
Temporal Analysis: Distribution of CPU time by operation type.
Hotspot Identification: Top files by read/write volume and operation frequency.
Performance Recommendations: Automated detection of potentially inefficient I/O patterns with actionable suggestions.
Use Cases
The recipe is particularly valuable for identifying and addressing the following scenarios (but not limited to these):
I/O Patterns: Understanding application I/O behavior to uncover usage trends and inefficiencies.
Small I/O Operations: Detection of frequent small read/write operations that could benefit from batching.
Caching Opportunities: Identification of frequently accessed read-only files that are candidates for local caching.
Metadata Contention: Identifying cases where frequent metadata operations by one process may cause contention, impacting storage access for other processes.
System File Noise: Filtering out system files (/dev/, /sys/, etc.) to focus on application-relevant I/O.
Prerequisites
This recipe requires that Nsight Systems reports be collected with specific tracing parameters:
--trace=osrt- Enables OS Runtime API tracing--osrt-file-access=true- Enables file access trackingOptional: To enable tracing of MPI rank information, use
--trace=mpialong with either--mpi-impl=openmpior--mpi-impl=mpich.Optional: To enable the NVTX range correlation table, instrument your application code with NVTX ranges. See Marking and Labeling Regions in the User Guide.
Usage
[1] Create a reports folder.
[2] Collect nsys-rep reports, using '--trace=osrt' and '--osrt-file-access=true' parameters, and save them to the reports folder.
[3] Run the recipe, using 'nsys recipe file_access_sum --input [reports folder path]'.
Output
As the main output, the recipe generates an interactive Jupyter notebook
file_access_stats.ipynb with the following sections:
- File Access Summary Table:
- Hottest Read/Write Files Tables:
- All Files Table:
- Read/Write Access Histogram:
- CPU Time Graph:
- Operations Count Chart:
- Performance Analysis:
- NVTX Ranges Analysis:
- Recommended Workflow
Start by setting file path ignore patterns to exclude system files from analysis.
Focus on application-specific files during the analysis by using regex filtering.
View the Nsight Systems report file alongside this analysis to gain a deeper understanding of the application’s behavior.
s3_access_sum Recipe#
This recipe analyzes S3 access patterns and I/O statistics from one or more Nsight Systems’ reports, aggregating data across processes and hosts.
Overview
The s3_access_sum recipe generates an interactive Jupyter notebook that analyzes S3 operations captured during profiling sessions. It provides a statistical overview of which buckets and objects were accessed and of access patterns.
Key Capabilities
The recipe provides insights into:
Bucket-Level Summary: Aggregated access statistics per S3 bucket, including object counts, download/upload bytes, and operation counts.
Object Access Patterns: Breakdown of download-only, upload-only, and mixed-access objects.
Hotspot Identification: Top objects by download/upload volume and operation frequency.
Cross-Process Analysis: S3 access patterns across multiple hosts, processes, and threads.
NVTX Range Correlation: S3 activity correlated with user-defined NVTX ranges, such as access-time statistics and I/O volume.
Use Cases
The recipe is particularly valuable for identifying and addressing the following scenarios (but not limited to these):
Hot Buckets and Objects: Identifying which S3 buckets and objects carry the most traffic to focus optimization efforts.
Small or Frequent Operations: Detecting frequent small S3 transfers that could benefit from batching or prefetching.
Application Phase Correlation: Using NVTX ranges to understand which application phases drive the most S3 I/O.
Multi-Rank Comparison: Comparing S3 usage across MPI ranks or hosts to identify load imbalances.
Supported S3 Client Libraries
The recipe counts the following operations from traced S3 client libraries:
AWS CRT (aws-c-s3):
GetObject,PutObject,CopyObjectAWS C++ SDK (aws-cpp-sdk-s3):
GetObject,PutObject,CopyObject,DeleteObject,HeadObject,CreateMultipartUpload,UploadPart,CompleteMultipartUpload,AbortMultipartUpload,ListObjects,ListObjectsV2,CreateBucketBoto3:
get_object,put_object,head_object,upload_part,create_multipart_upload,abort_multipart_upload,generate_presigned_url,generate_presigned_postS3TorchConnector:
SequentialS3Reader.read,S3Writer.writeTensorflow-io:
read,readline,readlines,write,copy,copy_v2viatf.io.gfilewiths3://paths
For more information about S3 tracing capabilities and setup, refer to S3 Trace in the User Guide.
Prerequisites
This recipe requires that Nsight Systems reports be collected with specific tracing parameters:
--trace=s3- Enables S3 operation tracing.This recipe will also work with
--trace=s3-verbose, but does not use any additional data from it.
Optional: To enable tracing of MPI rank information, use
--trace=mpialong with either--mpi-impl=openmpior--mpi-impl=mpich.Optional: To enable the NVTX range correlation table, instrument your application code with NVTX ranges. See Marking and Labeling Regions in the User Guide.
Usage
[1] Create a reports folder.
[2] Collect nsys-rep reports, using '--trace=s3' parameter, and save them to the reports folder.
[3] Run the recipe, using 'nsys recipe s3_access_sum --input [reports folder path]'.
Output
As the main output, the recipe generates an interactive Jupyter notebook
s3_access_stats.ipynb with the following sections:
- Bucket Access Summary:
- Objects Access Summary:
- Hottest Downloaded Objects:
- All Objects Table:
- S3 Access Statistics for NVTX Ranges:
Recipe Catalog and Sample Results#
Post-Collection Analysis enables deep insights into GPU application performance through a curated set of analysis recipes. Each recipe generates specialized visualizations and data summaries from profiling data collected during application execution.
This catalog documents all available recipes, their output formats, and sample results from profiling Llama-70B inference on 8 NVIDIA H100 GPUs using the NVIDIA Dynamo framework.
The recipes are organized by functionality:
CUDA API Recipes: Analyze CUDA kernel launches, memory operations, and synchronization.
GPU Kernel Recipes: Examine kernel execution times, duration distribution, and pacing.
NVTX Recipes: Analyze application-level instrumentation ranges.
Communication Recipes: Study multi-GPU communication patterns, including NCCL and NVLink.
System Recipes: Review OS runtime and I/O operations.
Comparison Recipes: Compare results between different profiling runs.
Each recipe includes sample data tables and visualization mockups representative of real profiling results.
CUDA API Recipes#
CUDA API Summary
Summarizes CUDA API calls by frequency and execution time. Identifies the most time-consuming CUDA operations.
Output Format: Summary table, top items bar chart, and duration distribution box plot.
API Function |
Call Count |
Total Time (us) |
Avg Time (us) |
Max Time (us) |
|---|---|---|---|---|
cudaMemcpy |
2850 |
125400.5 |
43.98 |
650.2 |
cudaLaunchKernel |
5600 |
85300.2 |
15.23 |
420.5 |
cudaDeviceSynchronize |
280 |
34520.0 |
123.29 |
850.3 |
cudaMemset |
450 |
12350.8 |
27.45 |
180.1 |
cudaMalloc |
125 |
8900.3 |
71.20 |
340.2 |
CUDA Synchronization APIs
Advisory analysis identifying synchronization bottlenecks and suggesting optimization strategies.
Output Format: Advisory table with severity levels and recommendations.
Issue |
Severity |
Count |
Total Impact (us) |
Recommendation |
|---|---|---|---|---|
Implicit Synchronization |
HIGH |
45 |
28500 |
Use async transfers with events |
Unnecessary cudaDeviceSynchronize |
MEDIUM |
28 |
12300 |
Replace with stream events |
Blocking Memory Transfers |
MEDIUM |
156 |
45200 |
Use pinned memory for H2D |
CUDA GPU Kernel Recipes#
CUDA GPU Kernel Summary
Comprehensive summary of GPU kernels executed. Shows kernel names, execution counts, and timing statistics.
Output Format: Summary table, top kernels bar chart, and duration distribution box plot.
Kernel Name |
Launch Count |
Total Time (us) |
Avg Time (us) |
Max Time (us) |
|---|---|---|---|---|
volta_sgemm_128x64_nn |
8400 |
2845320.0 |
338.73 |
4250.2 |
attention_forward_kernel |
2800 |
1950200.5 |
696.50 |
2850.3 |
layernorm_forward_kernel |
2800 |
892340.2 |
318.69 |
1620.5 |
fused_mlp_kernel |
2800 |
756200.1 |
270.07 |
1450.2 |
nccl_AllReduce_Ring_LL_Sum_f16 |
560 |
680450.3 |
1214.20 |
3200.1 |
CUDA GPU Kernel Duration Histogram
Distribution of kernel execution durations. Shows frequency of kernels across different execution time ranges.
Output Format: Histogram with distribution statistics and percentile markers.
Duration Range (us) |
Frequency |
Percentage |
Cumulative % |
|---|---|---|---|
100-500 |
1240 |
8.2% |
8.2% |
500-1000 |
2850 |
18.7% |
26.9% |
1000-2000 |
4200 |
27.5% |
54.4% |
2000-4000 |
3950 |
25.9% |
80.3% |
4000+ |
2860 |
18.7% |
99.0% |
CUDA GPU Kernel Pacing
Consistency analysis of kernel execution times across iterations and GPU ranks. Identifies pacing irregularities.
Output Format: Multi-line chart showing kernel timing per iteration across all GPU ranks.
Iteration |
Rank 0 (us) |
Rank 1 (us) |
Rank 2 (us) |
Rank 3 (us) |
Avg Variance |
|---|---|---|---|---|---|
0 |
1025.3 |
1048.2 |
1032.5 |
1041.8 |
8.6 |
1 |
1032.8 |
1055.3 |
1038.2 |
1049.5 |
8.9 |
2 |
1038.5 |
1062.1 |
1044.8 |
1055.2 |
9.2 |
3 |
1035.2 |
1058.9 |
1040.5 |
1051.8 |
8.7 |
CUDA Memory Operations Recipes#
CUDA GPU MemOps Summary (by Size)
Memory operations summarized by size. Identifies which memory operations transfer the most data. Memory decompression sizes are compressed source bytes.
Output Format: Summary table, size distribution box plot, and statistics line graph.
CUDA GPU MemOps Summary (by Time)
Memory operations sorted by execution time. Identifies most time-consuming memory transfers.
Output Format: Summary table, top operations bar chart, and duration distribution box plot.
CUDA Synchronous Memcpy
Advisory for synchronous memory copies that may impact performance.
Output Format: Advisory table with impact analysis and optimization suggestions.
Transfer Pattern |
Count |
Total Time (us) |
Severity |
Suggestion |
|---|---|---|---|---|
Pageable H2D |
1250 |
145200 |
HIGH |
Use cuda-pinned memory |
Pageable D2H |
1100 |
128500 |
HIGH |
Implement async with streams |
Device Sync |
280 |
34500 |
MEDIUM |
Use cudaEventSynchronize |
GPU Utilization Recipes#
GPU Metrics Utilization Summary
High-level metrics showing GPU resource utilization, including SM, Memory, Tensor Cores, and related metrics.
Output Format: Summary table, metric utilization bar chart, and distribution box plot.
Metric |
Avg Util (%) |
Max Util (%) |
Peak Time (ms) |
Stalls |
|---|---|---|---|---|
SM Utilization |
78.5 |
95.2 |
245.3 |
12 |
Memory Utilization |
65.3 |
88.1 |
238.5 |
28 |
L1 Cache Hit Rate |
82.4 |
96.1 |
250.2 |
5 |
L2 Cache Hit Rate |
74.2 |
91.3 |
242.8 |
18 |
Tensor Utilization |
81.5 |
94.7 |
246.1 |
8 |
GPU Metric Utilization Heatmap
Time-series heatmap showing metric utilization across GPU ranks over profiling duration.
Output Format: 2D heatmap with GPU ranks on the y-axis, time bins on the x-axis, and utilization percentage color-coded.
GPU Rank |
T0-T100 (%) |
T100-T200 (%) |
T200-T300 (%) |
T300-T400 (%) |
Avg (%) |
|---|---|---|---|---|---|
Rank 0 |
82 |
85 |
79 |
84 |
82.5 |
Rank 1 |
80 |
83 |
78 |
82 |
80.8 |
Rank 2 |
81 |
84 |
80 |
85 |
82.5 |
Rank 3 |
79 |
82 |
77 |
81 |
79.8 |
Graphics VRAM Usage
Memory allocation and deallocation trace showing VRAM consumption over time.
Output Format: Time-series line chart with peak VRAM markers and allocation events.
Time (ms) |
Allocated (MB) |
Free (MB) |
Total GPU Mem (MB) |
Peak Usage (%) |
|---|---|---|---|---|
0 |
2048 |
77952 |
80000 |
2.6% |
100 |
8192 |
71808 |
80000 |
10.2% |
200 |
24576 |
55424 |
80000 |
30.7% |
300 |
38912 |
41088 |
80000 |
48.6% |
400 |
42048 |
37952 |
80000 |
52.6% |
GPU Time Utilization
Advisory on GPU idle time, gaps, and utilization efficiency.
Output Format: Advisory table with gap analysis and improvement suggestions.
GPU |
Active Time (ms) |
Idle Time (ms) |
Util (%) |
Largest Gap (us) |
Recommendation |
|---|---|---|---|---|---|
GPU 0 |
485.2 |
14.8 |
96.8% |
2850 |
Excellent utilization |
GPU 1 |
478.5 |
21.5 |
95.7% |
3200 |
Minor gaps detected |
GPU 2 |
475.3 |
24.7 |
95.1% |
3850 |
Check kernel dependencies |
GPU 3 |
472.8 |
27.2 |
94.6% |
4100 |
Optimize host-device sync |
GPU Gaps
Identifies periods of GPU inactivity and analyzes root causes.
Output Format: Gap summary table with timeline and impact analysis.
Gap ID |
Start (us) |
Duration (us) |
GPU |
Probable Cause |
Impact |
|---|---|---|---|---|---|
Gap_001 |
125340 |
2850 |
GPU 0 |
Host-Device Sync |
Kernel stall |
Gap_002 |
245820 |
3200 |
GPU 1 |
Memory Transfer |
Pipeline flush |
Gap_003 |
385500 |
1950 |
GPU 2 |
Context Switch |
Minor impact |
Gap_004 |
458200 |
4100 |
GPU 3 |
Host Queue Latency |
Kernel delay |
NVTX Range Recipes#
NVTX Range Summary
Summary of NVIDIA Tools eXtension (NVTX) instrumentation ranges in application code.
Output Format: Summary table, top ranges bar chart, and duration distribution box plot.
Range Name |
Count |
Total Time (us) |
Avg Time (us) |
Max Time (us) |
|---|---|---|---|---|
forward_pass |
280 |
1850320.5 |
6608.29 |
8520.3 |
attention_compute |
2800 |
950200.2 |
339.36 |
2850.1 |
mlp_compute |
2800 |
756300.8 |
270.11 |
1520.2 |
backward_pass |
280 |
1420500.3 |
5073.21 |
7250.3 |
parameter_update |
280 |
385200.1 |
1375.71 |
2100.5 |
NVTX Pacing
Pacing consistency analysis of NVTX ranges across iterations.
Output Format: Multi-line chart showing range timing per iteration across GPU ranks.
Iteration |
Rank 0 (us) |
Rank 1 (us) |
Rank 2 (us) |
Rank 3 (us) |
Std Dev |
|---|---|---|---|---|---|
0 |
6580.2 |
6625.8 |
6590.5 |
6612.3 |
15.8 |
1 |
6598.5 |
6642.1 |
6608.2 |
6630.5 |
16.2 |
2 |
6612.3 |
6658.9 |
6622.8 |
6645.2 |
16.8 |
3 |
6625.8 |
6672.5 |
6638.5 |
6660.1 |
17.3 |
NVTX GPU Projection Summary
GPU utilization projection for NVTX ranges. Shows estimated GPU time within each range.
Output Format: Summary table with GPU projection metrics, bar chart, and box plot.
Range Name |
GPU Time (us) |
Total Time (us) |
GPU % |
Kernel Calls |
|---|---|---|---|---|
forward_pass |
1750280.3 |
1850320.5 |
94.6% |
2850 |
attention_compute |
905200.5 |
950200.2 |
95.3% |
2800 |
mlp_compute |
725300.2 |
756300.8 |
95.9% |
2800 |
backward_pass |
1385400.1 |
1420500.3 |
97.5% |
2800 |
parameter_update |
370200.8 |
385200.1 |
96.1% |
280 |
NVTX GPU Projection Pacing
Consistency of GPU utilization for NVTX ranges across iterations and ranks.
Output Format: Multi-line pacing chart with GPU projection metrics.
NVTX GPU Projection Trace
Detailed trace showing compute versus communication overlap within NVTX ranges.
Output Format: Overlap matrix and timeline visualization.
Range |
Compute (us) |
Comm (us) |
Overlap (us) |
Overlap % |
|---|---|---|---|---|
forward_pass |
1650280 |
185320 |
152850 |
82.5% |
attention_compute |
850200 |
120200 |
98560 |
82.0% |
mlp_compute |
725300 |
95300 |
78520 |
82.5% |
backward_pass |
1285400 |
210500 |
172450 |
81.9% |
parameter_update |
285200 |
65200 |
52850 |
81.0% |
Communication & NCCL Recipes#
NCCL Summary
Full-run summary of NVIDIA Collective Communications Library (NCCL) communication. The report shows communicators, ranks, operations, message-size buckets, datatypes, and rank-local operation counts and costs. With advanced NCCL tracing, collective and P2P GPU operations are summarized directly.
Output Format: Communicator/rank table, operation/datatype/message-size summary table, operation count and timing box plots, and rank-local count and timing box plots.
Operation |
Count |
Total Time (us) |
Avg Time (us) |
Max Time (us) |
|---|---|---|---|---|
AllReduce |
560 |
680450.3 |
1214.20 |
3200.1 |
ReduceScatter |
560 |
520300.5 |
928.75 |
2850.2 |
AllGather |
560 |
485200.2 |
866.07 |
2520.3 |
Broadcast |
280 |
185300.8 |
661.79 |
1850.2 |
Barrier |
560 |
125400.1 |
223.93 |
620.5 |
NCCL GPU Projection Summary
GPU utilization during NCCL operations. Breaks down compute versus communication time. For advanced NCCL tracing, use NCCL Summary instead because GPU projection is unnecessary.
Output Format: Summary table with GPU projection, bar chart, and box plot.
NCCL Op |
GPU Time (us) |
Total Time (us) |
GPU Util % |
Rank Participation |
|---|---|---|---|---|
AllReduce |
645200.2 |
680450.3 |
94.8% |
8 |
ReduceScatter |
485300.1 |
520300.5 |
93.3% |
8 |
AllGather |
450200.3 |
485200.2 |
92.8% |
8 |
Broadcast |
175300.5 |
185300.8 |
94.6% |
8 |
Barrier |
118200.2 |
125400.1 |
94.3% |
8 |
NCCL GPU Time Utilization Heatmap
Time-series heatmap of GPU utilization during NCCL operations across all ranks.
Output Format: 2D heatmap with GPU ranks on the y-axis and time bins on the x-axis.
GPU Rank |
T0 (%) |
T1 (%) |
T2 (%) |
T3 (%) |
Avg (%) |
|---|---|---|---|---|---|
Rank 0 |
92 |
94 |
93 |
95 |
93.5 |
Rank 1 |
90 |
92 |
91 |
93 |
91.5 |
Rank 2 |
91 |
93 |
92 |
94 |
92.5 |
Rank 3 |
89 |
91 |
90 |
92 |
90.5 |
NCCL GPU Overlap Trace
Overlap analysis between communication and compute during NCCL operations.
Output Format: Overlap matrix showing communication overlap percentages between ranks.
Op |
Overlap % |
Compute Time (us) |
Comm Time (us) |
Concurrent Time (us) |
|---|---|---|---|---|
AllReduce |
78.5 |
620300 |
145200 |
113850 |
ReduceScatter |
76.2 |
450200 |
98500 |
75000 |
AllGather |
74.8 |
425300 |
85200 |
63800 |
Broadcast |
72.5 |
160300 |
32500 |
23600 |
Barrier |
65.3 |
108200 |
18500 |
12100 |
Network & Storage Recipes#
Network Traffic Summary
Summary of network I/O operations and traffic patterns.
Output Format: Summary table, traffic distribution bar chart, and timing box plot.
Interface |
Packets |
Bytes (MB) |
Avg Latency (us) |
Max Latency (us) |
|---|---|---|---|---|
eth0 |
85230 |
2048.5 |
125.3 |
850.2 |
eth1 |
75600 |
1820.3 |
130.5 |
920.3 |
InfiniBand |
125400 |
3200.8 |
85.2 |
520.1 |
Network Devices Traffic Heatmap
Time-series heatmap showing network device utilization over profiling duration.
Output Format: 2D heatmap with network devices on the y-axis and time bins on the x-axis.
Device |
T0 (Mb/s) |
T1 (Mb/s) |
T2 (Mb/s) |
T3 (Mb/s) |
Avg (Mb/s) |
|---|---|---|---|---|---|
eth0 |
2500 |
2350 |
2400 |
2450 |
2425 |
eth1 |
2200 |
2100 |
2150 |
2200 |
2162 |
InfiniBand |
3500 |
3400 |
3450 |
3550 |
3475 |
AWS Metrics Heatmap
CloudWatch metrics heatmap for AWS-deployed systems.
Output Format: Heatmap of AWS metrics, including CPU, Memory, and Network, over time.
Metric |
T0 (%) |
T1 (%) |
T2 (%) |
T3 (%) |
Avg (%) |
|---|---|---|---|---|---|
CPU Util |
75 |
78 |
76 |
79 |
77.0 |
Memory Util |
62 |
64 |
63 |
65 |
63.5 |
Network Util |
58 |
60 |
59 |
62 |
59.8 |
Storage Metrics Heatmap
Storage I/O and utilization metrics over profiling duration.
Output Format: Heatmap showing storage device utilization percentages.
Storage Device |
T0 (%) |
T1 (%) |
T2 (%) |
T3 (%) |
Avg (%) |
|---|---|---|---|---|---|
NVMe-0 |
45 |
48 |
46 |
50 |
47.25 |
NVMe-1 |
42 |
44 |
43 |
46 |
43.75 |
HDD-0 |
28 |
30 |
29 |
32 |
29.75 |
System & OS Runtime Recipes#
OS Runtime Summary
Summary of OS runtime operations including system calls and I/O operations.
Output Format: Summary table, operation distribution bar chart, and timing box plot.
OS Operation |
Call Count |
Total Time (us) |
Avg Time (us) |
Max Time (us) |
|---|---|---|---|---|
read() |
1850 |
28500.3 |
15.41 |
250.2 |
write() |
1620 |
24300.5 |
15.01 |
220.3 |
open() |
450 |
8500.2 |
18.89 |
120.5 |
close() |
425 |
7200.1 |
16.94 |
95.2 |
mmap() |
120 |
4200.8 |
35.01 |
180.3 |
Linux File Access Summary
Specialized analysis of file system access patterns during profiling.
Output Format: File access statistics table with I/O patterns.
File/Path |
Access Count |
Bytes Read |
Bytes Written |
Total Time (us) |
|---|---|---|---|---|
/data/model.bin |
250 |
2048000 |
0 |
18500.2 |
/tmp/cache |
1850 |
512000 |
256000 |
12300.5 |
/proc/stat |
450 |
0 |
0 |
2800.3 |
/dev/null |
600 |
0 |
1024000 |
8200.1 |
NVLink & MPI Recipes#
NVLink Network Throughput Summary
Summary of NVLink GPU-to-GPU communication throughput and efficiency.
Output Format: Summary table, throughput distribution bar chart, and timing box plot.
Link Pair |
Transfers |
Total Data (MB) |
Total Time (us) |
Throughput (GB/s) |
|---|---|---|---|---|
GPU0-GPU1 |
450 |
2048.5 |
2350.3 |
871.0 |
GPU1-GPU2 |
450 |
2048.3 |
2340.2 |
875.1 |
GPU2-GPU3 |
450 |
2048.1 |
2360.5 |
868.2 |
GPU4-GPU5 |
450 |
2048.2 |
2345.8 |
873.1 |
GPU6-GPU7 |
450 |
2048.0 |
2355.2 |
869.8 |
MPI Summary
Summary of Message Passing Interface (MPI) operations and communication patterns.
Output Format: Summary table, operation distribution bar chart, and timing box plot.
MPI Operation |
Call Count |
Total Data (MB) |
Total Time (us) |
Avg Time (us) |
|---|---|---|---|---|
MPI_Allreduce |
560 |
520.0 |
680450.3 |
1214.20 |
MPI_Reduce |
280 |
280.5 |
350200.2 |
1250.71 |
MPI_Bcast |
280 |
256.2 |
185300.5 |
661.79 |
MPI_Send |
1850 |
850.3 |
125400.2 |
67.81 |
MPI_Recv |
1850 |
850.3 |
128200.1 |
69.30 |
MPI and GPU Time Utilization Heatmap
Correlation heatmap showing GPU utilization during MPI operations across ranks.
Output Format: 2D heatmap with MPI ranks on the y-axis and time bins on the x-axis.
MPI Rank |
T0 (%) |
T1 (%) |
T2 (%) |
T3 (%) |
Avg (%) |
|---|---|---|---|---|---|
Rank 0 |
88 |
90 |
89 |
92 |
89.75 |
Rank 1 |
86 |
88 |
87 |
90 |
87.75 |
Rank 2 |
87 |
89 |
88 |
91 |
88.75 |
Rank 3 |
85 |
87 |
86 |
89 |
86.75 |
UCX and GPU Time Utilization Heatmap
Heatmap showing GPU utilization during UCX (Unified Communication X) operations.
Output Format: 2D heatmap with UCX operation types on the y-axis and time bins on the x-axis.
UCX Op Type |
T0 (%) |
T1 (%) |
T2 (%) |
T3 (%) |
Avg (%) |
|---|---|---|---|---|---|
UCX_Put |
82 |
84 |
83 |
86 |
83.75 |
UCX_Get |
79 |
81 |
80 |
83 |
80.75 |
UCX_Atomic |
75 |
77 |
76 |
79 |
76.75 |
Comparison & Diff Recipes#
Statistics Diff
Side-by-side comparison of profiling statistics from two different runs.
Output Format: Comparison table with Run A versus Run B metrics and percentage differences.
Metric |
Run A |
Run B |
Difference |
Change % |
|---|---|---|---|---|
Total GPU Time (s) |
485.2 |
492.5 |
+7.3 |
+1.5% |
Avg Kernel Time (us) |
338.73 |
341.25 |
+2.52 |
+0.7% |
Memory Bandwidth (GB/s) |
750.2 |
745.8 |
-4.4 |
-0.6% |
NCCL Overhead (ms) |
125.3 |
128.5 |
+3.2 |
+2.6% |
Peak VRAM (MB) |
42048 |
42512 |
+464 |
+1.1% |
Recipe Composite Tables#
When writing a recipe, you use CompositeTable enum values to request
pre-processed DataFrames. These composite tables join, rename, and resolve
raw parquet export tables into analysis-ready DataFrames, so you don’t
have to do the merging and ID-to-string resolution yourself.
Available Composite Tables#
CompositeTable |
Description |
|---|---|
|
All GPU activities (kernels, memcpy, memset, mem_decompress) concatenated into one table. Columns are the union of the raw CUPTI activity columns. Use this when you want a broad view of everything the GPU did. |
|
Same as |
|
GPU activities + CPU runtime API calls, correlated by |
|
Runtime + kernels only (no memcpy/memset). Like |
|
Kernels only with |
|
Kernels only with |
|
NVTX ranges with |
|
NCCL communication operations parsed from NVTX events. Includes
|
|
NCCL API-level view showing call durations, parsed from NVTX events. |
|
NCCL GPU-level operations view parsed from NVTX events. Includes collective and P2P operations with common payload fields and normalized operation identity columns. |
|
NCCL collective GPU operation view with collective-specific payload
fields such as |
|
Network interface metrics joined with device info. Columns include
|
|
InfiniBand switch port metrics with |
|
All MPI event tables (P2P, collectives, start/wait, other) concatenated
and sorted by start time. |
|
All UCX event tables (submit, progress, general) concatenated and sorted
by start time. |
|
GPU hardware metrics pivoted so each metric name becomes its own column
(e.g., |
|
CPU performance counter events enriched with |
|
Generic metric events with type and field IDs resolved to
|
Choosing the Right CUDA Table#
There are several CUDA-related composite tables. Here is when to use each one:
Table |
Includes kernels |
Includes memcpy/memset |
Includes runtime API |
When to use |
|---|---|---|---|---|
|
Yes |
Yes |
No |
Overview of all GPU activity |
|
Yes |
Yes |
No |
Same, but with CUDA graph correlation |
|
Yes |
No |
No |
Kernel-only analysis with resolved names |
|
Yes |
No |
No |
Kernel-only analysis without name resolution |
|
Yes |
Yes |
Yes |
Full CPU-to-GPU pipeline analysis |
|
Yes |
No |
Yes |
Kernel launch overhead analysis with resolved names |
Output Columns#
Each composite table produces a DataFrame with specific columns. The columns
depend on which raw tables were joined and what transformations were applied.
Also note: when tables are loaded through DataService, bit-field
decomposition may add derived columns such as pid, tid, and gpuId.
CUDA_GPU
correlationId start end globalPid deviceId contextId greenContextId streamId pid
CUDA_GPU_GRAPH
correlationId start end globalPid deviceId contextId greenContextId streamId graphNodeId graphId pid
graphNodeId appears on kernel/memcpy/memset rows, and graphId appears on
graph-trace rows.
CUDA_KERNEL
correlationId globalPid start end deviceId shortName mangledName demangledName pid
The *Name columns contain resolved strings (not integer IDs).
CUDA_KERNEL_RAW
correlationId globalPid start end deviceId shortName mangledName demangledName pid
The *Name columns contain integer IDs. The
StringIds table maps them to names.
CUDA_COMBINED
start end globalTid name correlationId gpu_start gpu_end pid tid deviceId contextId greenContextId streamId
start/end are the CPU-side runtime API call times. gpu_start/gpu_end
are the GPU-side activity times. name is the resolved API function name
(e.g., "cudaLaunchKernel"). pid and tid are extracted from
globalTid by bit field decomposition (globalPid is dropped).
CUDA_COMBINED_KERNEL
start end globalTid correlationId gpu_start gpu_end pid tid deviceId shortName mangledName demangledName
Like CUDA_COMBINED but kernel-only, with resolved kernel names instead of
runtime API name. pid and tid are extracted from globalTid
(globalPid is dropped).
NVTX
text start end globalTid endGlobalTid domainId domainName eventType pid tid
text is the resolved NVTX annotation string. domainName is "Default"
if no domain was specified.
NCCL
text start end globalTid endGlobalTid domainId eventType jsonText pid tid
NCCL rows are filtered from NVTX to the domain named "NCCL" and keep NVTX
timing/thread columns. jsonText is present when advanced NCCL tracing payloads exist.
NCCL_API
text start end globalTid endGlobalTid domainId eventType jsonText count dataType rank pid tid
Extends NCCL with parsed API payload fields (count, dataType,
rank).
NCCL_GPU_OPERATIONS
localId (index) text start end commHash commRank count dataType operationClass operationId
One row per merged GPU operation (grouped from per-channel events). Includes
both collective and P2P operations, but only common NCCL GPU operation payload
fields. operationClass identifies "collective" versus "p2p"
operations, and the 64-bit numeric operationId acts as unique id to correlated
collectives across all ranks as well as sends with their corresponding receives.
The id is unique across all collective and P2P operations.
localId is the DataFrame index after grouping.
NCCL_GPU_OPERATIONS_COLLECTIVE
localId (index) text start end commHash commRank count dataType seqNumber apiId
One row per collective GPU operation merged across channels, not across ranks. This table exposes collective-specific payload fields.
MPI / UCX
globalTid start end text pid tid
All sub-tables concatenated, sorted by start. text is the resolved
operation name (e.g., "MPI_Send", "MPI_Allreduce").
NIC
start end globalId nicId value metricsListId metricsIdx GUID nic_name metric_name
IB_SWITCH
start end GUID value metricsListId metricsIdx metric_name
GPU_METRICS
timestamp typeId gpuId SMs Active SM Issue Tensor Active Unallocated Warps in Active SMs ...
Each metric is a column. The available columns depend on the GPU and capture settings.
PERF_EVENTS
start end vmId eventId count componentType cpu name
GENERIC
timestamp typeId genericEventId metricName metricValue dataSrc gpuId
Examples in Existing Recipes#
These recipes demonstrate how composite tables are used in practice:
cuda_gpu_kern_sum: UsesCUDA_KERNELfor kernel summary statisticsnvtx_sum: UsesNVTXfor NVTX range analysisnvtx_gpu_proj_sum: UsesCUDA_GPU_GRAPH+NVTXfor GPU projectionnetwork_map_aws: UsesGENERICfor AWS EFA metrics
See also Tutorial: Create a User-Defined Recipe for a step-by-step tutorial on creating your own recipe.
Tutorial: Create a User-Defined Recipe#
The Nsight Systems recipe system is designed to be extensible and we hope that many users will use it to create their own recipes. This short tutorial will highlight the steps needed to create a recipe that is a customized version of one of the recipes that is included in the Nsight Systems recipe package.
Before starting, you may want to review Recipe Composite Tables for an overview of the pre-processed data tables available to recipes.
Step 1: Create the recipe directory and script
Make a new directory in the
<install-dir>/target-<os>-<arch>/python/packages/nsys_recipe/recipes folder based on
the name of your new recipe. The target directory is named after your platform, for
example target-linux-x64 or target-windows-x64. For this example, we will call
our new recipe
new_metric_util_map. We will copy the existing gpu_metric_util_map.py script
and create a new script called
new_metric_util_map.py in the new_metric_util_map directory. We will also
copy the heatmap.ipynb and metadata json files into the new_metric_util_map
directory. Type these steps in a Linux shell or in Windows PowerShell:
> cd <install-dir>/target-<os>-<arch>/python/packages/nsys_recipe/recipes
> mkdir new_metric_util_map
> cp gpu_metric_util_map/metadata.json new_metric_util_map/metadata.json
> cp gpu_metric_util_map/heatmap.ipynb new_metric_util_map/heatmap.ipynb
> cp gpu_metric_util_map/gpu_metric_util_map.py new_metric_util_map/new_metric_util_map.py
In the Windows Command Prompt, use copy instead of cp and backslashes in the paths.
Replace the module name in metadata.json with new_metric_util_map
and update the display name and description to your preference. Also, rename
the class name GpuMetricUtilMap in new_metric_util_map.py to
NewMetricUtilMap. We will discuss the detailed functionality of the new
recipe code in the subsequent steps.
Step 2: Modify the mapper function
Many recipes are structured as a map-reduce algorithm. The mapper function is called for every .nsys-rep file in the report directory. The mapper function performs a series of calculations on the events in each Nsight Systems report and produces an intermediate data set. The intermediate results are then combined by the reduce function to produce the final results. The mapper function can be called in parallel, either on multiple cores of a single node (using the concurrent python module), or multiple ranks of a multi-node recipe analysis (using the Dask distributed module).
When we create a new recipe, we need to create a class that derives from the Recipe base class. For our example, that class will be called NewMetricUtilMap (which we had renamed in step 1).
The mapper_func() method dispatches _mapper_func() for each report.
That function uses DataService to load the GPU_METRICS composite table
and apply the configured time filters and timestamp adjustment. It then groups
the samples by GPU and time bin and computes the mean of each selected metric.
Use reports collected with GPU metrics enabled that contain all four metrics used in this example. The mapper skips reports missing any selected metric.
The original script retrieved three GPU metrics: SM Active, SM Issue, and Tensor Active. In our new version of the script, we will extract a fourth metric, Unallocated Warps in Active SMs.
Find this line in
new_metric_util_map.py:metric_cols = ["SMs Active", "SM Issue", "Tensor Active"]
Add the Unallocated Warps in Active SMs metric:
metric_cols = [ "SMs Active", "SM Issue", "Tensor Active", "Unallocated Warps in Active SMs", ]
Step 3: Modify the reduce function
Our new mapper function returns each report’s filename and a Pandas DataFrame
containing the four metrics. The reducer discards missing results, assigns
ranks to the remaining reports, and combines their DataFrames using
pandas.concat(). Since the reducer function is generic in our
case, no modifications are needed. However, if you would like to add any
additional post-processing, you can do so in this function.
Step 4: Add a plot to the Jupyter notebook
Our new recipe class will create a Parquet output file with all the data
produced by the reducer function, using the to_parquet() function. It will also
create a Jupyter notebook file using the create_notebook() function.
In new_metric_util_map/heatmap.ipynb, extend the metrics list to include
the fourth metric and update the notebook’s metric descriptions. The summary
graph and heatmaps both use this list. Change:
metrics = [
"SMs Active",
"SM Issue",
"Tensor Active",
]
To this:
metrics = [
"SMs Active",
"SM Issue",
"Tensor Active",
"Unallocated Warps in Active SMs",
]
That completes all the modifications for our NewMetricUtilMap class.
Step 5: Run the new recipe
If the new recipe is located in the default recipe directory nsys_recipe/recipes,
we can directly run it using the nsys recipe command like this:
> nsys recipe new_metric_util_map --input <directory of reports>
It is also possible to have a recipe located outside of this directory. In this
case, you need to set the environment variable NSYS_RECIPE_PATH to the directory
containing the recipe when running the nsys recipe command.
When successful, the recipe should produce a new recipe result directory called
new_metric_util_map-1.
If we open the Jupyter notebook in that recipe and execute the code, we should see our new heatmap along with the three plots produced by the original version of the recipe. Here is an example:
Recipe discovery#
Discovery checks NSYS_RECIPE_PATH first, then the current directory,
and finally the installed nsys_recipe/recipes directory. Each recipe
folder must contain a metadata.json file whose module_name
identifies the Python module that defines the recipe class.
If the CLI reports Unknown recipe, check that the recipe folder is
in one of these locations and contains metadata.json.
Loading recipe data#
Use DataService.queue_table() with a table name for raw data or a
CompositeTable value for a composite table. read_queued_tables()
returns a dictionary of DataFrames keyed by the requested tables, or
None if reading fails. filter_and_adjust_time() applies the
configured filters and timestamp adjustments to the supplied DataFrame
in place. See Recipe Composite Tables for the available composite tables.
Adding recipe arguments#
Override the get_argument_parser() class method and call
super().get_argument_parser() to retain the common output options. Add
custom options with parser.add_recipe_argument(). It accepts an option
name and the usual argparse arguments, or an Option value from
nsys_recipe.lib.args for a shared recipe option. Return the parser and
check your options with nsys recipe <recipe-name> --help.
Define arguments in Python, not in metadata.json. The metadata describes
the recipe and identifies its module; it is not a command-line argument schema.
Writing recipe output#
add_output_file(filename, filetype=None) returns a path in the recipe
output directory. The recipe writes the data to that path. Providing a
filetype also records the file for inclusion in the analysis metadata.
create_notebook() copies a notebook template into the output directory and
can replace strings using replace_dict. add_notebook_helper_file()
copies supporting files needed by that notebook.
create_analysis_file() writes the recipe’s analysis metadata to an
.nsys-analysis file. Built-in recipes populate that metadata with their
output information before calling it.
Call add_insights(texts) to report short findings calculated from recipe
results. JupyterLab and CSV output write insights and follow-up guidance to
insights.jsonl. JSON Lines output writes these records to standard output.
Writing Post-Collection Events#
The nsys_writer Python module writes NVTX events to a .nsys-rep file
during post-collection analysis. It can augment Nsight Systems reports with
analysis results or data from external profiling sources.
nsys_writer is the Nsight Systems integration for the nvtx.writer
module. For the writer object model and API, see NVTX Python Writer.
Set up the writer#
Install nvtx version 0.2.16 or later in the Python environment that will
run nsys_writer. For the default nsys recipe environment, after the
environment has been created, run:
On Linux:
$HOME/.nsightsystems/venv/bin/python -m pip install "nvtx>=0.2.16"
On Windows (PowerShell):
& "$env:LOCALAPPDATA\NVIDIA Nsight Systems\venv\Scripts\python.exe" -m pip install "nvtx>=0.2.16"
For another recipe environment or a standalone script, use the Python executable for that environment.
When nsys_writer is used from an nsys recipe, Nsight Systems
automatically selects the writer backend.
For a standalone Python script, install nsys_writer package
in the Python environment. Use the target-<os>-<arch> directory that
matches your platform, for example target-linux-x64 or
target-windows-x64:
/path/to/python -m pip install "<nsys-install-folder>/target-<os>-<arch>/python/packages/nsys_writer"
Also set the NSYS_WRITER_BACKEND environment variable to the full path of
the Nsight Systems writer backend library.
On Linux:
export NSYS_WRITER_BACKEND=<nsys-install-folder>/host-linux-x64/libNvtxwBackend.so
On Windows (PowerShell):
$env:NSYS_WRITER_BACKEND = "<nsys-install-folder>\host-windows-x64\NvtxwBackend.dll"
The backend is loaded when the first Session is created.
Create a report#
The following example uses the default UTC time base to write a
three-millisecond range to analysis.nsys-rep. time.time_ns() provides
timestamps in nanoseconds since the Unix epoch:
import time
from nsys_writer import Session
start = time.time_ns()
with Session("analysis") as session:
domain = session.get_domain("My Analysis")
with session.create_stream("My Stream", domain=domain) as stream:
stream.write_pushpop(
start=start,
end=start + 3_000_000,
message="My Message",
color="green",
)
The session name determines the output path. Specify the desired path without
the .nsys-rep suffix.
Add events to an existing report#
Pass report_merge to copy an existing report and add events to the copy.
The session name determines the output path, which must differ from the
report_merge path. Using the same path can truncate the source report on
Linux. On Windows the merge fails instead, because the source report stays
open while the new report is written. With distinct paths, the original
report is not modified. This example creates profile-annotated.nsys-rep
and places the range on a custom scope for derived analysis results:
from nsys_writer import Session, TimeBase
with Session(
name="profile-annotated",
report_merge="profile.nsys-rep",
time_base=TimeBase.RELATIVE,
) as session:
domain = session.get_domain("Analysis Results")
scope = domain.get_scope("Derived Analysis")
with session.create_stream(
"Post-Processing", domain=domain, scope=scope
) as stream:
stream.write_pushpop(
start=2_000_000,
end=6_000_000,
message="derived range",
)
The report_merge path must identify an existing .nsys-rep file.
Choose a timestamp base#
Use a timestamp base that matches the source of the event timestamps:
TimeBase.RELATIVE: nanoseconds relative to the report analysis start.TimeBase.UTC(default): nanoseconds since the Unix epoch.TimeBase.CLOCK_MONOTONIC_RAW: LinuxCLOCK_MONOTONIC_RAWnanoseconds.TimeBase.CNTVCT: ARM virtual counter ticks.
Use TimeBase.CLOCK_MONOTONIC_RAW or TimeBase.CNTVCT when merging into a
report that contains matching clock-conversion data.
API reference#
nsys_writer.Session inherits the event, counter, schema, domain, scope, and
stream APIs from nvtx.writer.Session.
nsys_writer.Session.create_stream does not expose the stream ordering and
skid controls available in nvtx.writer.Session.create_stream because they
have no effect in this context.
For the complete API, including counter and batch-writing examples, see
the NVTX Python Writer API reference.
Available Export Formats#
You may want to create your own analysis system interrogating the data produced by Nsight Systems. However, the .nsys-rep output format of Nsight Systems is not designed to be accessed by users, and we cannot guarantee its stability. Therefore you will want to export into a good format for your further processing.
SQLite Schema Reference#
Nsight Systems has the ability to export SQLite database files from the .nsys-rep
results file. From the CLI, use nsys export. From the GUI, call
File->Export....
Note
The .nsys-rep report format is the only data format for Nsight Systems that should be considered forward-compatible. The SQLite schema can and will change in the future.
The schema for a concrete database can be obtained with the sqlite3 tool
built-in command .schema. The sqlite3 tool can be located in the Target or
Host directory of your Nsight Systems installation.
Note
Currently, tables are created lazily, and therefore not every table described in the documentation will be present in a particular database. This will change in a future version of the product. If you want a full schema of all possible tables, use nsys export --lazy=false during the export phase.
Currently, a table is created for each data type in the exported database. Since usage patterns for exported data may vary greatly and no default use cases have been established, no indexes or extra constraints are created. Instead, refer to the SQLite Examples section for a list of common recipes. This may change in a future version of the product.
To check the version of your exported SQLite file, check the value of
EXPORT_SCHEMA_VERSION in the META_DATA_EXPORT table. The schema version
is a common three-value major/minor/micro version number. The first value, or
major value, indicates the overall format of the database, and is only changed
if there is a major re-write or re-factor of the entire database format. It is
assumed that if the major version changes, all scripts or queries will break.
The middle, or minor, version is changed anytime there is a more localized, but
potentially breaking change, such as renaming an existing column, or changing
the type of an existing column. The last, or micro version is changed any time
there are additions, such as a new table or column, that should not introduce
any breaking change when used with well-written, best-practices queries.
The changes between schema versions are documented in
<install_dir>/host*/exporter/export_schema_version_notes.txt.
This is the schema as of the 2026.6 release, schema version 3.31.4.
Note
When dynamic NVTX payload export is enabled (nsys export --dynamic-tables=all),
each NVTX binary payload schema is additionally exported as its own data table
named DYNAMIC_NVTXPAYLOAD_DATATYPE_<globalPid>_<domainId>_<schemaId>, with
one column per scalar field (plus a ..._<arrayFieldName> child table per
array field). A schema with no scalar fields gets no data table of its own —
only its array child tables (and its nested schemas’ tables); the
NVTX_EVENTS_PAYLOAD_SCHEMAS and NVTX_PAYLOAD_SCHEMA_RELATIONS rows
still record the schema. Because these names depend on the schemas present in
the report, they are not listed below. Join them back to NVTX_EVENTS on eventId;
see the SQLite Examples section for a worked example. This affects SQLite,
Arrow, and Arrow/Parquet directory exports only.
CREATE TABLE StringIds (
-- Consolidation of repetitive string values.
id INTEGER NOT NULL PRIMARY KEY, -- ID reference value.
value TEXT NOT NULL -- String value.
);
CREATE TABLE ANALYSIS_FILE (
-- Analysis file content
id INTEGER NOT NULL PRIMARY KEY, -- ID reference value.
filename TEXT, -- File path
contentId INTEGER, -- REFERENCES StringIds(id) -- File content
globalPid INTEGER NOT NULL -- Serialized GlobalId.
);
CREATE TABLE ThreadNames (
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Thread name
priority INTEGER, -- Priority of the thread.
globalTid INTEGER -- Serialized GlobalId.
);
CREATE TABLE ProcessStreams (
globalPid INTEGER NOT NULL, -- Serialized GlobalId.
filenameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- File name
contentId INTEGER NOT NULL -- REFERENCES StringIds(id) -- Stream content
);
CREATE TABLE TARGET_INFO_SYSTEM_ENV (
globalVid INTEGER, -- Serialized GlobalId.
devStateName TEXT NOT NULL, -- Device state name.
name TEXT NOT NULL, -- Property name.
nameEnum INTEGER NOT NULL, -- Property enum value.
value TEXT NOT NULL -- Property value.
);
CREATE TABLE TARGET_INFO_NIC_INFO (
GUID INTEGER NOT NULL, -- Network interface GUID
stateName TEXT NOT NULL, -- Device state name
nicId INTEGER NOT NULL, -- Network interface Id
name TEXT NOT NULL, -- Network interface name
deviceId INTEGER NOT NULL, -- REFERENCES ENUM_NET_DEVICE_ID(id)
vendorId INTEGER NOT NULL, -- REFERENCES ENUM_NET_VENDOR_ID(id)
linkLayer INTEGER NOT NULL -- REFERENCES ENUM_NET_LINK_TYPE(id)
);
CREATE TABLE NIC_ID_MAP (
-- Map between NIC info nicId and NIC metric globalId
nicId INTEGER NOT NULL, -- REFERENCES TARGET_INFO_NIC_INFO(nicId)
globalId INTEGER NOT NULL -- REFERENCES NET_NIC_METRIC(globalId)
);
CREATE TABLE TARGET_INFO_SESSION_START_TIME (
utcEpochNs INTEGER, -- UTC Epoch timestamp at start of the capture (ns).
utcTime TEXT, -- Start of the capture in UTC.
localTime TEXT, -- Start of the capture in local time of target.
systemClockNs INTEGER, -- Target system clock timestamp at start of the capture (ns).
sessionStartCntVctNs INTEGER -- CntVctNs value at SessionNs=0; chain offset for CntVctNs->SessionNs (ns).
);
CREATE TABLE TIME_DOMAIN (
-- Time domain information. This is an experimental table and its format may change. Please avoid relying on the content for now.
timeDomainUuid TEXT NOT NULL, -- Time domain UUID.
clockName TEXT NOT NULL -- Name of the clock of this time domain.
);
CREATE TABLE TIME_SYNCHRONIZATION (
-- Time synchronization data between time domains. This is an experimental table and its format may change. Please avoid relying on the content for now.
sourceTimeDomainUuid TEXT NOT NULL, -- Source time domain UUID.
targetTimeDomainUuid TEXT NOT NULL, -- Target time domain UUID.
rateRatio REAL, -- Target clock rate divided by source clock rate of the segment ending at this synchronization point. Empty for the start of the synchronized range.
sourceTimestamp INTEGER NOT NULL, -- Source timestamp at the synchronization point.
targetTimestamp INTEGER NOT NULL -- Target timestamp at the synchronization point.
);
CREATE TABLE ANALYSIS_DETAILS (
-- Details about the analysis session.
globalVid INTEGER NOT NULL, -- Serialized GlobalId.
duration INTEGER NOT NULL, -- The total time span of the entire trace (ns).
startTime INTEGER NOT NULL, -- Trace start timestamp in nanoseconds.
stopTime INTEGER NOT NULL -- Trace stop timestamp in nanoseconds.
);
CREATE TABLE PMU_EVENT_REQUESTS (
-- PMU event requests
id INTEGER NOT NULL, -- PMU event request.
eventid INTEGER, -- PMU counter event id.
source INTEGER NOT NULL, -- REFERENCES ENUM_PMU_EVENT_SOURCE(id)
unit_type INTEGER NOT NULL, -- REFERENCES ENUM_PMU_UNIT_TYPE(id)
event_name TEXT, -- PMU counter unique name
PRIMARY KEY (id)
);
CREATE TABLE TARGET_INFO_GPU (
vmId INTEGER NOT NULL, -- Serialized GlobalId.
id INTEGER NOT NULL, -- Device ID.
name TEXT, -- Device name.
busLocation TEXT, -- PCI bus location.
isDiscrete INTEGER, -- True if discrete, false if integrated.
l2CacheSize INTEGER, -- Size of L2 cache (B).
totalMemory INTEGER, -- Total amount of memory on the device (B).
memoryBandwidth INTEGER, -- Amount of memory transferred (B).
clockRate INTEGER, -- Clock frequency (Hz).
smCount INTEGER, -- Number of multiprocessors on the device.
pwGpuId INTEGER, -- PerfWorks GPU ID.
uuid TEXT, -- Device UUID.
luid INTEGER, -- Device LUID.
chipName TEXT, -- Chip name.
cuDevice INTEGER, -- CUDA device ID.
ctxswDevPath TEXT, -- GPU context switch device node path.
ctrlDevPath TEXT, -- GPU control device node path.
revision INTEGER, -- Revision number.
nodeMask INTEGER, -- Device node mask.
constantMemory INTEGER, -- Memory available on device for __constant__ variables (B).
maxIPC INTEGER, -- Maximum instructions per count.
maxRegistersPerBlock INTEGER, -- Maximum number of 32-bit registers available per block.
maxShmemPerBlock INTEGER, -- Maximum optin shared memory per block.
maxShmemPerBlockOptin INTEGER, -- Maximum optin shared memory per block.
maxShmemPerSm INTEGER, -- Maximum shared memory available per multiprocessor (B).
maxRegistersPerSm INTEGER, -- Maximum number of 32-bit registers available per multiprocessor.
threadsPerWarp INTEGER, -- Warp size in threads.
asyncEngines INTEGER, -- Number of asynchronous engines.
maxWarpsPerSm INTEGER, -- Maximum number of warps per multiprocessor.
maxBlocksPerSm INTEGER, -- Maximum number of blocks per multiprocessor.
maxThreadsPerBlock INTEGER, -- Maximum number of threads per block.
maxBlockDimX INTEGER, -- Maximum X-dimension of a block.
maxBlockDimY INTEGER, -- Maximum Y-dimension of a block.
maxBlockDimZ INTEGER, -- Maximum Z-dimension of a block.
maxGridDimX INTEGER, -- Maximum X-dimension of a grid.
maxGridDimY INTEGER, -- Maximum Y-dimension of a grid.
maxGridDimZ INTEGER, -- Maximum Z-dimension of a grid.
computeMajor INTEGER, -- Major compute capability version number.
computeMinor INTEGER, -- Minor compute capability version number.
smMajor INTEGER, -- Major multiprocessor version number.
smMinor INTEGER, -- Minor multiprocessor version number.
numaNodeId INTEGER -- NUMA node ID.
);
CREATE TABLE TARGET_INFO_XMC_SPEC (
vmId INTEGER NOT NULL, -- Serialized GlobalId.
clientId INTEGER NOT NULL, -- Client ID.
type TEXT NOT NULL, -- Client type.
name TEXT NOT NULL, -- Client name.
groupId TEXT NOT NULL -- Client group ID.
);
CREATE TABLE TARGET_INFO_CUDA_DEVICE (
gpuId INTEGER, -- GPU ID.
cudaId INTEGER NOT NULL, -- CUDA device ID.
pid INTEGER NOT NULL, -- Process ID.
uuid TEXT, -- Device UUID.
numMultiprocessors INTEGER -- Number of SMs available on the device.
);
CREATE TABLE TARGET_INFO_PROCESS (
processId INTEGER NOT NULL, -- Process ID.
openGlVersion TEXT NOT NULL, -- OpenGL version.
correlationId INTEGER NOT NULL, -- Correlation ID of the kernel.
nameId INTEGER NOT NULL -- REFERENCES StringIds(id) -- Function name
);
CREATE TABLE TARGET_INFO_NVTX_CUDA_DEVICE (
name TEXT NOT NULL, -- CUDA device name assigned using NVTX.
hwId INTEGER NOT NULL, -- Hardware ID.
vmId INTEGER NOT NULL, -- VM ID.
deviceId INTEGER NOT NULL -- Device ID.
);
CREATE TABLE TARGET_INFO_NVTX_CUDA_CONTEXT (
name TEXT NOT NULL, -- CUDA context name assigned using NVTX.
hwId INTEGER NOT NULL, -- Hardware ID.
vmId INTEGER NOT NULL, -- VM ID.
processId INTEGER NOT NULL, -- Process ID.
deviceId INTEGER NOT NULL, -- Device ID.
contextId INTEGER NOT NULL -- Context ID.
);
CREATE TABLE TARGET_INFO_NVTX_CUDA_STREAM (
name TEXT NOT NULL, -- CUDA stream name assigned using NVTX.
hwId INTEGER NOT NULL, -- Hardware ID.
vmId INTEGER NOT NULL, -- VM ID.
processId INTEGER NOT NULL, -- Process ID.
deviceId INTEGER NOT NULL, -- Device ID.
contextId INTEGER NOT NULL, -- Context ID.
streamId INTEGER NOT NULL -- Stream ID.
);
CREATE TABLE TARGET_INFO_CUDA_CONTEXT_INFO (
nullStreamId INTEGER NOT NULL, -- Stream ID.
hwId INTEGER NOT NULL, -- Hardware ID.
vmId INTEGER NOT NULL, -- VM ID.
processId INTEGER NOT NULL, -- Process ID.
deviceId INTEGER NOT NULL, -- Device ID.
contextId INTEGER NOT NULL, -- Context ID.
parentContextId INTEGER, -- For green context, this is the parent context id.
isGreenContext INTEGER, -- Is this a Green Context?
numMultiprocessors INTEGER, -- For green context, number of SMs allocated.
numTpcs INTEGER, -- For green context, number of TPCs allocated.
tpcMask TEXT, -- For green context, comma-separated hex TPC mask values (e.g., '0xf,0x0').
workqueueResourceId INTEGER, -- For green context, workqueue resource ID.
workqueueConcurrencyLimit INTEGER, -- For green context, workqueue concurrency limit.
workqueueSharingScope INTEGER, -- For green context, workqueue sharing scope.
workqueueChannelCount INTEGER, -- For green context, workqueue channel count captured at green context creation; may not reflect later reassignment, for example after other green contexts are created or destroyed.
workqueueChannelIds TEXT -- For green context, comma-separated workqueue channel IDs captured at green context creation; may not reflect later reassignment, for example after other green contexts are created or destroyed.
);
CREATE TABLE TARGET_INFO_CUDA_STREAM (
streamId INTEGER NOT NULL, -- Stream ID.
hwId INTEGER NOT NULL, -- Hardware ID.
vmId INTEGER NOT NULL, -- VM ID.
processId INTEGER NOT NULL, -- Process ID.
contextId INTEGER NOT NULL, -- Context ID.
priority INTEGER NOT NULL, -- Priority of the stream.
flag INTEGER NOT NULL -- REFERENCES ENUM_CUPTI_STREAM_TYPE(id)
);
CREATE TABLE TARGET_INFO_WDDM_CONTEXTS (
context INTEGER NOT NULL,
engineType INTEGER NOT NULL,
nodeOrdinal INTEGER NOT NULL,
friendlyName TEXT NOT NULL
);
CREATE TABLE TARGET_INFO_PERF_METRIC (
id INTEGER NOT NULL, -- Event or Metric ID value
name TEXT NOT NULL, -- Event or Metric name
description TEXT NOT NULL, -- Event or Metric description
unit TEXT NOT NULL, -- Event or Metric measurement unit
displayName TEXT -- GUI friendly name of the Event or Metric
);
CREATE TABLE TARGET_INFO_NETWORK_METRICS (
metricsListId INTEGER NOT NULL, -- Metric list ID
metricsIdx INTEGER NOT NULL, -- List index of metric
name TEXT NOT NULL, -- Name of metric
description TEXT NOT NULL, -- Description of metric
unit TEXT NOT NULL -- Measurement unit of metric
);
CREATE TABLE TARGET_INFO_COMPONENT (
componentId INTEGER NOT NULL, -- Component ID
name TEXT NOT NULL, -- Component name
instance INTEGER, -- Component instance
parentId INTEGER -- Parent Component ID
);
CREATE TABLE NET_DEVICE_INFO (
networkId INTEGER NOT NULL, -- The Device's Network ID
guid INTEGER, -- Device Guid
name TEXT, -- Device Name
des TEXT, -- Device description
lid INTEGER -- Device Lid
);
CREATE TABLE NET_DEVICE_PORT_INFO (
guid INTEGER, -- REFERENCES NET_DEVICE_INFO(guid) -- Device Global Identifier
portNumber INTEGER NOT NULL, -- Internal Port Number
portLabel TEXT NOT NULL, -- Port Label
portLid INTEGER NOT NULL -- Port Lid
);
CREATE TABLE NET_DEVICE_TYPE_MAP (
guid INTEGER, -- REFERENCES NET_DEVICE_INFO(guid) -- Device Global Identifier
deviceType INTEGER NOT NULL -- REFERENCES ENUM_NET_DEVICE_TYPE(id)
);
CREATE TABLE META_DATA_CAPTURE (
-- information about nsys capture parameters
name TEXT NOT NULL, -- Name of meta-data record
value TEXT -- Value of meta-data record
);
CREATE TABLE META_DATA_EXPORT (
-- information about nsys export process
name TEXT NOT NULL, -- Name of meta-data record
value TEXT -- Value of meta-data record
);
CREATE TABLE ENUM_NSYS_EVENT_TYPE (
-- Nsys event type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_NSYS_EVENT_CLASS (
-- Nsys event class labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_NSYS_GENERIC_EVENT_SOURCE (
-- Nsys generic event source labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_NSYS_GENERIC_EVENT_GROUP (
-- Nsys generic event group labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_NSYS_GENERIC_EVENT_FIELD_TYPE (
-- Nsys generic event field type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_NSYS_GENERIC_EVENT_FIELD_ETW_PROPERTY (
-- Nsys generic event field ETW property flag labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_NSYS_GENERIC_EVENT_FIELD_ETW_TYPE (
-- Nsys generic event field ETW type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_NSYS_GENERIC_EVENT_FIELD_ETW_FLAGS (
-- Nsys generic event field ETW map info flag labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_GPU_CTX_SWITCH (
-- GPU context switch labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_CUDA_MEMCPY_OPER (
-- CUDA memcpy operation labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_CUDA_MEM_KIND (
-- CUDA memory kind labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_CUDA_MEMPOOL_TYPE (
-- CUDA mempool type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_CUDA_MEMPOOL_OPER (
-- CUDA mempool operation labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_CUDA_DEV_MEM_EVENT_OPER (
-- CUDA device mem event operation labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_CUDA_KERNEL_LAUNCH_TYPE (
-- CUDA kernel launch type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_CUDA_SHARED_MEM_LIMIT_CONFIG (
-- CUDA shared memory limit config labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_CUDA_UNIF_MEM_MIGRATION (
-- CUDA unified memory migration cause labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_CUDA_UNIF_MEM_ACCESS_TYPE (
-- CUDA unified memory access type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_CUDA_FUNC_CACHE_CONFIG (
-- CUDA function cache config labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_CUPTI_STREAM_TYPE (
-- CUPTI stream type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_CUPTI_SYNC_TYPE (
-- CUPTI synchronization type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_CUPTI_OVERHEAD_TYPE (
-- CUPTI overhead type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_CUPTI_JIT_ENTRY_TYPE (
-- CUDA JIT entry type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_CUPTI_JIT_OPERATION_TYPE (
-- CUDA JIT operation type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_STACK_UNWIND_METHOD (
-- Stack unwind method labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_SAMPLING_THREAD_STATE (
-- Sampling thread state labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_SCHEDULING_THREAD_BLOCK (
-- Scheduling thread block labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_OPENGL_DEBUG_SOURCE (
-- OpenGL debug source labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_OPENGL_DEBUG_TYPE (
-- OpenGL debug type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_OPENGL_DEBUG_SEVERITY (
-- OpenGL debug severity labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_VULKAN_PIPELINE_CREATION_FLAGS (
-- Vulkan pipeline creation feedback flag labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_VULKAN_HEAP_TYPE (
-- Vulkan heap type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_VULKAN_HEAP_FLAGS (
-- Vulkan heap flag labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_VULKAN_MEMORY_PROPERTY_FLAGS (
-- Vulkan memory property flag labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_D3D12_HEAP_TYPE (
-- D3D12 heap type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_D3D12_PAGE_PROPERTY (
-- D3D12 CPU page property labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_D3D12_HEAP_FLAGS (
-- D3D12 heap flag labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_D3D12_CMD_LIST_TYPE (
-- D3D12 command list type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_OPENACC_DEVICE (
-- OpenACC device type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_OPENACC_EVENT_KIND (
-- OpenACC event type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_OPENMP_EVENT_KIND (
-- OpenMP event kind labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_OPENMP_THREAD (
-- OpenMP thread labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_OPENMP_DISPATCH (
-- OpenMP dispatch labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_OPENMP_SYNC_REGION (
-- OpenMP sync region labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_OPENMP_WORK (
-- OpenMP work labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_OPENMP_MUTEX (
-- OpenMP mutex labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_OPENMP_TASK_FLAG (
-- OpenMP task flags labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_OPENMP_TASK_STATUS (
-- OpenMP task status labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_NVDRIVER_EVENT_ID (
-- NV-Driver event it labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_WDDM_PAGING_QUEUE_TYPE (
-- WDDM paging queue type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_WDDM_PACKET_TYPE (
-- WDDM packet type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_WDDM_ENGINE_TYPE (
-- WDDM engine type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_WDDM_INTERRUPT_TYPE (
-- WDDM DMA interrupt type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_WDDM_VIDMM_OP_TYPE (
-- WDDM VidMm operation type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_NET_LINK_TYPE (
-- NIC link layer labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_NET_DEVICE_ID (
-- NIC PCIe device id labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_NET_VENDOR_ID (
-- NIC PCIe vendor id labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_ETW_MEMORY_TRANSFER_TYPE (
-- memory transfer type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_PMU_EVENT_SOURCE (
-- PMU event source labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_PMU_UNIT_TYPE (
-- PMU unit type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_VIDEO_ENGINE_TYPE (
-- Video engine type id labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_VIDEO_ENGINE_CODEC (
-- Video engine codec labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_DIAGNOSTIC_SEVERITY_LEVEL (
-- Diagnostic message severity level labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_DIAGNOSTIC_SOURCE_TYPE (
-- Diagnostic message source type labels
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_DIAGNOSTIC_TIMESTAMP_SOURCE (
-- Diagnostic message timestamp source lables
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_NET_DEVICE_TYPE (
-- network device types
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_NET_IB_CONGESTION_EVENT_TYPE (
-- IB Switch congestion event types
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE ENUM_OSRT_FILE_ACCESS_EVENT_TYPE (
-- OSRT File Access event type
id INTEGER NOT NULL PRIMARY KEY, -- Enum numerical value.
name TEXT, -- Enum symbol name.
label TEXT -- Enum human name.
);
CREATE TABLE GENERIC_EVENT_SOURCES (
-- Generic event source modules
sourceId INTEGER NOT NULL PRIMARY KEY, -- Serialized GlobalId.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Event source name
timeSourceId INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_GENERIC_EVENT_SOURCE(id)
sourceGroupId INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_GENERIC_EVENT_GROUP(id)
hyperType TEXT, -- Hypervisor Type
hyperVersion TEXT, -- Hypervisor Version
hyperStructPrefix TEXT, -- Hypervisor Struct Prefix
hyperMacroPrefix TEXT, -- Hypervisor Macro Prefix
hyperFilterFlags INTEGER, -- Hypervisor Custom Filter Flags
hyperDomain TEXT, -- Hypervisor Domain
data TEXT -- JSON encoded generic event source description.
);
CREATE TABLE GENERIC_EVENT_TYPES (
-- Generic event type/schema descriptions.
typeId INTEGER NOT NULL PRIMARY KEY, -- Serialized GlobalId.
sourceId INTEGER NOT NULL, -- REFERENCES GENERIC_EVENT_SOURCES(sourceId)
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Event type name
hyperComment TEXT, -- Event Type Hypervisor Comment
ftraceFormat TEXT, -- Event Type FTrace Format
etwProviderId INTEGER, -- Event Type ETW Provider Id
etwProviderNameId INTEGER, -- Event Type ETW Provider Name Id
etwTaskId INTEGER, -- Event Type ETW Task Id
etwTaskNameId INTEGER, -- Event Type ETW Task Name Id
etwEventId INTEGER, -- Event Type ETW Event Id
etwVersion INTEGER, -- Event Type ETW Version
etwGuidHigh INTEGER, -- Event Type ETW GUID high
etwGuidLow INTEGER, -- Event Type ETW GUID low
etwGuid TEXT, -- ETW Provider GUID.
data TEXT -- JSON encoded generic event type description.
);
CREATE TABLE GENERIC_EVENT_TYPE_FIELDS (
-- Generic event type/schema individual data field descriptions.
typeId INTEGER NOT NULL, -- Serialized GlobalId.
fieldIdx INTEGER NOT NULL, -- Index of type field
fieldNameId INTEGER NOT NULL, -- Name of field.
offset INTEGER NOT NULL, -- Field alignment offset size, in bytes.
size INTEGER NOT NULL, -- Field size, in bytes.
type INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_GENERIC_EVENT_FIELD_TYPE(id)
hyperTypeName TEXT, -- Event Field Hypervisor Type Name
hyperFormat TEXT, -- Event Field Hypervisor Format
hyperComment TEXT, -- Event Field Hypervisor Comment
ftracePrefix TEXT, -- Event Field FTrace Prefix
ftraceSuffix TEXT, -- Event Field FTrace Suffix
etwFlags INTEGER, -- REFERENCES ENUM_NSYS_GENERIC_EVENT_FIELD_ETW_PROPERTY(id)
etwCountFieldIndex INTEGER, -- Event Field ETW Count Field Index
etwLengthFieldIndex INTEGER, -- Event Field ETW Length Field Index
etwType INTEGER, -- REFERENCES ENUM_NSYS_GENERIC_EVENT_FIELD_ETW_TYPE(id)
etwMapInfoFlags INTEGER, -- REFERENCES ENUM_NSYS_GENERIC_EVENT_FIELD_ETW_FLAGS(id)
etwOrderedFieldIndex INTEGER -- Event Field ETW Ordered Field Index
);
CREATE TABLE GENERIC_EVENT_TYPE_FIELD_MAP (
-- Generic event ENUM data. Mostly used by ETW.
typeId INTEGER NOT NULL, -- Serialized GlobalId.
fieldIdx INTEGER NOT NULL, -- Index of type field
enum INTEGER NOT NULL, -- Event Field ETW Map Info enum.
name TEXT NOT NULL, -- Event Field ETW Map Info Name.
nameId INTEGER NOT NULL -- Event Field ETW Map Info Name Id.
);
CREATE TABLE GENERIC_EVENTS (
-- Dynamic or unstructured event data.
genericEventId INTEGER NOT NULL PRIMARY KEY, -- Id of particular generic event
rawTimestamp INTEGER NOT NULL, -- Raw event timestamp recorded during profiling.
timestamp INTEGER NOT NULL, -- Event timestamp converted to the profiling session timeline.
typeId INTEGER NOT NULL, -- REFERENCES GENERIC_EVENT_TYPES(typeId)
globalTid INTEGER, -- Serialized GlobalId.
data TEXT -- JSON encoded event data.
);
CREATE TABLE GENERIC_EVENT_DATA (
-- GENERIC_EVENTS data values.
genericEventId INTEGER NOT NULL, -- REFERENCES GENERIC_EVENTS(genericEventId)
fieldIdx INTEGER NOT NULL, -- Index of type field
intVal INTEGER, -- Integer value, signed
uintVal INTEGER, -- Integer value, unsigned
floatVal REAL, -- Floating point value, 32-bit
doubleVal REAL -- Floating point value, 64-bit
);
CREATE TABLE ETW_PROVIDERS (
-- Names and identifiers of ETW providers captured in the report.
providerId INTEGER NOT NULL PRIMARY KEY, -- Provider ID.
providerNameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Provider name
guid TEXT NOT NULL -- ETW Provider GUID.
);
CREATE TABLE ETW_TASKS (
-- Names and identifiers of ETW tasks captured in the report.
taskNameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Task name
taskId INTEGER NOT NULL, -- The event task ID.
providerId INTEGER NOT NULL -- Provider ID.
);
CREATE TABLE ETW_EVENTS (
-- Raw ETW events captured in the report.
rawTimestamp INTEGER NOT NULL, -- Raw event timestamp recorded during profiling.
timestamp INTEGER NOT NULL, -- Event start timestamp (ns).
typeId INTEGER NOT NULL, -- REFERENCES GENERIC_EVENT_TYPES(typeId)
globalTid INTEGER, -- Serialized GlobalId.
opcode INTEGER, -- The event opcode.
data TEXT NOT NULL -- JSON encoded event data.
);
CREATE TABLE TARGET_INFO_GPU_METRICS (
-- GPU Metrics, metric names and ids.
typeId INTEGER NOT NULL, -- REFERENCES GENERIC_EVENT_TYPES(typeId)
sourceId INTEGER NOT NULL, -- REFERENCES GENERIC_EVENT_SOURCES(sourceId)
typeName TEXT NOT NULL, -- Name of event type.
metricId INTEGER NOT NULL, -- Id of metric in event; not assumed to be stable.
metricName TEXT NOT NULL -- Definitive name of metric.
);
CREATE TABLE GPU_METRICS (
-- GPU Metrics, events and values.
rawTimestamp INTEGER NOT NULL, -- Raw event timestamp recorded during profiling.
timestamp INTEGER NOT NULL, -- Event timestamp (ns).
typeId INTEGER NOT NULL, -- REFERENCES TARGET_INFO_GPU_METRICS(typeId) and GENERIC_EVENT_TYPES(typeId)
metricId INTEGER NOT NULL, -- REFERENCES TARGET_INFO_GPU_METRICS(metricId)
value REAL -- Counter data value
);
CREATE TABLE TARGET_INFO_SOC_METRICS (
-- SoC Metrics, metric names and ids.
typeId INTEGER NOT NULL, -- REFERENCES GENERIC_EVENT_TYPES(typeId)
sourceId INTEGER NOT NULL, -- REFERENCES GENERIC_EVENT_SOURCES(sourceId)
typeName TEXT NOT NULL, -- Name of event type.
metricId INTEGER NOT NULL, -- Id of metric in event; not assumed to be stable.
metricName TEXT NOT NULL -- Definitive name of metric.
);
CREATE TABLE SOC_METRICS (
-- SoC Metrics, events and values.
rawTimestamp INTEGER NOT NULL, -- Raw event timestamp recorded during profiling.
timestamp INTEGER NOT NULL, -- Event timestamp (ns).
typeId INTEGER NOT NULL, -- REFERENCES TARGET_INFO_SOC_METRICS(typeId) and GENERIC_EVENT_TYPES(typeId)
metricId INTEGER NOT NULL, -- REFERENCES TARGET_INFO_GPU_METRICS(metricId)
value REAL -- Counter data value
);
CREATE TABLE MPI_COMMUNICATORS (
-- Identification of MPI communication groups.
rank INTEGER, -- Active MPI rank
timestamp INTEGER, -- Time of MPI communicator creation.
commHandle INTEGER, -- MPI communicator handle.
parentHandle INTEGER, -- MPI communicator handle.
localRank INTEGER, -- Local MPI rank in a communicator.
size INTEGER, -- MPI communicator size.
groupRoot INTEGER, -- Root rank (global) in MPI communicator.
groupRootUid INTEGER, -- Group root's communicator ID.
members TEXT, -- MPI communicator members (index is global rank).
commUid INTEGER -- Globally unique MPI communicator ID.
);
CREATE TABLE NVTX_PAYLOAD_SCHEMAS (
-- NVTX payload schema attributes.
globalPid INTEGER NOT NULL, -- Serialized GlobalId.
domainId INTEGER, -- User-controlled ID that can be used to group events.
schemaId INTEGER, -- Identifier of the payload schema.
name TEXT, -- Schema name.
type INTEGER, -- Schema type.
flags INTEGER, -- Schema flags.
numEntries INTEGER, -- Number of payload schema entries.
payloadSize INTEGER, -- Size of the static payload.
alignTo INTEGER -- Field alignment in bytes.
);
CREATE TABLE NVTX_PAYLOAD_SCHEMA_ENTRIES (
-- NVTX payload schema entries.
globalPid INTEGER NOT NULL, -- Serialized GlobalId.
domainId INTEGER NOT NULL, -- User-controlled ID that can be used to group events.
schemaId INTEGER NOT NULL, -- Identifier of the payload schema.
idx INTEGER NOT NULL, -- Index of the entry in the payload schema.
flags INTEGER, -- Payload entry flags.
type INTEGER, -- Payload entry type.
name TEXT, -- Label of the payload entry.
description TEXT, -- Description of the payload entry.
arrayOrUnionDetail INTEGER, -- Array length (index) or selected union member.
offset INTEGER -- Entry offset in the binary data in bytes.
);
CREATE TABLE NVTX_PAYLOAD_ENUMS (
-- NVTX payload enum attributes.
globalPid INTEGER NOT NULL, -- Serialized GlobalId.
domainId INTEGER, -- User-controlled ID that can be used to group events.
schemaId INTEGER, -- Identifier of the payload schema.
name TEXT, -- Schema name.
numEntries INTEGER, -- Number of entries in the enum.
size INTEGER -- Size of enumeration type in bytes.
);
CREATE TABLE NVTX_PAYLOAD_ENUM_ENTRIES (
-- NVTX payload enum entries.
globalPid INTEGER NOT NULL, -- Serialized GlobalId.
domainId INTEGER NOT NULL, -- User-controlled ID that can be used to group events.
schemaId INTEGER NOT NULL, -- Identifier of the payload schema.
idx INTEGER NOT NULL, -- Index of the entry in the payload schema.
name TEXT, -- Name of the enum value.
value INTEGER, -- Value of the enum entry.
isFlag INTEGER -- Indicates that the entry sets a specific set of bits, which can be used to define bitsets.
);
CREATE TABLE NVTX_SCOPES (
-- NVTX scopes.
globalPid INTEGER NOT NULL, -- Serialized GlobalId.
domainId INTEGER, -- User-controlled ID that can be used to group events.
scopeId INTEGER, -- Scope ID.
parentScopeId INTEGER, -- Parent scope ID.
path TEXT -- Scope path.
);
CREATE TABLE NVTX_SEMANTICS_COUNTER (
-- Counter semantics on NVTX payload schema entries and counter groups.
globalPid INTEGER NOT NULL, -- Serialized GlobalId.
domainId INTEGER NOT NULL, -- User-controlled ID that can be used to group events.
schemaId INTEGER NOT NULL, -- Identifier of the payload or counter schema.
entryIndex INTEGER, -- Payload schema entry index, or NULL for counter-group semantics.
counterGroupId INTEGER, -- Counter group identifier, or NULL for payload-schema-entry semantics.
unit TEXT, -- Counter unit.
unitScaleNumerator INTEGER NOT NULL, -- Counter unit scale numerator.
unitScaleDenominator INTEGER NOT NULL, -- Counter unit scale denominator.
flags INTEGER NOT NULL, -- Counter semantic flags.
limitMinInt64 INTEGER, -- Minimum signed-integer counter limit.
limitMinUint64 INTEGER, -- Minimum unsigned-integer counter limit.
limitMinDouble REAL, -- Minimum floating-point counter limit.
limitMaxInt64 INTEGER, -- Maximum signed-integer counter limit.
limitMaxUint64 INTEGER, -- Maximum unsigned-integer counter limit.
limitMaxDouble REAL -- Maximum floating-point counter limit.
);
CREATE TABLE NVTX_SEMANTICS_CORRELATION (
-- Correlation semantics on NVTX payload schema entries.
globalPid INTEGER NOT NULL, -- Serialized GlobalId.
domainId INTEGER NOT NULL, -- User-controlled ID that can be used to group events.
schemaId INTEGER NOT NULL, -- Identifier of the payload schema.
entryIndex INTEGER NOT NULL, -- Payload schema entry index.
correlationUuid TEXT NOT NULL, -- Correlation-domain UUID as 32 lowercase hexadecimal digits.
displayName TEXT, -- Optional correlation-domain display name.
role INTEGER NOT NULL -- Role of this event in a correlation with other events.
);
CREATE TABLE CUPTI_ACTIVITY_KIND_MEMCPY (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
deviceId INTEGER NOT NULL, -- Device ID.
contextId INTEGER NOT NULL, -- Context ID.
greenContextId INTEGER, -- Green context ID.
streamId INTEGER NOT NULL, -- Stream ID.
correlationId INTEGER, -- REFERENCES CUPTI_ACTIVITY_KIND_RUNTIME(correlationId)
globalPid INTEGER, -- Serialized GlobalId.
bytes INTEGER NOT NULL, -- Number of bytes transferred (B).
copyKind INTEGER NOT NULL, -- REFERENCES ENUM_CUDA_MEMCPY_OPER(id)
deprecatedSrcId INTEGER, -- Deprecated, use srcDeviceId instead.
srcKind INTEGER, -- REFERENCES ENUM_CUDA_MEM_KIND(id)
dstKind INTEGER, -- REFERENCES ENUM_CUDA_MEM_KIND(id)
srcDeviceId INTEGER, -- Source device ID.
srcContextId INTEGER, -- Source context ID.
dstDeviceId INTEGER, -- Destination device ID.
dstContextId INTEGER, -- Destination context ID.
migrationCause INTEGER, -- REFERENCES ENUM_CUDA_UNIF_MEM_MIGRATION(id)
graphNodeId INTEGER, -- REFERENCES CUDA_GRAPH_NODE_EVENTS(graphNodeId)
virtualAddress INTEGER, -- Virtual base address of the page/s being transferred.
copyCount INTEGER, -- The total number of memcopy operations traced in this record. In CUDA MemcpyBatchAsync APIs, multiple memcpy operations may be batched together for optimization purposes based on certain heuristics.
graphId INTEGER, -- Executable graph ID.
sourceGraphNodeId INTEGER, -- Node used for graph instantiation or the last node update.
sourceGraphId INTEGER -- Graph containing the node used for instantiation or the last node update.
);
CREATE TABLE CUPTI_ACTIVITY_KIND_MEMSET (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
deviceId INTEGER NOT NULL, -- Device ID.
contextId INTEGER NOT NULL, -- Context ID.
greenContextId INTEGER, -- Green context ID.
streamId INTEGER NOT NULL, -- Stream ID.
correlationId INTEGER, -- REFERENCES CUPTI_ACTIVITY_KIND_RUNTIME(correlationId)
globalPid INTEGER, -- Serialized GlobalId.
value INTEGER NOT NULL, -- Value assigned to memory.
bytes INTEGER NOT NULL, -- Number of bytes set (B).
graphNodeId INTEGER, -- REFERENCES CUDA_GRAPH_NODE_EVENTS(graphNodeId)
memKind INTEGER, -- REFERENCES ENUM_CUDA_MEM_KIND(id)
graphId INTEGER, -- Executable graph ID.
sourceGraphNodeId INTEGER, -- Node used for graph instantiation or the last node update.
sourceGraphId INTEGER -- Graph containing the node used for instantiation or the last node update.
);
CREATE TABLE CUPTI_ACTIVITY_KIND_MEM_DECOMPRESS (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
deviceId INTEGER NOT NULL, -- Device ID.
contextId INTEGER NOT NULL, -- Context ID.
streamId INTEGER NOT NULL, -- Stream ID.
correlationId INTEGER, -- REFERENCES CUPTI_ACTIVITY_KIND_RUNTIME(correlationId)
globalPid INTEGER, -- Serialized GlobalId.
ChannelID INTEGER NOT NULL, -- Channel Id of MemDecompress Operation.
sourceBytes INTEGER NOT NULL, -- Number of source bytes (B).
NumberOfOperations INTEGER NOT NULL -- Number of operations.
);
CREATE TABLE CUPTI_ACTIVITY_KIND_KERNEL (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
deviceId INTEGER NOT NULL, -- Device ID.
contextId INTEGER NOT NULL, -- Context ID.
greenContextId INTEGER, -- Green context ID.
streamId INTEGER NOT NULL, -- Stream ID.
correlationId INTEGER, -- REFERENCES CUPTI_ACTIVITY_KIND_RUNTIME(correlationId)
globalPid INTEGER, -- Serialized GlobalId.
demangledName INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Kernel function name w/ templates
shortName INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Base kernel function name
mangledName INTEGER, -- REFERENCES StringIds(id) -- Raw C++ mangled kernel function name
launchType INTEGER, -- REFERENCES ENUM_CUDA_KERNEL_LAUNCH_TYPE(id)
cacheConfig INTEGER, -- REFERENCES ENUM_CUDA_FUNC_CACHE_CONFIG(id)
registersPerThread INTEGER NOT NULL, -- Number of registers required for each thread executing the kernel.
gridX INTEGER NOT NULL, -- X-dimension grid size.
gridY INTEGER NOT NULL, -- Y-dimension grid size.
gridZ INTEGER NOT NULL, -- Z-dimension grid size.
blockX INTEGER NOT NULL, -- X-dimension block size.
blockY INTEGER NOT NULL, -- Y-dimension block size.
blockZ INTEGER NOT NULL, -- Z-dimension block size.
staticSharedMemory INTEGER NOT NULL, -- Static shared memory allocated for the kernel (B).
dynamicSharedMemory INTEGER NOT NULL, -- Dynamic shared memory reserved for the kernel (B).
localMemoryPerThread INTEGER NOT NULL, -- Amount of local memory reserved for each thread (B).
localMemoryTotal INTEGER NOT NULL, -- Total amount of local memory reserved for the kernel (B).
gridId INTEGER NOT NULL, -- Unique grid ID of the kernel assigned at runtime.
sharedMemoryExecuted INTEGER, -- Shared memory size set by the driver.
graphNodeId INTEGER, -- REFERENCES CUDA_GRAPH_NODE_EVENTS(graphNodeId)
sharedMemoryLimitConfig INTEGER, -- REFERENCES ENUM_CUDA_SHARED_MEM_LIMIT_CONFIG(id)
qmdBulkReleaseDone INTEGER, -- QMD bulk release done timestamp from CWD events.
qmdPreexitDone INTEGER, -- QMD pre-exit done timestamp from CWD events.
qmdLastCtaDone INTEGER, -- QMD last CTA done timestamp from CWD events.
graphId INTEGER, -- Kernel graph ID.
correlationIdQmd INTEGER, -- Correlation ID of the QMD associated with the kernel.
clusterX INTEGER, -- Cluster X dimension.
clusterY INTEGER, -- Cluster Y dimension.
clusterZ INTEGER, -- Cluster Z dimension.
clusterSchedulingPolicy INTEGER, -- Cluster scheduling policy.
maxPotentialClusterSize INTEGER, -- Maximum potential cluster size.
maxActiveClusters INTEGER, -- Maximum active clusters.
sharedMemoryRequestedPercentage INTEGER, -- Shared memory requested percentage.
tensorSizeMinusOneElements TEXT NOT NULL, -- TMA descriptor tensor size minus one elements array.
sourceGraphNodeId INTEGER, -- Node used for graph instantiation or the last node update.
sourceGraphId INTEGER -- Graph containing the node used for instantiation or the last node update.
);
CREATE TABLE CUPTI_ACTIVITY_KIND_SYNCHRONIZATION (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
deviceId INTEGER NOT NULL, -- Device ID.
contextId INTEGER NOT NULL, -- Context ID.
greenContextId INTEGER, -- Green context ID.
streamId INTEGER NOT NULL, -- Stream ID.
correlationId INTEGER, -- Correlation ID of the synchronization API to which this result is associated.
globalPid INTEGER, -- Serialized GlobalId.
deprecatedSyncType INTEGER, -- Deprecated, use syncType instead. For older report, REFERENCES ENUM_CUPTI_SYNC_TYPE(id)
syncType INTEGER NOT NULL, -- REFERENCES ENUM_CUPTI_SYNC_TYPE(id)
eventId INTEGER NOT NULL, -- Event ID for which the synchronization API is called.
eventSyncId INTEGER -- CUDA Event Sync ID to link the synchronization API to associated event record API.
);
CREATE TABLE CUPTI_ACTIVITY_KIND_CUDA_EVENT (
timestamp INTEGER, -- The device-side CUDA Event completion timestamp. 0 if not collected.
deviceId INTEGER NOT NULL, -- Device ID.
contextId INTEGER NOT NULL, -- Context ID.
greenContextId INTEGER, -- Green context ID.
streamId INTEGER NOT NULL, -- Stream ID.
correlationId INTEGER, -- Correlation ID of the event record API to which this result is associated.
globalPid INTEGER, -- Serialized GlobalId.
eventId INTEGER NOT NULL, -- Event ID for which the event record API is called.
eventSyncId INTEGER -- CUDA Event Sync ID to link event record API to related synchronization APIs.
);
CREATE TABLE CUPTI_ACTIVITY_KIND_GRAPH_HOST_NODE_AND_HOST_LAUNCH (
-- This table includes both CUPTI_ACTIVITY_KIND_GRAPH_HOST_NODE and CUPTI_ACTIVITY_KIND_HOST_LAUNCH events. These are stored as a single internal event type in Nsys reports. For HOST_LAUNCH events, graphNodeId and graphId fields will be NULL.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
deviceId INTEGER NOT NULL, -- Device ID.
contextId INTEGER NOT NULL, -- Context ID.
greenContextId INTEGER, -- Green context ID.
streamId INTEGER NOT NULL, -- Stream ID.
correlationId INTEGER, -- REFERENCES CUPTI_ACTIVITY_KIND_RUNTIME(correlationId)
globalPid INTEGER, -- Serialized GlobalId.
graphNodeId INTEGER, -- REFERENCES CUDA_GRAPH_NODE_EVENTS(graphNodeId)
graphId INTEGER -- REFERENCES CUDA_GRAPH_EVENTS(graphId)
);
CREATE TABLE CUPTI_ACTIVITY_KIND_GRAPH_TRACE (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
deviceId INTEGER NOT NULL, -- Device ID.
contextId INTEGER NOT NULL, -- Context ID.
greenContextId INTEGER, -- Green context ID.
streamId INTEGER NOT NULL, -- Stream ID.
correlationId INTEGER, -- REFERENCES CUPTI_ACTIVITY_KIND_RUNTIME(correlationId)
globalPid INTEGER, -- Serialized GlobalId.
graphId INTEGER NOT NULL, -- REFERENCES CUDA_GRAPH_EVENTS(graphId)
graphExecId INTEGER NOT NULL -- REFERENCES CUDA_GRAPH_EVENTS(graphExecId)
);
CREATE TABLE CUPTI_ACTIVITY_KIND_OVERHEAD (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- ID used to identify events that this function call has triggered.
nameId INTEGER, -- REFERENCES StringIds(id) -- Function name
overheadType INTEGER NOT NULL -- REFERENCES ENUM_CUPTI_OVERHEAD_TYPE(id)
);
CREATE TABLE CUPTI_ACTIVITY_KIND_JIT (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- ID used to identify events that this function call has triggered.
nameId INTEGER, -- REFERENCES StringIds(id) -- Function name
overheadType INTEGER NOT NULL, -- REFERENCES ENUM_CUPTI_OVERHEAD_TYPE(id)
jitEntryType INTEGER NOT NULL, -- REFERENCES ENUM_CUPTI_JIT_ENTRY_TYPE(id)
jitOperationType INTEGER NOT NULL, -- REFERENCES ENUM_CUPTI_JIT_OPERATION_TYPE(id)
deviceId INTEGER, -- Device ID.
jitOperationCorrelationId INTEGER,
cacheSize INTEGER,
cachePath TEXT
);
CREATE TABLE CUDA_HOST_CALLBACK (
-- Host-side callback functions triggered from CUDA streams. These represent CPU-side execution from CUDA graph host function nodes or cudaLaunchHostFunc() calls. Corresponding device-side activities are stored at the CUPTI_ACTIVITY_KIND_GRAPH_HOST_NODE_AND_HOST_LAUNCH table.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- ID used to identify events that this function call has triggered.
nameId INTEGER -- REFERENCES StringIds(id) -- Function name
);
CREATE TABLE CUDNN_EVENTS (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
nameId INTEGER NOT NULL -- REFERENCES StringIds(id) -- Function name
);
CREATE TABLE CUBLAS_EVENTS (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
nameId INTEGER NOT NULL -- REFERENCES StringIds(id) -- Function name
);
CREATE TABLE CUDA_GRAPH_NODE_EVENTS (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
graphNodeId INTEGER NOT NULL, -- Graph node ID.
originalGraphNodeId INTEGER, -- Reference to the original graph node ID, if cloned node.
graphId INTEGER -- Graph containing this node.
);
CREATE TABLE CUDA_GRAPH_EVENTS (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
graphId INTEGER, -- Graph ID.
originalGraphId INTEGER, -- Reference to the original graph ID, if cloned.
graphExecId INTEGER -- Executable graph ID.
);
CREATE TABLE CUPTI_ACTIVITY_KIND_RUNTIME (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- ID used to identify events that this function call has triggered.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
returnValue INTEGER NOT NULL, -- Return value of the function call.
callchainId INTEGER -- REFERENCES CUDA_CALLCHAINS(id)
);
CREATE TABLE CUPTI_ACTIVITY_KIND_BLOCK_TRACE (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
deviceId INTEGER, -- Device ID.
correlationId INTEGER, -- Correlation ID of the event record API to which this result is associated.
nodeId INTEGER, -- Node ID of the event record API to which this result is associated.
SMId INTEGER, -- SM ID of the event on which the particular event was running.
globalPid INTEGER, -- Serialized GlobalId.
BlockID INTEGER NOT NULL, -- Block ID.
UGPUId INTEGER, -- uGPU ID of the event on which the particular event was running.
CGAId INTEGER -- CGA ID of the event on which the particular event was running.
);
CREATE TABLE CUPTI_ACTIVITY_KIND_BLOCK_PHASE_TRACE (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
deviceId INTEGER, -- Device ID.
correlationId INTEGER, -- Correlation ID of the event record API to which this result is associated.
nodeId INTEGER, -- Node ID of the event record API to which this result is associated.
SMId INTEGER, -- SM ID of the event on which the particular event was running.
globalPid INTEGER, -- Serialized GlobalId.
BlockID INTEGER NOT NULL, -- Block ID.
phase1Timestamp INTEGER NOT NULL, -- Phase start timestamp.
phase2Timestamp INTEGER NOT NULL, -- Phase stop timestamp.
UGPUId INTEGER, -- uGPU ID of the event on which the particular event was running.
CGAId INTEGER -- CGA ID of the event on which the particular event was running.
);
CREATE TABLE CUPTI_ACTIVITY_KIND_WARP_TRACE (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
deviceId INTEGER, -- Device ID.
correlationId INTEGER, -- Correlation ID of the event record API to which this result is associated.
nodeId INTEGER, -- Node ID of the event record API to which this result is associated.
SMId INTEGER, -- SM ID of the event on which the particular event was running.
globalPid INTEGER, -- Serialized GlobalId.
BlockID INTEGER NOT NULL, -- Block ID.
WarpID INTEGER NOT NULL, -- Warp ID.
UGPUId INTEGER, -- uGPU ID of the event on which the particular event was running.
CGAId INTEGER -- CGA ID of the event on which the particular event was running.
);
CREATE TABLE CUPTI_ACTIVITY_KIND_WARP_PHASE_TRACE (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
deviceId INTEGER, -- Device ID.
correlationId INTEGER, -- Correlation ID of the event record API to which this result is associated.
nodeId INTEGER, -- Node ID of the event record API to which this result is associated.
SMId INTEGER, -- SM ID of the event on which the particular event was running.
globalPid INTEGER, -- Serialized GlobalId.
BlockID INTEGER NOT NULL, -- Block ID.
WarpID INTEGER NOT NULL, -- Warp ID.
EventName INTEGER NOT NULL, -- Event Name.
InternalEventCount INTEGER NOT NULL, -- Internal event count.
eventType INTEGER NOT NULL, -- Event type, 0 = range, 1 = marker, 2 = warp start/end
WarpEventIds TEXT NOT NULL, -- warp event ids.
WarpEventTimestampOffsets TEXT NOT NULL, -- warp event timestamp offsets.
UGPUId INTEGER, -- uGPU ID of the event on which the particular event was running.
CGAId INTEGER, -- CGA ID of the event on which the particular event was running.
PayloadType INTEGER, -- Payload type of the primary IKET event (CUpti_ActivityIketPayloadType enum value).
PayloadVal INTEGER -- Raw payload value of the primary IKET event, interpreted according to PayloadType.
);
CREATE TABLE CUDA_UM_CPU_PAGE_FAULT_EVENTS (
start INTEGER NOT NULL, -- Event start timestamp (ns).
globalPid INTEGER NOT NULL, -- Serialized GlobalId.
address INTEGER NOT NULL, -- Virtual address of the page that faulted.
originalFaultPc INTEGER, -- Program counter of the CPU instruction that caused the page fault.
CpuInstruction INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
module INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Module name
unresolvedFaultPc INTEGER, -- True if the program counter was not resolved.
sourceFile INTEGER, -- Source file where the page fault occurred.
sourceLine INTEGER -- Source line number that caused the page fault in the source file.
);
CREATE TABLE CUDA_UM_GPU_PAGE_FAULT_EVENTS (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalPid INTEGER NOT NULL, -- Serialized GlobalId.
deviceId INTEGER NOT NULL, -- Device ID.
address INTEGER NOT NULL, -- Virtual address of the page that faulted.
numberOfPageFaults INTEGER NOT NULL, -- Number of page faults for the same page.
faultAccessType INTEGER NOT NULL -- REFERENCES ENUM_CUDA_UNIF_MEM_ACCESS_TYPE(id)
);
CREATE TABLE CUDA_GPU_MEMORY_USAGE_EVENTS (
start INTEGER NOT NULL, -- Event start timestamp (ns).
globalPid INTEGER NOT NULL, -- Serialized GlobalId.
deviceId INTEGER NOT NULL, -- Device ID.
contextId INTEGER NOT NULL, -- Context ID.
address INTEGER NOT NULL, -- Virtual address of the allocation/deallocation.
pc INTEGER NOT NULL, -- Program counter of the allocation/deallocation.
bytes INTEGER NOT NULL, -- Number of bytes allocated/deallocated (B).
memKind INTEGER NOT NULL, -- REFERENCES ENUM_CUDA_MEM_KIND(id)
memoryOperationType INTEGER NOT NULL, -- REFERENCES ENUM_CUDA_DEV_MEM_EVENT_OPER(id)
name TEXT, -- Variable name, if available.
correlationId INTEGER, -- REFERENCES CUPTI_ACTIVITY_KIND_RUNTIME(correlationId)
streamId INTEGER, -- Stream ID.
localMemoryPoolAddress INTEGER, -- Base address of the local memory pool used
localMemoryPoolReleaseThreshold INTEGER, -- Release threshold of the local memory pool used
localMemoryPoolSize INTEGER, -- Size of the local memory pool used
localMemoryPoolUtilizedSize INTEGER, -- Utilized size of the local memory pool used
importedMemoryPoolAddress INTEGER, -- Base address of the imported memory pool used
importedMemoryPoolProcessId INTEGER -- Process ID of the imported memory pool used
);
CREATE TABLE CUDA_GPU_MEMORY_POOL_EVENTS (
start INTEGER NOT NULL, -- Event start timestamp (ns).
globalPid INTEGER NOT NULL, -- Serialized GlobalId.
deviceId INTEGER NOT NULL, -- Device ID.
address INTEGER NOT NULL, -- The base virtual address of the memory pool.
operationType INTEGER NOT NULL, -- REFERENCES ENUM_CUDA_MEMPOOL_OPER(id)
poolType INTEGER NOT NULL, -- REFERENCES ENUM_CUDA_MEMPOOL_TYPE(id)
correlationId INTEGER, -- REFERENCES CUPTI_ACTIVITY_KIND_RUNTIME(correlationId)
minBytesToKeep INTEGER, -- Minimum number of bytes to keep of the memory pool.
localMemoryPoolReleaseThreshold INTEGER, -- Release threshold of the local memory pool used
localMemoryPoolSize INTEGER, -- Size of the local memory pool used
localMemoryPoolUtilizedSize INTEGER -- Utilized size of the local memory pool used
);
CREATE TABLE CUDA_CALLCHAINS (
id INTEGER NOT NULL, -- Part of PRIMARY KEY (id, stackDepth).
symbol INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
module INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Module name
unresolved INTEGER, -- True if the symbol was not resolved.
originalIP INTEGER, -- Instruction pointer value.
stackDepth INTEGER NOT NULL, -- Zero-base index of the given function in call stack.
PRIMARY KEY (id, stackDepth)
);
CREATE TABLE MPI_RANKS (
-- Mapping of global thread IDs (gtid) to MPI ranks
globalTid INTEGER NOT NULL, -- Serialized GlobalId.
rank INTEGER NOT NULL -- MPI rank
);
CREATE TABLE MPI_P2P_EVENTS (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER, -- Event end timestamp (ns).
globalTid INTEGER, -- Serialized GlobalId.
textId INTEGER, -- REFERENCES StringIds(id) -- Registered NVTX domain/string
commHandle INTEGER, -- MPI communicator handle.
commUid INTEGER, -- Globally unique MPI communicator ID.
tag INTEGER, -- MPI message tag
remoteRank INTEGER, -- MPI remote rank (destination or source)
size INTEGER, -- MPI message size in bytes
requestHandle INTEGER -- MPI request handle.
);
CREATE TABLE MPI_COLLECTIVES_EVENTS (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER, -- Event end timestamp (ns).
globalTid INTEGER, -- Serialized GlobalId.
textId INTEGER, -- REFERENCES StringIds(id) -- Registered NVTX domain/string
commHandle INTEGER, -- MPI communicator handle.
commUid INTEGER, -- Globally unique MPI communicator ID.
rootRank INTEGER, -- Root rank in the collective
size INTEGER, -- MPI message size in bytes (send size for bidirectional ops)
recvSize INTEGER, -- MPI receive size in bytes
requestHandle INTEGER -- MPI request handle.
);
CREATE TABLE MPI_START_WAIT_EVENTS (
-- MPI_Start*, MPI_Test* and MPI_Wait*
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER, -- Event end timestamp (ns).
globalTid INTEGER, -- Serialized GlobalId.
textId INTEGER, -- REFERENCES StringIds(id) -- Registered NVTX domain/string
requestHandle INTEGER -- MPI request handle.
);
CREATE TABLE MPI_OTHER_EVENTS (
-- MPI events without additional parameters
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER, -- Event end timestamp (ns).
globalTid INTEGER, -- Serialized GlobalId.
textId INTEGER -- REFERENCES StringIds(id) -- Registered NVTX domain/string
);
CREATE TABLE UCP_WORKERS (
globalTid INTEGER NOT NULL, -- Serialized GlobalId.
workerUid INTEGER NOT NULL -- UCP worker UID
);
CREATE TABLE UCP_SUBMIT_EVENTS (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER, -- Event end timestamp (ns).
globalTid INTEGER, -- Serialized GlobalId.
textId INTEGER, -- REFERENCES StringIds(id) -- Registered NVTX domain/string
bufferAddr INTEGER, -- Address of the message buffer
packedSize INTEGER, -- Message size (packed) in bytes
peerWorkerUid INTEGER, -- Peer's UCP worker UID
tag INTEGER -- UCP message tag
);
CREATE TABLE UCP_PROGRESS_EVENTS (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER, -- Event end timestamp (ns).
globalTid INTEGER, -- Serialized GlobalId.
textId INTEGER, -- REFERENCES StringIds(id) -- Registered NVTX domain/string
bufferAddr INTEGER, -- Address of the message buffer
packedSize INTEGER, -- Message size (packed) in bytes
peerWorkerUid INTEGER, -- Peer's UCP worker UID
tag INTEGER -- UCP message tag
);
CREATE TABLE UCP_EVENTS (
-- UCP events without additional parameters
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER, -- Event end timestamp (ns).
globalTid INTEGER, -- Serialized GlobalId.
textId INTEGER -- REFERENCES StringIds(id) -- Registered NVTX domain/string
);
CREATE TABLE NVTX_EVENTS (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER, -- Event end timestamp (ns).
eventType INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_TYPE(id)
rangeId INTEGER, -- Correlation ID returned from a nvtxRangeStart call.
category INTEGER, -- User-controlled ID that can be used to group events.
color INTEGER, -- Encoded ARGB color value.
text TEXT, -- Explicit name/text (non-registered string)
globalTid INTEGER, -- Serialized GlobalId.
endGlobalTid INTEGER, -- Serialized GlobalId.
textId INTEGER, -- REFERENCES StringIds(id) -- Registered NVTX domain/string
domainId INTEGER, -- User-controlled ID that can be used to group events.
uint64Value INTEGER, -- One of possible payload value union members.
int64Value INTEGER, -- One of possible payload value union members.
doubleValue REAL, -- One of possible payload value union members.
uint32Value INTEGER, -- One of possible payload value union members.
int32Value INTEGER, -- One of possible payload value union members.
floatValue REAL, -- One of possible payload value union members.
jsonTextId INTEGER, -- One of possible payload value union members.
jsonText TEXT, -- One of possible payload value union members.
binaryData TEXT, -- Binary payload. See docs for format.
eventId INTEGER NOT NULL PRIMARY KEY -- Unique NVTX event row identifier.
);
CREATE TABLE NVTXT_EVENTS (
-- NVTXT events imported from nvtxt file.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER, -- Event end timestamp (ns).
eventType INTEGER, -- REFERENCES ENUM_NSYS_EVENT_TYPE(id)
sourceId INTEGER NOT NULL, -- String id for NVTXT source filename
category INTEGER, -- User-controlled ID that can be used to group events.
color INTEGER, -- Encoded ARGB color value.
text TEXT, -- Explicit name/text (non-registered string)
globalTid INTEGER, -- Serialized GlobalThreadId.
domainId INTEGER, -- User-controlled ID that can be used to group events.
uint64Value INTEGER, -- One of possible payload value union members.
int64Value INTEGER, -- One of possible payload value union members.
doubleValue REAL, -- One of possible payload value union members.
uint32Value INTEGER, -- One of possible payload value union members.
int32Value INTEGER, -- One of possible payload value union members.
floatValue REAL -- One of possible payload value union members.
);
CREATE TABLE NVTXT_META_EVENTS (
-- NVTXT meta events imported from nvtxt file.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER, -- Event end timestamp (ns).
sourceId INTEGER NOT NULL, -- String id for NVTXT source filename
eventType INTEGER, -- REFERENCES ENUM_NSYS_EVENT_TYPE(id)
globalThreadId INTEGER, -- Serialized GlobalThreadId.
threadName TEXT, -- Name alias of the thread
globalProcessId INTEGER, -- Serialized GlobalProcessId.
processName TEXT, -- Name alias of the process
globalVmId INTEGER, -- Serialized GlobalVmId.
fileName TEXT -- Name of the NVTXT file
);
CREATE TABLE OPENGL_API (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_TYPE(id)
globalTid INTEGER, -- Serialized GlobalId.
endGlobalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- First ID matching an API call to GPU workloads.
endCorrelationId INTEGER, -- Last ID matching an API call to GPU workloads.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- First function name
endNameId INTEGER, -- REFERENCES StringIds(id) -- Last function name
returnValue INTEGER NOT NULL, -- Return value of the function call.
frameId INTEGER, -- Index of the graphics frame starting from 1.
contextId INTEGER, -- Context ID.
gpu INTEGER, -- GPU index.
display INTEGER -- Display ID.
);
CREATE TABLE OPENGL_WORKLOAD (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_TYPE(id)
globalTid INTEGER, -- Serialized GlobalId.
endGlobalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- First ID matching an API call to GPU workloads.
endCorrelationId INTEGER, -- Last ID matching an API call to GPU workloads.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- First function name
endNameId INTEGER, -- REFERENCES StringIds(id) -- Last function name
returnValue INTEGER NOT NULL, -- Return value of the function call.
frameId INTEGER, -- Index of the graphics frame starting from 1.
contextId INTEGER, -- Context ID.
gpu INTEGER, -- GPU index.
display INTEGER -- Display ID.
);
CREATE TABLE KHR_DEBUG_EVENTS (
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_TYPE(id)
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER, -- Event end timestamp (ns).
textId INTEGER, -- REFERENCES StringIds(id) -- Debug marker/group text
globalTid INTEGER, -- Serialized GlobalId.
source INTEGER, -- REFERENCES ENUM_OPENGL_DEBUG_SOURCE(id)
khrdType INTEGER, -- REFERENCES ENUM_OPENGL_DEBUG_TYPE(id)
id INTEGER, -- KHR event ID.
severity INTEGER, -- REFERENCES ENUM_OPENGL_DEBUG_SEVERITY(id)
correlationId INTEGER, -- ID used to correlate KHR CPU trace to GPU trace.
context INTEGER -- Context ID.
);
CREATE TABLE OSRT_API (
-- OS runtime libraries traced to gather information about low-level userspace APIs.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
returnValue INTEGER NOT NULL, -- Return value of the function call.
nestingLevel INTEGER, -- Zero-base index of the nesting level.
callchainId INTEGER NOT NULL, -- REFERENCES OSRT_CALLCHAINS(id)
argumentsId INTEGER NOT NULL -- REFERENCES OSRT_ARGUMENTS(id) -- Experimental.
);
CREATE TABLE OSRT_CALLCHAINS (
-- Callchains attached to OSRT events, depending on selected profiling settings.
id INTEGER NOT NULL, -- Part of PRIMARY KEY (id, stackDepth).
symbol INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
module INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Module name
kernelMode INTEGER, -- True if kernel mode.
thumbCode INTEGER, -- True if thumb code.
unresolved INTEGER, -- True if the symbol was not resolved.
specialEntry INTEGER, -- True if artifical entry added during processing callchain.
originalIP INTEGER, -- Instruction pointer value.
unwindMethod INTEGER, -- REFERENCES ENUM_STACK_UNWIND_METHOD(id)
stackDepth INTEGER NOT NULL, -- Zero-base index of the given function in call stack.
PRIMARY KEY (id, stackDepth)
);
CREATE TABLE OSRT_ARGUMENTS (
-- Arguments OSRT functions were called with. This is an experimental feature and arguments are collected for some specific functions only. Please avoid relying on the content for now.
id INTEGER NOT NULL, -- Part of PRIMARY KEY (id, argumentIndex).
value INTEGER NOT NULL, -- Value of the argument.
argumentIndex INTEGER NOT NULL, -- Zero-base index of the argument.
PRIMARY KEY (id, argumentIndex)
);
CREATE TABLE OSRT_FILE_ACCESS_EVENTS (
-- OS Runtime events related to file accesses (opening, closing, reading, and writing).
fileAccessId INTEGER NOT NULL, -- REFERENCES OSRT_FILE_ACCESS_DESCRIPTORS(fileAccessId)
threadId INTEGER NOT NULL, -- Thread ID.
startedAt INTEGER NOT NULL, -- Event start timestamp (ns).
endedAt INTEGER NOT NULL, -- Event end timestamp (ns).
eventType INTEGER NOT NULL, -- REFERENCES ENUM_OSRT_FILE_ACCESS_EVENT_TYPE(id)
apiCallId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
bytesProcessed INTEGER NOT NULL, -- Actual bytes read/written.
context TEXT NOT NULL -- Additional information about the event
);
CREATE TABLE OSRT_FILE_ACCESS_DESCRIPTORS (
-- Metadata of all the file accesses that were made by the OS during the recording.
fileAccessId INTEGER NOT NULL, -- File Access Id.
processId INTEGER NOT NULL, -- Process ID.
openedAt INTEGER NOT NULL, -- The time when the file was opened (ns).
closedAt INTEGER NOT NULL, -- The time when the file was closed (ns).
filePath TEXT NOT NULL -- The opened file path.
);
CREATE TABLE PROFILER_OVERHEAD (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalTid INTEGER, -- Serialized GlobalId.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
overheadType INTEGER NOT NULL -- REFERENCES ENUM_CUPTI_OVERHEAD_TYPE(id)
);
CREATE TABLE SCHED_EVENTS (
-- Thread scheduling events.
start INTEGER NOT NULL, -- Event start timestamp (ns).
cpu INTEGER NOT NULL, -- ID of CPU this thread was scheduled in or out.
isSchedIn INTEGER NOT NULL, -- 0 if thread was scheduled out, non-zero otherwise.
globalTid INTEGER, -- Serialized GlobalId.
threadState INTEGER, -- REFERENCES ENUM_SAMPLING_THREAD_STATE(id)
threadBlock INTEGER -- REFERENCES ENUM_SCHEDULING_THREAD_BLOCK(id)
);
CREATE TABLE COMPOSITE_EVENTS (
-- Thread sampling events.
id INTEGER NOT NULL PRIMARY KEY, -- ID of the composite event.
start INTEGER NOT NULL, -- Event start timestamp (ns).
cpu INTEGER, -- ID of CPU this thread was running on.
threadState INTEGER, -- REFERENCES ENUM_SAMPLING_THREAD_STATE(id)
globalTid INTEGER, -- Serialized GlobalId.
cpuCycles INTEGER NOT NULL -- Value of Performance Monitoring Unit (PMU) counter.
);
CREATE TABLE SAMPLING_CALLCHAINS (
-- Callchain entries obtained from composite events, used to construct function table views.
id INTEGER NOT NULL, -- REFERENCES COMPOSITE_EVENTS(id)
symbol INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
module INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Module name
kernelMode INTEGER, -- True if kernel mode.
thumbCode INTEGER, -- True if thumb code.
unresolved INTEGER, -- True if the symbol was not resolved.
specialEntry INTEGER, -- True if artifical entry added during processing callchain.
originalIP INTEGER, -- Instruction pointer value.
unwindMethod INTEGER, -- REFERENCES ENUM_STACK_UNWIND_METHOD(id)
stackDepth INTEGER NOT NULL, -- Zero-base index of the given function in call stack.
PRIMARY KEY (id, stackDepth)
);
CREATE TABLE PERF_EVENT_SOC_OR_CPU_RAW_EVENT (
-- SoC and CPU raw event values from Sampled Performance Counters.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
vmId INTEGER, -- VM ID.
componentId INTEGER, -- REFERENCES TARGET_INFO_COMPONENT(componentId)
eventId INTEGER, -- REFERENCES TARGET_INFO_PERF_METRIC(id)
count INTEGER -- Counter data value
);
CREATE TABLE PERF_EVENT_SOC_OR_CPU_METRIC_EVENT (
-- SoC and CPU metric values from Sampled Performance Counters.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
vmId INTEGER, -- VM ID.
componentId INTEGER, -- REFERENCES TARGET_INFO_COMPONENT(componentId)
metricId INTEGER, -- REFERENCES TARGET_INFO_PERF_METRIC(id)
value REAL -- Metric data value
);
CREATE TABLE DX12_API (
id INTEGER NOT NULL PRIMARY KEY,
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- First ID matching an API call to GPU workloads.
endCorrelationId INTEGER, -- Last ID matching an API call to GPU workloads.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
shortContextId INTEGER, -- Short form of the COM interface object address.
frameId INTEGER, -- Index of the graphics frame starting from 1.
color INTEGER, -- Encoded ARGB color value.
textId INTEGER, -- REFERENCES StringIds(id) -- PIX marker text
commandListType INTEGER, -- REFERENCES ENUM_D3D12_CMD_LIST_TYPE(id)
objectNameId INTEGER, -- REFERENCES StringIds(id) -- D3D12 object name
longContextId INTEGER -- Long form of the COM interface object address.
);
CREATE TABLE DX12_WORKLOAD (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- First ID matching an API call to GPU workloads.
endCorrelationId INTEGER, -- Last ID matching an API call to GPU workloads.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
shortContextId INTEGER, -- Short form of the COM interface object address.
frameId INTEGER, -- Index of the graphics frame starting from 1.
gpu INTEGER, -- GPU index.
color INTEGER, -- Encoded ARGB color value.
textId INTEGER, -- REFERENCES StringIds(id) -- PIX marker text
commandListType INTEGER, -- REFERENCES ENUM_D3D12_CMD_LIST_TYPE(id)
objectNameId INTEGER, -- REFERENCES StringIds(id) -- D3D12 object name
longContextId INTEGER -- Long form of the COM interface object address.
);
CREATE TABLE DX12_MEMORY_OPERATION (
gpu INTEGER, -- GPU index.
rangeStart INTEGER, -- Time offset denoting the beginning of a memory range (B).
rangeEnd INTEGER, -- Time offset denoting the end of a memory range (B).
subresourceId INTEGER, -- Subresource index.
heapType INTEGER, -- REFERENCES ENUM_D3D12_HEAP_TYPE(id)
heapFlags INTEGER, -- REFERENCES ENUM_D3D12_HEAP_FLAGS(id)
cpuPageProperty INTEGER, -- REFERENCES ENUM_D3D12_PAGE_PROPERTY(id)
nvApiFlags INTEGER, -- NV specific flags. See docs for specifics.
objectHandle INTEGER, -- Handle to the graphics object created or modified by the memory operation.
bindTargetHandle INTEGER, -- Handle to the target resource for bind operations.
memorySize INTEGER, -- Size of the memory allocation (B).
memoryOffset INTEGER, -- Offset within the memory allocation (B).
resourceFlags INTEGER, -- Combination of D3D12_RESOURCE_FLAGS enum values specifying resource usage options.
dimension INTEGER, -- D3D12_RESOURCE_DIMENSION enum value specifying the resource type.
traceEventId INTEGER NOT NULL -- REFERENCES DX12_API(id)
);
CREATE TABLE DXGI_API (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- First ID matching an API call to GPU workloads.
endCorrelationId INTEGER, -- Last ID matching an API call to GPU workloads.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
shortContextId INTEGER, -- Short form of the COM interface object address.
frameId INTEGER, -- Index of the graphics frame starting from 1.
color INTEGER, -- Encoded ARGB color value.
textId INTEGER -- REFERENCES StringIds(id) -- PIX marker text
);
CREATE TABLE NVAPI_API (
id INTEGER NOT NULL PRIMARY KEY,
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
longContextId INTEGER -- Long form of the COM interface object address.
);
CREATE TABLE NVAPI_MEMORY_OPERATION (
rangeStart INTEGER, -- Time offset denoting the beginning of a memory range (B).
rangeEnd INTEGER, -- Time offset denoting the end of a memory range (B).
subresourceId INTEGER, -- Subresource index.
heapType INTEGER, -- REFERENCES ENUM_D3D12_HEAP_TYPE(id)
heapFlags INTEGER, -- REFERENCES ENUM_D3D12_HEAP_FLAGS(id)
memoryProperty INTEGER, -- REFERENCES ENUM_D3D12_PAGE_PROPERTY(id)
nvApiFlags INTEGER, -- NV specific flags. See docs for specifics.
traceEventId INTEGER NOT NULL -- REFERENCES NVAPI_API(id)
);
CREATE TABLE VULKAN_API (
id INTEGER NOT NULL PRIMARY KEY,
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- First ID matching an API call to GPU workloads.
endCorrelationId INTEGER, -- Last ID matching an API call to GPU workloads.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
contextId INTEGER, -- Short form of the interface object address.
objectNameId INTEGER, -- REFERENCES StringIds(id) -- Vulkan object name
interfaceAddress INTEGER -- Interface object address.
);
CREATE TABLE VULKAN_WORKLOAD (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- First ID matching an API call to GPU workloads.
endCorrelationId INTEGER, -- Last ID matching an API call to GPU workloads.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
gpu INTEGER, -- GPU index.
contextId INTEGER, -- Short form of the interface object address.
color INTEGER, -- Encoded ARGB color value.
textId INTEGER -- REFERENCES StringIds(id) -- Vulkan CPU debug marker string
);
CREATE TABLE VULKAN_DEBUG_API (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- First ID matching an API call to GPU workloads.
endCorrelationId INTEGER, -- Last ID matching an API call to GPU workloads.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
contextId INTEGER, -- Short form of the interface object address.
color INTEGER, -- Encoded ARGB color value.
textId INTEGER -- REFERENCES StringIds(id) -- Vulkan CPU debug marker string
);
CREATE TABLE VULKAN_PIPELINE_CREATION_EVENTS (
id INTEGER NOT NULL PRIMARY KEY, -- ID of the pipeline creation event.
duration INTEGER, -- Event duration (ns).
flags INTEGER, -- REFERENCES ENUM_VULKAN_PIPELINE_CREATION_FLAGS(id)
traceEventId INTEGER NOT NULL -- REFERENCES VULKAN_API(id) -- ID of the attached vulkan API.
);
CREATE TABLE VULKAN_PIPELINE_STAGE_EVENTS (
id INTEGER NOT NULL PRIMARY KEY, -- ID of the pipeline stage event.
duration INTEGER, -- Event duration (ns).
flags INTEGER, -- REFERENCES ENUM_VULKAN_PIPELINE_CREATION_FLAGS(id)
creationEventId INTEGER NOT NULL -- REFERENCES VULKAN_PIPELINE_CREATION_EVENTS(id) -- ID of the attached pipeline creation event.
);
CREATE TABLE VULKAN_MEMORY_OPERATION (
id INTEGER NOT NULL PRIMARY KEY, -- ID of the memory operation.
gpu INTEGER, -- GPU index.
rangeStart INTEGER, -- Time offset denoting the beginning of a memory range (B).
rangeEnd INTEGER, -- Time offset denoting the end of a memory range (B).
contextId INTEGER, -- Interface object address.
objectHandle INTEGER, -- Handle to the graphics object created or modified by the memory operation.
memorySize INTEGER, -- Size of the memory allocation (B).
memoryOffset INTEGER, -- Offset within the memory allocation (B).
bindTargetHandle INTEGER, -- Handle to the target resource for bind operations.
traceEventId INTEGER NOT NULL -- REFERENCES VULKAN_API(id)
);
CREATE TABLE VULKAN_MEMORY_TYPES (
id INTEGER NOT NULL PRIMARY KEY, -- ID of the memory type entry.
heapIndex INTEGER, -- Index of the heap that this memory type references.
heapFlags INTEGER, -- REFERENCES ENUM_VULKAN_HEAP_FLAGS(id)
memoryProperty INTEGER, -- REFERENCES ENUM_VULKAN_MEMORY_PROPERTY_FLAGS(id)
heapType INTEGER, -- REFERENCES ENUM_VULKAN_HEAP_TYPE(id)
memOpId INTEGER NOT NULL -- REFERENCES VULKAN_MEMORY_OPERATION(id) -- ID of the parent memory operation.
);
CREATE TABLE GPU_CONTEXT_SWITCH_EVENTS (
tag INTEGER NOT NULL, -- REFERENCES ENUM_GPU_CTX_SWITCH(id)
vmId INTEGER NOT NULL, -- VM ID.
seqNo INTEGER NOT NULL, -- Sequential event number.
contextId INTEGER NOT NULL, -- Context ID.
timestamp INTEGER NOT NULL, -- Event start timestamp (ns).
globalPid INTEGER, -- Serialized GlobalId.
gpuId INTEGER -- GPU index.
);
CREATE TABLE OPENMP_EVENT_KIND_THREAD (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- Currently unused.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Event name
eventKind INTEGER, -- REFERENCES ENUM_OPENMP_EVENT_KIND(id)
threadId INTEGER, -- Internal thread sequence starting from 1.
threadType INTEGER -- REFERENCES ENUM_OPENMP_THREAD(id)
);
CREATE TABLE OPENMP_EVENT_KIND_PARALLEL (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- Currently unused.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Event name
eventKind INTEGER, -- REFERENCES ENUM_OPENMP_EVENT_KIND(id)
parallelId INTEGER, -- Internal parallel region sequence starting from 1.
parentTaskId INTEGER -- ID for task that creates this parallel region.
);
CREATE TABLE OPENMP_EVENT_KIND_SYNC_REGION_WAIT (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- Currently unused.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Event name
eventKind INTEGER, -- REFERENCES ENUM_OPENMP_EVENT_KIND(id)
parallelId INTEGER, -- ID of the parallel region that this event belongs to.
taskId INTEGER, -- ID of the task that this event belongs to.
kind INTEGER -- REFERENCES ENUM_OPENMP_SYNC_REGION(id)
);
CREATE TABLE OPENMP_EVENT_KIND_SYNC_REGION (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- Currently unused.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Event name
eventKind INTEGER, -- REFERENCES ENUM_OPENMP_EVENT_KIND(id)
parallelId INTEGER, -- ID of the parallel region that this event belongs to.
taskId INTEGER, -- ID of the task that this event belongs to.
kind INTEGER -- REFERENCES ENUM_OPENMP_SYNC_REGION(id)
);
CREATE TABLE OPENMP_EVENT_KIND_TASK (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- Currently unused.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Event name
eventKind INTEGER, -- REFERENCES ENUM_OPENMP_EVENT_KIND(id)
parallelId INTEGER, -- ID of the parallel region that this event belongs to.
taskId INTEGER, -- ID of the task that this event belongs to.
kind INTEGER -- REFERENCES ENUM_OPENMP_TASK_FLAG(id)
);
CREATE TABLE OPENMP_EVENT_KIND_MASTER (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- Currently unused.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Event name
eventKind INTEGER, -- REFERENCES ENUM_OPENMP_EVENT_KIND(id)
parallelId INTEGER, -- ID of the parallel region that this event belongs to.
taskId INTEGER -- ID of the task that this event belongs to.
);
CREATE TABLE OPENMP_EVENT_KIND_REDUCTION (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- Currently unused.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Event name
eventKind INTEGER, -- REFERENCES ENUM_OPENMP_EVENT_KIND(id)
parallelId INTEGER, -- ID of the parallel region that this event belongs to.
taskId INTEGER -- ID of the task that this event belongs to.
);
CREATE TABLE OPENMP_EVENT_KIND_TASK_CREATE (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- Currently unused.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Event name
eventKind INTEGER, -- REFERENCES ENUM_OPENMP_EVENT_KIND(id)
parentTaskId INTEGER, -- ID of the parent task that is creating a new task.
newTaskId INTEGER -- ID of the new task that is being created.
);
CREATE TABLE OPENMP_EVENT_KIND_TASK_SCHEDULE (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- Currently unused.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Event name
eventKind INTEGER, -- REFERENCES ENUM_OPENMP_EVENT_KIND(id)
parallelId INTEGER, -- ID of the parallel region that this event belongs to.
priorTaskId INTEGER, -- ID of the task that is being switched out.
priorTaskStatus INTEGER, -- REFERENCES ENUM_OPENMP_TASK_STATUS(id)
nextTaskId INTEGER -- ID of the task that is being switched in.
);
CREATE TABLE OPENMP_EVENT_KIND_CANCEL (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- Currently unused.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Event name
eventKind INTEGER, -- REFERENCES ENUM_OPENMP_EVENT_KIND(id)
taskId INTEGER -- ID of the task that is being cancelled.
);
CREATE TABLE OPENMP_EVENT_KIND_MUTEX_WAIT (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- Currently unused.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Event name
eventKind INTEGER, -- REFERENCES ENUM_OPENMP_EVENT_KIND(id)
kind INTEGER, -- REFERENCES ENUM_OPENMP_MUTEX(id)
waitId INTEGER, -- ID indicating the object being waited.
taskId INTEGER -- ID of the task that this event belongs to.
);
CREATE TABLE OPENMP_EVENT_KIND_CRITICAL_SECTION (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- Currently unused.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Event name
eventKind INTEGER, -- REFERENCES ENUM_OPENMP_EVENT_KIND(id)
kind INTEGER, -- REFERENCES ENUM_OPENMP_MUTEX(id)
waitId INTEGER -- ID indicating the object being held.
);
CREATE TABLE OPENMP_EVENT_KIND_MUTEX_RELEASED (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- Currently unused.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Event name
eventKind INTEGER, -- REFERENCES ENUM_OPENMP_EVENT_KIND(id)
kind INTEGER, -- REFERENCES ENUM_OPENMP_MUTEX(id)
waitId INTEGER, -- ID indicating the object being released.
taskId INTEGER -- ID of the task that this event belongs to.
);
CREATE TABLE OPENMP_EVENT_KIND_LOCK_INIT (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- Currently unused.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Event name
eventKind INTEGER, -- REFERENCES ENUM_OPENMP_EVENT_KIND(id)
kind INTEGER, -- REFERENCES ENUM_OPENMP_MUTEX(id)
waitId INTEGER -- ID indicating object being created/destroyed.
);
CREATE TABLE OPENMP_EVENT_KIND_LOCK_DESTROY (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- Currently unused.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Event name
eventKind INTEGER, -- REFERENCES ENUM_OPENMP_EVENT_KIND(id)
kind INTEGER, -- REFERENCES ENUM_OPENMP_MUTEX(id)
waitId INTEGER -- ID indicating object being created/destroyed.
);
CREATE TABLE OPENMP_EVENT_KIND_WORKSHARE (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- Currently unused.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Event name
eventKind INTEGER, -- REFERENCES ENUM_OPENMP_EVENT_KIND(id)
kind INTEGER, -- REFERENCES ENUM_OPENMP_WORK(id)
parallelId INTEGER, -- ID of the parallel region that this event belongs to.
taskId INTEGER, -- ID of the task that this event belongs to.
count INTEGER -- Measure of the quantity of work involved in the region.
);
CREATE TABLE OPENMP_EVENT_KIND_DISPATCH (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- Currently unused.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Event name
eventKind INTEGER, -- REFERENCES ENUM_OPENMP_EVENT_KIND(id)
kind INTEGER, -- REFERENCES ENUM_OPENMP_DISPATCH(id)
parallelId INTEGER, -- ID of the parallel region that this event belongs to.
taskId INTEGER -- ID of the task that this event belongs to.
);
CREATE TABLE OPENMP_EVENT_KIND_FLUSH (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- Currently unused.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Event name
eventKind INTEGER, -- REFERENCES ENUM_OPENMP_EVENT_KIND(id)
threadId INTEGER -- ID of the thread that this event belongs to.
);
CREATE TABLE D3D11_PIX_DEBUG_API (
-- D3D11 debug marker events.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- First ID matching an API call to GPU workloads.
endCorrelationId INTEGER, -- Last ID matching an API call to GPU workloads.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
shortContextId INTEGER, -- Short form of the COM interface object address.
frameId INTEGER, -- Index of the graphics frame starting from 1.
color INTEGER, -- Encoded ARGB color value.
textId INTEGER -- REFERENCES StringIds(id) -- PIX marker text
);
CREATE TABLE D3D12_PIX_DEBUG_API (
-- D3D12 debug marker events.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalTid INTEGER, -- Serialized GlobalId.
correlationId INTEGER, -- First ID matching an API call to GPU workloads.
endCorrelationId INTEGER, -- Last ID matching an API call to GPU workloads.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
shortContextId INTEGER, -- Short form of the COM interface object address.
frameId INTEGER, -- Index of the graphics frame starting from 1.
color INTEGER, -- Encoded ARGB color value.
textId INTEGER, -- REFERENCES StringIds(id) -- PIX marker text
commandListType INTEGER, -- REFERENCES ENUM_D3D12_CMD_LIST_TYPE(id)
objectNameId INTEGER, -- REFERENCES StringIds(id) -- D3D12 object name
longContextId INTEGER -- Long form of the COM interface object address.
);
CREATE TABLE WDDM_EVICT_ALLOCATION_EVENTS (
-- Raw ETW EvictAllocation events.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalTid INTEGER, -- Serialized GlobalId.
gpu INTEGER NOT NULL, -- GPU index.
allocationHandle INTEGER NOT NULL -- Global allocation handle.
);
CREATE TABLE WDDM_PAGING_QUEUE_PACKET_START_EVENTS (
-- Raw ETW PagingQueuePacketStart events.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalTid INTEGER, -- Serialized GlobalId.
gpu INTEGER NOT NULL, -- GPU index.
dxgDevice INTEGER, -- Address of an IDXGIDevice.
dxgAdapter INTEGER, -- Address of an IDXGIAdapter.
pagingQueue INTEGER NOT NULL, -- Address of the paging queue.
pagingQueuePacket INTEGER NOT NULL, -- Address of the paging queue packet.
sequenceId INTEGER NOT NULL, -- Internal sequence starting from 0.
alloc INTEGER, -- Allocation handle.
vidMmOpType INTEGER NOT NULL, -- REFERENCES ENUM_WDDM_VIDMM_OP_TYPE(id)
pagingQueueType INTEGER NOT NULL -- REFERENCES ENUM_WDDM_PAGING_QUEUE_TYPE(id)
);
CREATE TABLE WDDM_PAGING_QUEUE_PACKET_STOP_EVENTS (
-- Raw ETW PagingQueuePacketStop events.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalTid INTEGER, -- Serialized GlobalId.
gpu INTEGER NOT NULL, -- GPU index.
pagingQueue INTEGER NOT NULL, -- Address of the paging queue.
pagingQueuePacket INTEGER NOT NULL, -- Address of the paging queue packet.
sequenceId INTEGER NOT NULL -- Internal sequence starting from 0.
);
CREATE TABLE WDDM_PAGING_QUEUE_PACKET_INFO_EVENTS (
-- Raw ETW PagingQueuePacketInfo events.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalTid INTEGER, -- Serialized GlobalId.
gpu INTEGER NOT NULL, -- GPU index.
pagingQueue INTEGER NOT NULL, -- Address of the paging queue.
pagingQueuePacket INTEGER NOT NULL, -- Address of the paging queue packet.
sequenceId INTEGER NOT NULL -- Internal sequence starting from 0.
);
CREATE TABLE WDDM_QUEUE_PACKET_START_EVENTS (
-- Raw ETW QueuePacketStart events.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalTid INTEGER, -- Serialized GlobalId.
gpu INTEGER NOT NULL, -- GPU index.
context INTEGER NOT NULL, -- The context ID of WDDM queue.
dmaBufferSize INTEGER NOT NULL, -- The dma buffer size.
dmaBuffer INTEGER NOT NULL, -- The reported address of dma buffer.
queuePacket INTEGER NOT NULL, -- The address of queue packet.
progressFenceValue INTEGER NOT NULL, -- The fence value.
packetType INTEGER NOT NULL, -- REFERENCES ENUM_WDDM_PACKET_TYPE(id)
submitSequence INTEGER NOT NULL, -- Internal sequence starting from 1.
allocationListSize INTEGER NOT NULL, -- The number of allocations referenced.
patchLocationListSize INTEGER NOT NULL, -- The number of patch locations.
present INTEGER NOT NULL, -- True or False if the packet is a present packet.
engineType INTEGER NOT NULL, -- REFERENCES ENUM_WDDM_ENGINE_TYPE(id)
syncObject INTEGER -- The address of fence object.
);
CREATE TABLE WDDM_QUEUE_PACKET_STOP_EVENTS (
-- Raw ETW QueuePacketStop events.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalTid INTEGER, -- Serialized GlobalId.
gpu INTEGER NOT NULL, -- GPU index.
context INTEGER NOT NULL, -- The context ID of WDDM queue.
queuePacket INTEGER NOT NULL, -- The address of queue packet.
packetType INTEGER NOT NULL, -- REFERENCES ENUM_WDDM_PACKET_TYPE(id)
submitSequence INTEGER NOT NULL, -- Internal sequence starting from 1.
preempted INTEGER NOT NULL, -- True or False if the packet is preempted.
timeouted INTEGER NOT NULL, -- True or False if the packet is timeouted.
engineType INTEGER NOT NULL -- REFERENCES ENUM_WDDM_ENGINE_TYPE(id)
);
CREATE TABLE WDDM_QUEUE_PACKET_INFO_EVENTS (
-- Raw ETW QueuePacketInfo events.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalTid INTEGER, -- Serialized GlobalId.
gpu INTEGER NOT NULL, -- GPU index.
context INTEGER NOT NULL, -- The context ID of WDDM queue.
packetType INTEGER NOT NULL, -- REFERENCES ENUM_WDDM_PACKET_TYPE(id)
submitSequence INTEGER NOT NULL, -- Internal sequence starting from 1.
engineType INTEGER NOT NULL -- REFERENCES ENUM_WDDM_ENGINE_TYPE(id)
);
CREATE TABLE WDDM_DMA_PACKET_START_EVENTS (
-- Raw ETW DmaPacketStart events.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalTid INTEGER, -- Serialized GlobalId.
gpu INTEGER NOT NULL, -- GPU index.
context INTEGER NOT NULL, -- The context ID of WDDM queue.
queuePacketContext INTEGER NOT NULL, -- The queue packet context.
uliSubmissionId INTEGER NOT NULL, -- The queue packet submission ID.
dmaBuffer INTEGER NOT NULL, -- The reported address of dma buffer.
packetType INTEGER NOT NULL, -- REFERENCES ENUM_WDDM_PACKET_TYPE(id)
ulQueueSubmitSequence INTEGER NOT NULL, -- Internal sequence starting from 1.
quantumStatus INTEGER NOT NULL, -- The quantum Status.
engineType INTEGER NOT NULL -- REFERENCES ENUM_WDDM_ENGINE_TYPE(id)
);
CREATE TABLE WDDM_DMA_PACKET_STOP_EVENTS (
-- Raw ETW DmaPacketStop events.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalTid INTEGER, -- Serialized GlobalId.
gpu INTEGER NOT NULL, -- GPU index.
context INTEGER NOT NULL, -- The context ID of WDDM queue.
uliCompletionId INTEGER NOT NULL, -- The queue packet completion ID.
packetType INTEGER NOT NULL, -- REFERENCES ENUM_WDDM_PACKET_TYPE(id)
ulQueueSubmitSequence INTEGER NOT NULL, -- Internal sequence starting from 1.
preempted INTEGER NOT NULL, -- True or False if the packet is preempted.
engineType INTEGER NOT NULL -- REFERENCES ENUM_WDDM_ENGINE_TYPE(id)
);
CREATE TABLE WDDM_DMA_PACKET_INFO_EVENTS (
-- Raw ETW DmaPacketInfo events.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalTid INTEGER, -- Serialized GlobalId.
gpu INTEGER NOT NULL, -- GPU index.
context INTEGER NOT NULL, -- The context ID of WDDM queue.
uliCompletionId INTEGER NOT NULL, -- The queue packet completion ID.
faultedVirtualAddress INTEGER NOT NULL, -- The virtual address of faulted process.
faultedProcessHandle INTEGER NOT NULL, -- The address of faulted process.
packetType INTEGER NOT NULL, -- REFERENCES ENUM_WDDM_PACKET_TYPE(id)
ulQueueSubmitSequence INTEGER NOT NULL, -- Internal sequence starting from 1.
interruptType INTEGER NOT NULL, -- REFERENCES ENUM_WDDM_INTERRUPT_TYPE(id)
quantumStatus INTEGER NOT NULL, -- The quantum Status.
pageFaultFlags INTEGER NOT NULL, -- The page fault flag ID.
engineType INTEGER NOT NULL -- REFERENCES ENUM_WDDM_ENGINE_TYPE(id)
);
CREATE TABLE WDDM_HW_QUEUE_EVENTS (
-- Raw ETW HwQueueStart events.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalTid INTEGER, -- Serialized GlobalId.
gpu INTEGER NOT NULL, -- GPU index.
context INTEGER NOT NULL, -- The context ID of WDDM queue.
hwQueue INTEGER NOT NULL, -- The address of HW queue.
parentDxgHwQueue INTEGER NOT NULL -- The address of parent Dxg HW queue.
);
CREATE TABLE NVVIDEO_ENCODER_API (
-- NV Video Encoder API traced to gather information about NVIDIA Video Codek SDK Encoder APIs.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalThreadId.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
apiId INTEGER -- REFERENCES GPU_VIDEO_ENGINE_WORKLOAD(apiId)
);
CREATE TABLE NVVIDEO_DECODER_API (
-- NV Video Encoder API traced to gather information about NVIDIA Video Codek SDK Decoder APIs.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalThreadId.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
apiId INTEGER -- REFERENCES GPU_VIDEO_ENGINE_WORKLOAD(apiId)
);
CREATE TABLE NVVIDEO_JPEG_API (
-- NV Video Encoder API traced to gather information about NVIDIA Video Codek SDK JPEG APIs.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalThreadId.
nameId INTEGER NOT NULL -- REFERENCES StringIds(id) -- Function name
);
CREATE TABLE GPU_VIDEO_ENGINE_WORKLOAD (
-- Video engine workload events
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalEngineId INTEGER NOT NULL, -- Serialized GlobalId.
engineType INTEGER NOT NULL, -- REFERENCES ENUM_VIDEO_ENGINE_TYPE(id)
engineId INTEGER NOT NULL,
vmId INTEGER NOT NULL, -- Driver provided ID.
contextId INTEGER, -- Context ID.
globalPid INTEGER, -- Serialized GlobalProcessId.
apiId INTEGER NOT NULL, -- ID used to correlate API and workload trace.
codecId INTEGER -- REFERENCES ENUM_VIDEO_ENGINE_CODEC(id)
);
CREATE TABLE GPU_VIDEO_ENGINE_MISSING (
-- Video engine missing ranges
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalEngineId INTEGER NOT NULL, -- Serialized GlobalId.
rangeCount INTEGER NOT NULL -- Number of missing ranges.
);
CREATE TABLE MEMORY_TRANSFER_EVENTS (
-- Raw ETW Memory Transfer events.
start INTEGER NOT NULL, -- Event start timestamp (ns).
globalTid INTEGER, -- Serialized GlobalId.
gpu INTEGER, -- GPU index.
taskId INTEGER NOT NULL, -- The event task ID.
eventId INTEGER NOT NULL, -- Event ID.
allocationGlobalHandle INTEGER NOT NULL, -- Address of the global allocation handle.
dmaBuffer INTEGER NOT NULL, -- The reported address of dma buffer.
size INTEGER NOT NULL, -- The size of the dma buffer in bytes.
offset INTEGER NOT NULL, -- The offset from the start of the reported dma buffer in bytes.
memoryTransferType INTEGER NOT NULL -- REFERENCES ENUM_ETW_MEMORY_TRANSFER_TYPE(id)
);
CREATE TABLE NV_LOAD_BALANCE_MASTER_EVENTS (
-- Raw ETW NV-wgf2um LoadBalanceMaster events.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalTid INTEGER NOT NULL, -- Serialized GlobalId.
gpu INTEGER NOT NULL, -- GPU index.
eventId INTEGER NOT NULL, -- Event ID.
task TEXT NOT NULL, -- The task name.
frameCount INTEGER NOT NULL, -- The frame ID.
frameTime REAL NOT NULL, -- Frame duration.
averageFrameTime REAL NOT NULL, -- Average of frame duration.
averageLatency REAL NOT NULL, -- Average of latency.
minLatency REAL NOT NULL, -- The minimum latency.
averageQueuedFrames REAL NOT NULL, -- Average number of queued frames.
totalActiveMs REAL NOT NULL, -- Total active time in milliseconds.
totalIdleMs REAL NOT NULL, -- Total idle time in milliseconds.
idlePercent REAL NOT NULL, -- The percentage of idle time.
isGPUAlmostOneFrameAhead INTEGER NOT NULL -- True or False if GPU is almost one frame ahead.
);
CREATE TABLE NV_LOAD_BALANCE_EVENTS (
-- Raw ETW NV-wgf2um LoadBalance events.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalTid INTEGER NOT NULL, -- Serialized GlobalId.
gpu INTEGER NOT NULL, -- GPU index.
eventId INTEGER NOT NULL, -- Event ID.
task TEXT NOT NULL, -- The task name.
averageFPS REAL NOT NULL, -- Average frame per second.
queuedFrames REAL NOT NULL, -- The amount of queued frames.
averageQueuedFrames REAL NOT NULL, -- Average number of queued frames.
currentCPUTime REAL NOT NULL, -- The current CPU time.
averageCPUTime REAL NOT NULL, -- Average CPU time.
averageStallTime REAL NOT NULL, -- Average of stall time.
averageCPUIdleTime REAL NOT NULL, -- Average CPU idle time.
isGPUAlmostOneFrameAhead INTEGER NOT NULL -- True or False if GPU is almost one frame ahead.
);
CREATE TABLE PROCESSES (
-- Names and identifiers of processes captured in the report.
globalPid INTEGER, -- Serialized GlobalId.
pid INTEGER, -- The process ID.
name TEXT -- The process name.
);
CREATE TABLE CUPTI_ACTIVITY_KIND_OPENACC_DATA (
-- OpenACC data events collected using CUPTI.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Event name
globalTid INTEGER, -- Serialized GlobalId.
eventKind INTEGER NOT NULL, -- REFERENCES ENUM_OPENACC_EVENT_KIND(id)
DeviceType INTEGER NOT NULL, -- REFERENCES ENUM_OPENACC_DEVICE(id)
lineNo INTEGER NOT NULL, -- Line number of the directive or program construct.
cuDeviceId INTEGER NOT NULL, -- CUDA device ID. Valid only if deviceType is acc_device_nvidia.
cuContextId INTEGER NOT NULL, -- CUDA context ID. Valid only if deviceType is acc_device_nvidia.
cuStreamId INTEGER NOT NULL, -- CUDA stream ID. Valid only if deviceType is acc_device_nvidia.
srcFile INTEGER, -- REFERENCES StringIds(id) -- Source file name or path
funcName INTEGER, -- REFERENCES StringIds(id) -- Function in which event occurred
correlationId INTEGER, -- REFERENCES CUPTI_ACTIVITY_KIND_RUNTIME(correlationId)
bytes INTEGER, -- Number of bytes.
varName INTEGER -- REFERENCES StringIds(id) -- Variable name
);
CREATE TABLE CUPTI_ACTIVITY_KIND_OPENACC_LAUNCH (
-- OpenACC launch events collected using CUPTI.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Event name
globalTid INTEGER, -- Serialized GlobalId.
eventKind INTEGER NOT NULL, -- REFERENCES ENUM_OPENACC_EVENT_KIND(id)
DeviceType INTEGER NOT NULL, -- REFERENCES ENUM_OPENACC_DEVICE(id)
lineNo INTEGER NOT NULL, -- Line number of the directive or program construct.
cuDeviceId INTEGER NOT NULL, -- CUDA device ID. Valid only if deviceType is acc_device_nvidia.
cuContextId INTEGER NOT NULL, -- CUDA context ID. Valid only if deviceType is acc_device_nvidia.
cuStreamId INTEGER NOT NULL, -- CUDA stream ID. Valid only if deviceType is acc_device_nvidia.
srcFile INTEGER, -- REFERENCES StringIds(id) -- Source file name or path
funcName INTEGER, -- REFERENCES StringIds(id) -- Function in which event occurred
correlationId INTEGER, -- REFERENCES CUPTI_ACTIVITY_KIND_RUNTIME(correlationId)
numGangs INTEGER, -- Number of gangs created for this kernel launch.
numWorkers INTEGER, -- Number of workers created for this kernel launch.
vectorLength INTEGER, -- Number of vector lanes created for this kernel launch.
kernelName INTEGER -- REFERENCES StringIds(id) -- Kernel name
);
CREATE TABLE CUPTI_ACTIVITY_KIND_OPENACC_OTHER (
-- OpenACC other events collected using CUPTI.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Event name
globalTid INTEGER, -- Serialized GlobalId.
eventKind INTEGER NOT NULL, -- REFERENCES ENUM_OPENACC_EVENT_KIND(id)
DeviceType INTEGER NOT NULL, -- REFERENCES ENUM_OPENACC_DEVICE(id)
lineNo INTEGER NOT NULL, -- Line number of the directive or program construct.
cuDeviceId INTEGER NOT NULL, -- CUDA device ID. Valid only if deviceType is acc_device_nvidia.
cuContextId INTEGER NOT NULL, -- CUDA context ID. Valid only if deviceType is acc_device_nvidia.
cuStreamId INTEGER NOT NULL, -- CUDA stream ID. Valid only if deviceType is acc_device_nvidia.
srcFile INTEGER, -- REFERENCES StringIds(id) -- Source file name or path
funcName INTEGER, -- REFERENCES StringIds(id) -- Function in which event occurred
correlationId INTEGER -- REFERENCES CUPTI_ACTIVITY_KIND_RUNTIME(correlationId)
);
CREATE TABLE NET_NIC_METRIC (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalId INTEGER NOT NULL, -- Serialized GlobalId.
portId INTEGER NOT NULL, -- REFERENCES NET_DEVICE_PORT_INFO(portNumber) -- Port ID
metricsListId INTEGER NOT NULL, -- REFERENCES TARGET_INFO_NETWORK_METRICS(metricsListId)
metricsIdx INTEGER NOT NULL, -- REFERENCES TARGET_INFO_NETWORK_METRICS(metricsIdx)
value INTEGER NOT NULL -- Counter data value
);
CREATE TABLE NET_IB_SWITCH_METRIC (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalId INTEGER NOT NULL, -- Serialized GlobalId.
portId INTEGER NOT NULL, -- REFERENCES NET_DEVICE_PORT_INFO(portNumber) -- Port ID
metricsListId INTEGER NOT NULL, -- REFERENCES TARGET_INFO_NETWORK_METRICS(metricsListId)
metricsIdx INTEGER NOT NULL, -- REFERENCES TARGET_INFO_NETWORK_METRICS(metricsIdx)
value INTEGER NOT NULL -- Counter data value
);
CREATE TABLE NET_ETH_SWITCH_METRIC (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalId INTEGER NOT NULL, -- Serialized GlobalId.
portId INTEGER NOT NULL, -- REFERENCES NET_DEVICE_PORT_INFO(portNumber) -- Port ID
metricsListId INTEGER NOT NULL, -- REFERENCES TARGET_INFO_NETWORK_METRICS(metricsListId)
metricsIdx INTEGER NOT NULL, -- REFERENCES TARGET_INFO_NETWORK_METRICS(metricsIdx)
value INTEGER NOT NULL -- Counter data value
);
CREATE TABLE NET_IB_SWITCH_CONGESTION_EVENT (
start INTEGER NOT NULL, -- Event start timestamp (ns).
globalId INTEGER NOT NULL, -- Serialized GlobalId (view in hex).
congestionType INTEGER, -- REFERENCES ENUM_NET_IB_CONGESTION_EVENT_TYPE(id)
packetSLID INTEGER, -- Packet Source LID
packetDLID INTEGER, -- Packet Destination LID
packetSL INTEGER, -- Packet Service Level
packetOpCode INTEGER, -- Packet Operation Code
packetSourceQP INTEGER, -- Packet Source Queue Pair
packetDestinationQP INTEGER, -- Packet Destination Queue Pair
switchIngressPort INTEGER, -- Packet's Ingress Switch Port
switchEgressPort INTEGER -- Packet's Egress Switch Port
);
CREATE TABLE PMU_EVENTS (
-- CPU Core events.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalVm INTEGER NOT NULL, -- Serialized GlobalId.
cpu INTEGER NOT NULL, -- CPU ID
counter_id INTEGER -- REFERENCES PMU_EVENT_COUNTERS(id)
);
CREATE TABLE PMU_EVENT_COUNTERS (
-- CPU Core events counters.
id INTEGER NOT NULL,
idx INTEGER NOT NULL, -- REFERENCES PMU_EVENT_REQUESTS(id).
value INTEGER NOT NULL -- Counter data value
);
CREATE TABLE TEGRA_INTERNAL_API_CALLS (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
nameId INTEGER NOT NULL -- REFERENCES StringIds(id) -- Function name
);
CREATE TABLE TRACE_PROCESS_EVENT_NVMEDIA (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
correlationId INTEGER -- First ID matching an API call to GPU workloads.
);
CREATE TABLE UNCORE_PMU_EVENTS (
-- PMU Uncore events.
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalVm INTEGER NOT NULL, -- Serialized GlobalId.
clusterId INTEGER, -- Cluster ID.
counterId INTEGER -- REFERENCES UNCORE_PMU_EVENT_VALUES(id).
);
CREATE TABLE UNCORE_PMU_EVENT_VALUES (
-- Uncore events values.
id INTEGER NOT NULL,
type INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_TYPE(id)
value INTEGER NOT NULL, -- Event value.
rawId INTEGER NOT NULL, -- Event value raw ID.
clusterId INTEGER -- Cluster ID.
);
CREATE TABLE DIAGNOSTIC_EVENT (
timestamp INTEGER NOT NULL, -- Event timestamp (ns).
timestampType INTEGER NOT NULL, -- REFERENCES ENUM_DIAGNOSTIC_TIMESTAMP_SOURCE(id)
source INTEGER NOT NULL, -- REFERENCES ENUM_DIAGNOSTIC_SOURCE_TYPE(id)
severity INTEGER NOT NULL, -- REFERENCES ENUM_DIAGNOSTIC_SEVERITY_LEVEL(id)
text TEXT NOT NULL, -- Diagnostic message text
globalPid INTEGER -- Serialized GlobalId.
);
CREATE TABLE SYSCALL (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
eventClass INTEGER NOT NULL, -- REFERENCES ENUM_NSYS_EVENT_CLASS(id)
globalTid INTEGER, -- Serialized GlobalId.
nameId INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
callchainId INTEGER NOT NULL -- REFERENCES SYSCALL_CALLCHAINS(id)
);
CREATE TABLE SYSCALL_CALLCHAINS (
-- Callchains attached to syscall events, depending on selected profiling settings.
id INTEGER NOT NULL, -- Part of PRIMARY KEY (id, stackDepth).
stackDepth INTEGER NOT NULL, -- Zero-base index of the given function in call stack.
symbol INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Function name
module INTEGER NOT NULL, -- REFERENCES StringIds(id) -- Module name
unresolved INTEGER, -- True if the symbol was not resolved.
PRIMARY KEY (id, stackDepth)
);
CREATE TABLE BANDWIDTH_USAGE_EVENTS (
start INTEGER NOT NULL, -- Event start timestamp (ns).
end INTEGER NOT NULL, -- Event end timestamp (ns).
globalVm INTEGER NOT NULL, -- Serialized GlobalId.
valuesId INTEGER -- REFERENCES BANDWIDTH_USAGE_VALUES(id)
);
CREATE TABLE BANDWIDTH_USAGE_VALUES (
-- BandwidthUsage event XmcClient values.
id INTEGER NOT NULL,
idx INTEGER NOT NULL,
value INTEGER NOT NULL, -- Counter data value
PRIMARY KEY (id, idx)
);
Note
GENERIC_EVENTS.typeId is a composite bit field that combines HW ID, VM ID, source ID, and type ID with the following structure:
<Hardware ID:8><VM ID:8><Source ID:16><Type ID:32>
The type ID is yet another composite bit field that combines the GPU metrics event tag and the GPU ID. To extract the latter, you need to get the lower 8 bits:
SELECT typeId & 0xFF AS gpuId FROM GENERIC_EVENTS
Note
The NET_DEVICE_* tables hold every network device, whatever its kind. A device’s kind is given by NET_DEVICE_TYPE_MAP.deviceType, which references ENUM_NET_DEVICE_TYPE and carries one row per set bit, so a device may appear with more than one type. Despite the similar name, ENUM_NET_DEVICE_TYPE is unrelated to ENUM_NET_DEVICE_ID, which labels NIC PCIe device IDs.
CollectX does not report Ethernet switches GUIDs. Therefore, we use the switch name’s hash as the switch identifier in NET_DEVICE_PORT_INFO and NET_DEVICE_TYPE_MAP. The hash is stable, allowing running joins between tables. The switch name is held in NET_DEVICE_INFO.name.
Some event types have been deprecated and are no longer supported by Nsight Systems. While tables for these event will no longer appear in exported SQL databases, databases exported by older versions of Nsight Systems may still contain them.
CREATE TABLE ETW_EVENTS_DEPRECATED_TABLE (
[...]
);
CREATE TABLE GPU_MEMORY_BUDGET_EVENTS (
-- Raw ETW VidMmProcessBudgetChange events (deprecated).
[...]
);
CREATE TABLE GPU_MEMORY_USAGE_EVENTS (
-- Raw ETW VidMmProcessUsageChange events (deprecated).
[...]
);
CREATE TABLE DEMOTED_BYTES_EVENTS (
-- Raw ETW VidMmProcessDemotedCommitmentChange events (deprecated).
[...]
);
CREATE TABLE TOTAL_BYTES_RESIDENT_IN_SEGMENT_EVENTS (
-- Raw ETW TotalBytesResidentInSegment events (deprecated).
[...]
);
SQLite Schema Event Values#
Here are the set values stored in enums in the Nsight Systems SQLite schema
CUDA Memcopy Kind
0 - CUDA_MEMCPY_KIND_UNKNOWN
1 - CUDA_MEMCPY_KIND_HTOD
2 - CUDA_MEMCPY_KIND_DTOH
3 - CUDA_MEMCPY_KIND_HTOA
4 - CUDA_MEMCPY_KIND_ATOH
5 - CUDA_MEMCPY_KIND_ATOA
6 - CUDA_MEMCPY_KIND_ATOD
7 - CUDA_MEMCPY_KIND_DTOA
8 - CUDA_MEMCPY_KIND_DTOD
9 - CUDA_MEMCPY_KIND_HTOH
10 - CUDA_MEMCPY_KIND_PTOP
11 - CUDA_MEMCPY_KIND_UVM_HTOD
12 - CUDA_MEMCPY_KIND_UVM_DTOH
13 - CUDA_MEMCPY_KIND_UVM_DTOD
CUDA Memory Operations Memory Kind
0 - CUDA_MEMOPR_MEMORY_KIND_PAGEABLE
1 - CUDA_MEMOPR_MEMORY_KIND_PINNED
2 - CUDA_MEMOPR_MEMORY_KIND_DEVICE
3 - CUDA_MEMOPR_MEMORY_KIND_ARRAY
4 - CUDA_MEMOPR_MEMORY_KIND_MANAGED
5 - CUDA_MEMOPR_MEMORY_KIND_DEVICE_STATIC
6 - CUDA_MEMOPR_MEMORY_KIND_MANAGED_STATIC
7 - CUDA_MEMOPR_MEMORY_KIND_UNKNOWN
CUDA Event Class Values
0 - TRACE_PROCESS_EVENT_CUDA_RUNTIME
1 - TRACE_PROCESS_EVENT_CUDA_DRIVER
13 - TRACE_PROCESS_EVENT_CUDA_EGL_DRIVER
28 - TRACE_PROCESS_EVENT_CUDNN
29 - TRACE_PROCESS_EVENT_CUBLAS
33 - TRACE_PROCESS_EVENT_CUDNN_START
34 - TRACE_PROCESS_EVENT_CUDNN_FINISH
35 - TRACE_PROCESS_EVENT_CUBLAS_START
36 - TRACE_PROCESS_EVENT_CUBLAS_FINISH
67 - TRACE_PROCESS_EVENT_CUDABACKTRACE
77 - TRACE_PROCESS_EVENT_CUDA_GRAPH_NODE_CREATION
See CUPTI documentation for detailed information on collected event and data types.
NVTX Event Type Values
33 - NvtxCategory
34 - NvtxMark
39 - NvtxThread
59 - NvtxPushPopRange
60 - NvtxStartEndRange
75 - NvtxDomainCreate
76 - NvtxDomainDestroy
The difference between text and textId columns is that if an NVTX event message was passed via call to nvtxDomainRegisterString function, then the message will be available through textId field, otherwise the text field will contain the message if it was provided.
OpenGL Events
KHR event class values
62 - KhrDebugPushPopRange
63 - KhrDebugGpuPushPopRange
KHR source kind values
0x8249 - GL_DEBUG_SOURCE_THIRD_PARTY
0x824A - GL_DEBUG_SOURCE_APPLICATION
KHR type values
0x824C - GL_DEBUG_TYPE_ERROR
0x824D - GL_DEBUG_TYPE_DEPRECATED_BEHAVIOR
0x824E - GL_DEBUG_TYPE_UNDEFINED_BEHAVIOR
0x824F - GL_DEBUG_TYPE_PORTABILITY
0x8250 - GL_DEBUG_TYPE_PERFORMANCE
0x8251 - GL_DEBUG_TYPE_OTHER
0x8268 - GL_DEBUG_TYPE_MARKER
0x8269 - GL_DEBUG_TYPE_PUSH_GROUP
0x826A - GL_DEBUG_TYPE_POP_GROUP
KHR severity values
0x826B - GL_DEBUG_SEVERITY_NOTIFICATION
0x9146 - GL_DEBUG_SEVERITY_HIGH
0x9147 - GL_DEBUG_SEVERITY_MEDIUM
0x9148 - GL_DEBUG_SEVERITY_LOW
OSRT Event Class Values
OS runtime libraries can be traced to gather information about low-level userspace APIs. This traces the system call wrappers and thread synchronization interfaces exposed by the C runtime and POSIX Threads (pthread) libraries. This does not perform a complete runtime library API trace, but instead focuses on the functions that can take a long time to execute, or could potentially cause your thread be unscheduled from the CPU while waiting for an event to complete.
OSRT events may have callchains attached to them, depending on selected profiling settings. In such cases, one can use callchainId column to select relevant callchains from OSRT_CALLCHAINS table
OSRT event class values
27 - TRACE_PROCESS_EVENT_OS_RUNTIME
31 - TRACE_PROCESS_EVENT_OS_RUNTIME_START
32 - TRACE_PROCESS_EVENT_OS_RUNTIME_FINISH
DX12 Event Class Values
41 - TRACE_PROCESS_EVENT_DX12_API
42 - TRACE_PROCESS_EVENT_DX12_WORKLOAD
43 - TRACE_PROCESS_EVENT_DX12_START
44 - TRACE_PROCESS_EVENT_DX12_FINISH
52 - TRACE_PROCESS_EVENT_DX12_DISPLAY
59 - TRACE_PROCESS_EVENT_DX12_CREATE_OBJECT
PIX Event Class Values
65 - TRACE_PROCESS_EVENT_DX12_DEBUG_API
75 - TRACE_PROCESS_EVENT_DX11_DEBUG_API
Vulkan Event Class Values
53 - TRACE_PROCESS_EVENT_VULKAN_API
54 - TRACE_PROCESS_EVENT_VULKAN_WORKLOAD
55 - TRACE_PROCESS_EVENT_VULKAN_START
56 - TRACE_PROCESS_EVENT_VULKAN_FINISH
60 - TRACE_PROCESS_EVENT_VULKAN_CREATE_OBJECT
66 - TRACE_PROCESS_EVENT_VULKAN_DEBUG_API
Vulkan Flags
VALID_BIT = 0x00000001
CACHE_HIT_BIT = 0x00000002
BASE_PIPELINE_ACCELERATION_BIT = 0x00000004
WDDM Event Values
VIDMM operation type values
0 - None
101 - RestoreSegments
102 - PurgeSegments
103 - CleanupPrimary
104 - AllocatePagingBufferResources
105 - FreePagingBufferResources
106 - ReportVidMmState
107 - RunApertureCoherencyTest
108 - RunUnmapToDummyPageTest
109 - DeferredCommand
110 - SuspendMemorySegmentAccess
111 - ResumeMemorySegmentAccess
112 - EvictAndFlush
113 - CommitVirtualAddressRange
114 - UncommitVirtualAddressRange
115 - DestroyVirtualAddressAllocator
116 - PageInDevice
117 - MapContextAllocation
118 - InitPagingProcessVaSpace
200 - CloseAllocation
202 - ComplexLock
203 - PinAllocation
204 - FlushPendingGpuAccess
205 - UnpinAllocation
206 - MakeResident
207 - Evict
208 - LockInAperture
209 - InitContextAllocation
210 - ReclaimAllocation
211 - DiscardAllocation
212 - SetAllocationPriority
1000 - EvictSystemMemoryOfferList
Paging queue type values
0 - VIDMM_PAGING_QUEUE_TYPE_UMD
1 - VIDMM_PAGING_QUEUE_TYPE_Default
2 - VIDMM_PAGING_QUEUE_TYPE_Evict
3 - VIDMM_PAGING_QUEUE_TYPE_Reclaim
Packet type values
0 - DXGKETW_RENDER_COMMAND_BUFFER
1 - DXGKETW_DEFERRED_COMMAND_BUFFER
2 - DXGKETW_SYSTEM_COMMAND_BUFFER
3 - DXGKETW_MMIOFLIP_COMMAND_BUFFER
4 - DXGKETW_WAIT_COMMAND_BUFFER
5 - DXGKETW_SIGNAL_COMMAND_BUFFER
6 - DXGKETW_DEVICE_COMMAND_BUFFER
7 - DXGKETW_SOFTWARE_COMMAND_BUFFER
Engine type values
0 - DXGK_ENGINE_TYPE_OTHER
1 - DXGK_ENGINE_TYPE_3D
2 - DXGK_ENGINE_TYPE_VIDEO_DECODE
3 - DXGK_ENGINE_TYPE_VIDEO_ENCODE
4 - DXGK_ENGINE_TYPE_VIDEO_PROCESSING
5 - DXGK_ENGINE_TYPE_SCENE_ASSEMBLY
6 - DXGK_ENGINE_TYPE_COPY
7 - DXGK_ENGINE_TYPE_OVERLAY
8 - DXGK_ENGINE_TYPE_CRYPTO
DMA interrupt type values
1 = DXGK_INTERRUPT_DMA_COMPLETED
2 = DXGK_INTERRUPT_DMA_PREEMPTED
4 = DXGK_INTERRUPT_DMA_FAULTED
9 = DXGK_INTERRUPT_DMA_PAGE_FAULTED
Queue type values
0 = Queue_Packet
1 = Dma_Packet
2 = Paging_Queue_Packet
Driver Events
Load balance event type values
1 - LoadBalanceEvent_GPU
8 - LoadBalanceEvent_CPU
21 - LoadBalanceMasterEvent_GPU
22 - LoadBalanceMasterEvent_CPU
OpenMP Events
OpenMP event class values
78 - TRACE_PROCESS_EVENT_OPENMP
79 - TRACE_PROCESS_EVENT_OPENMP_START
80 - TRACE_PROCESS_EVENT_OPENMP_FINISH
OpenMP event kind values
15 - OPENMP_EVENT_KIND_TASK_CREATE
16 - OPENMP_EVENT_KIND_TASK_SCHEDULE
17 - OPENMP_EVENT_KIND_CANCEL
20 - OPENMP_EVENT_KIND_MUTEX_RELEASED
21 - OPENMP_EVENT_KIND_LOCK_INIT
22 - OPENMP_EVENT_KIND_LOCK_DESTROY
25 - OPENMP_EVENT_KIND_DISPATCH
26 - OPENMP_EVENT_KIND_FLUSH
27 - OPENMP_EVENT_KIND_THREAD
28 - OPENMP_EVENT_KIND_PARALLEL
29 - OPENMP_EVENT_KIND_SYNC_REGION_WAIT
30 - OPENMP_EVENT_KIND_SYNC_REGION
31 - OPENMP_EVENT_KIND_TASK
32 - OPENMP_EVENT_KIND_MASTER
33 - OPENMP_EVENT_KIND_REDUCTION
34 - OPENMP_EVENT_KIND_MUTEX_WAIT
35 - OPENMP_EVENT_KIND_CRITICAL_SECTION
36 - OPENMP_EVENT_KIND_WORKSHARE
OpenMP thread type values
1 - OpenMP Initial Thread
2 - OpenMP Worker Thread
3 - OpenMP Internal Thread
4 - Unknown
OpenMP sync region kind values
1 - Barrier
2 - Implicit barrier
3 - Explicit barrier
4 - Implementation-dependent barrier
5 - Taskwait
6 - Taskgroup
OpenMP task kind values
1 - Initial task
2 - Implicit task
3 - Explicit task
OpenMP prior task status values
1 - Task completed
2 - Task yielded to another task
3 - Task was cancelled
7 - Task was switched out for other reasons
OpenMP mutex kind values
1 - Waiting for lock
2 - Testing lock
3 - Waiting for nested lock
4 - Tesing nested lock
5 - Waitng for entering critical section region
6 - Waiting for entering atomic region
7 - Waiting for entering ordered region
OpenMP critical section kind values
5 - Critical section region
6 - Atomic region
7 - Ordered region
OpenMP workshare kind values
1 - Loop region
2 - Sections region
3 - Single region (executor)
4 - Single region (waiting)
5 - Workshare region
6 - Distrubute region
7 - Taskloop region
OpenMP dispatch kind values
1 - Iteration
2 - Section
Common SQLite Examples#
Common Helper Commands
When utilizing the sqlite3 command line tool, it’s helpful to have data printed as named columns, this can be done with:
.mode column
.headers on
The default column width is determined by the data in the first row of results. If this doesn’t work out well, you can specify widths manually.
.width 10 20 50
Obtaining Sample Report
The CLI interface of Nsight Systems was used to profile the radixSortThrust CUDA sample, then the resulting .nsys-rep file was exported using the nsys export.
nsys profile --trace=cuda,osrt radixSortThrust
nsys export --type sqlite report1.nsys-rep
Serialized Process and Thread Identifiers
Note
The globalTid field is a 64-bit identifier that encodes multiple components into a single value: <Hardware ID:8><VM ID:8><Process ID:24><Thread ID:24> It follows the structure:
Thread ID: bits 0–23
Process ID: bits 24–47
VM ID: bits 48–55
Hardware ID: bits 56–63
Goal: Extract readable process ID (PID) and thread ID (TID) values from Nsight Systems’s serialized identifier format.
Nsight Systems stores identifiers where events originated in serialized form to efficiently pack multiple values into a single field. This example shows how to decode them back to standard PID/TID format.
What the code does: Uses bit shifting and modulo operations to extract the embedded PID and TID values from the globalTid field.
For events that have globalTid or globalPid fields exported, use the following code to extract numeric TID and PID.
SELECT globalTid / 0x1000000 % 0x1000000 AS PID, globalTid % 0x1000000 AS TID FROM TABLE_NAME;
# Python equivalent:
def extract_pid_tid(global_tid):
PID = (global_tid // 0x1000000) % 0x1000000
TID = global_tid % 0x1000000
return PID, TID
Note
globalTid field includes both TID and PID values, while globalPid only contains the PID value.
Read NVTX Binary Payload Fields
Goal: Read the fields of an NVTX binary payload alongside the event that carried it.
When exported with nsys export --dynamic-tables=all, each NVTX binary payload
schema becomes its own table named
DYNAMIC_NVTXPAYLOAD_DATATYPE_<globalPid>_<domainId>_<schemaId>, one row per
payload instance. The NVTX_EVENTS_PAYLOAD_SCHEMAS link table maps each event
to the globalPid, domainId, and payloadSchemaID that name its table.
What the code does: Lists the payload links, then joins one payload table
back to NVTX_EVENTS on eventId to read the payload fields together with
the event’s timing and text.
SELECT eventId, globalPid, domainId, payloadSchemaID FROM NVTX_EVENTS_PAYLOAD_SCHEMAS;
Substitute the ids from a row above into the table name.
SELECT e.start, e.end, e.text, d.*
FROM DYNAMIC_NVTXPAYLOAD_DATATYPE_<globalPid>_<domainId>_<payloadSchemaID> AS d
JOIN NVTX_EVENTS AS e USING (eventId);
Note
The same tables and join are available in the Arrow, Arrow directory, and
Parquet directory exports. SQLite has no unsigned or fixed-width integer
types, so payload columns typed uint8/uint16/uint32/uint64 or
32-bit float are widened to signed 64-bit integers or REAL in a SQLite
export, while those formats preserve the payload’s native column types.
Understanding Event Types and Tags
Many tables in the nsys SQLite export use numeric codes for event types, tags, and classes. These codes represent different categories of events:
Event Types: Identify the kind of operation or marker (e.g., NVTX marks vs ranges)
Tags: Specify event phases like BEGIN/END for state transitions
Event Classes: Categorize broad event categories (e.g., different types of GPU operations)
When you see numeric values in queries, refer to the documentation sections above or query the relevant string tables to understand their meanings.
Correlate CUDA Kernel Launches With CUDA API Kernel Launches
Goal: Link CUDA runtime API calls to the actual GPU kernels they launch, enabling analysis of which API calls resulted in the longest-running kernels.
What the code does: 1. Adds human-readable columns to the runtime table 2. Joins runtime API calls with GPU kernel executions using correlation IDs 3. Populates kernel names and API function names from the string table 4. Finds the 10 longest API calls that resulted in kernel execution
ALTER TABLE CUPTI_ACTIVITY_KIND_RUNTIME ADD COLUMN name TEXT;
ALTER TABLE CUPTI_ACTIVITY_KIND_RUNTIME ADD COLUMN kernelName TEXT;
UPDATE CUPTI_ACTIVITY_KIND_RUNTIME SET kernelName =
(SELECT value FROM StringIds
JOIN CUPTI_ACTIVITY_KIND_KERNEL AS cuda_gpu
ON cuda_gpu.shortName = StringIds.id
AND CUPTI_ACTIVITY_KIND_RUNTIME.correlationId = cuda_gpu.correlationId);
UPDATE CUPTI_ACTIVITY_KIND_RUNTIME SET name =
(SELECT value FROM StringIds WHERE nameId = StringIds.id);
Select the 10 longest CUDA API ranges that resulted in kernel execution.
SELECT name, kernelName, start, end FROM CUPTI_ACTIVITY_KIND_RUNTIME
WHERE kernelName IS NOT NULL ORDER BY end - start LIMIT 10;
# Python equivalent workflow:
# 1. Load runtime and kernel data
# 2. Join on correlation_id to match API calls with kernels
# 3. Add readable names from string table
# 4. Sort by duration and get top 10
runtime_with_kernels = runtime.merge(kernels, on='correlationId')
runtime_with_kernels['duration'] = runtime_with_kernels['end'] - runtime_with_kernels['start']
top_10_longest = runtime_with_kernels.nlargest(10, 'duration')
Results:
name kernelName start end
---------------------- ----------------------- ---------- ----------
cudaLaunchKernel_v7000 RadixSortScanBinsKernel 658863435 658868490
cudaLaunchKernel_v7000 RadixSortScanBinsKernel 609755015 609760075
cudaLaunchKernel_v7000 RadixSortScanBinsKernel 632683286 632688349
cudaLaunchKernel_v7000 RadixSortScanBinsKernel 606495356 606500439
cudaLaunchKernel_v7000 RadixSortScanBinsKernel 603114486 603119586
cudaLaunchKernel_v7000 RadixSortScanBinsKernel 802729785 802734906
cudaLaunchKernel_v7000 RadixSortScanBinsKernel 593381170 593386294
cudaLaunchKernel_v7000 RadixSortScanBinsKernel 658759955 658765090
cudaLaunchKernel_v7000 RadixSortScanBinsKernel 681549917 681555059
cudaLaunchKernel_v7000 RadixSortScanBinsKernel 717812527 717817671
Remove Ranges Overlapping With Overhead
Goal: Identify and remove CUDA API calls that overlap with profiler overhead to get cleaner performance measurements.
What the code does: Uses spatial overlap detection to find CUDA runtime ranges that intersect with profiler overhead periods. The query checks for three types of overlap: range starts within overhead, range ends within overhead, or range completely encompasses overhead.
Use the this query to count CUDA API ranges overlapping with the overhead ones.
Replace “SELECT COUNT(*)” with “DELETE” to remove such ranges.
SELECT COUNT(*) FROM CUPTI_ACTIVITY_KIND_RUNTIME WHERE rowid IN
(
SELECT cuda.rowid
FROM PROFILER_OVERHEAD as overhead
INNER JOIN CUPTI_ACTIVITY_KIND_RUNTIME as cuda ON
(cuda.start BETWEEN overhead.start and overhead.end)
OR (cuda.end BETWEEN overhead.start and overhead.end)
OR (cuda.start < overhead.start AND cuda.end > overhead.end)
);
# Python equivalent for finding overlaps:
def ranges_overlap(range1_start, range1_end, range2_start, range2_end):
return (range1_start <= range2_end and range1_end >= range2_start)
overlapping_ranges = []
for cuda_range in cuda_ranges:
for overhead_range in overhead_ranges:
if ranges_overlap(cuda_range.start, cuda_range.end,
overhead_range.start, overhead_range.end):
overlapping_ranges.append(cuda_range)
Results:
COUNT(*)
----------
1095
Find CUDA API Calls that Resulted in the Original Graph Node Creation
Goal: Identify which CUDA API calls were responsible for creating the original nodes in CUDA graphs (as opposed to cloned or instantiated nodes).
What the code does: 1. Filters graph nodes to find only original creations (those without originalGraphNodeId) 2. Groups by graphNodeId to get the first occurrence 3. Correlates with CUDA runtime API calls that were active when the graph node was created 4. Joins with string table to get readable API function names
SELECT graph.graphNodeId, api.start, graph.start as graphStart, api.end,
api.globalTid, api.correlationId, api.globalTid,
(SELECT value FROM StringIds where api.nameId == id) as name
FROM CUPTI_ACTIVITY_KIND_RUNTIME as api
JOIN
(
SELECT start, graphNodeId, globalTid from CUDA_GRAPH_NODE_EVENTS
GROUP BY graphNodeId
HAVING COUNT(originalGraphNodeId) = 0
) as graph
ON api.globalTid == graph.globalTid AND api.start < graph.start AND api.end > graph.start
ORDER BY graphNodeId;
Results:
graphNodeId start graphStart end globalTid correlationId globalTid name
----------- ---------- ---------- ---------- --------------- ------------- --------------- -----------------------------
1 584366518 584378040 584379102 281560221750233 109 281560221750233 cudaGraphAddMemcpyNode_v10000
2 584379402 584382428 584383139 281560221750233 110 281560221750233 cudaGraphAddMemsetNode_v10000
3 584390663 584395352 584396053 281560221750233 111 281560221750233 cudaGraphAddKernelNode_v10000
4 584396314 584397857 584398438 281560221750233 112 281560221750233 cudaGraphAddMemsetNode_v10000
5 584398759 584400311 584400812 281560221750233 113 281560221750233 cudaGraphAddKernelNode_v10000
6 584401083 584403047 584403527 281560221750233 114 281560221750233 cudaGraphAddMemcpyNode_v10000
7 584403928 584404920 584405491 281560221750233 115 281560221750233 cudaGraphAddHostNode_v10000
29 632107852 632117921 632121407 281560221750233 144 281560221750233 cudaMemcpyAsync_v3020
30 632122168 632125545 632127989 281560221750233 145 281560221750233 cudaMemsetAsync_v3020
31 632131546 632133339 632135584 281560221750233 147 281560221750233 cudaMemsetAsync_v3020
34 632162514 632167393 632169297 281560221750233 151 281560221750233 cudaMemcpyAsync_v3020
35 632170068 632173334 632175388 281560221750233 152 281560221750233 cudaLaunchHostFunc_v10000
Backtraces for OSRT Ranges
Goal: Analyze operating system runtime (OSRT) function calls with their full call stacks to understand performance bottlenecks and call patterns.
What the code does: 1. Adds human-readable columns for function names, symbol names, and module names 2. Populates these from the string table for better readability 3. Shows how to query the longest OSRT call with its complete backtrace
Adding text columns makes results of the query below more human-readable.
ALTER TABLE OSRT_API ADD COLUMN name TEXT;
UPDATE OSRT_API SET name = (SELECT value FROM StringIds WHERE OSRT_API.nameId = StringIds.id);
ALTER TABLE OSRT_CALLCHAINS ADD COLUMN symbolName TEXT;
UPDATE OSRT_CALLCHAINS SET symbolName = (SELECT value FROM StringIds WHERE symbol = StringIds.id);
ALTER TABLE OSRT_CALLCHAINS ADD COLUMN moduleName TEXT;
UPDATE OSRT_CALLCHAINS SET moduleName = (SELECT value FROM StringIds WHERE module = StringIds.id);
Print backtrace of the longest OSRT range.
SELECT globalTid / 0x1000000 % 0x1000000 AS PID, globalTid % 0x1000000 AS TID,
start, end, name, callchainId, stackDepth, symbolName, moduleName
FROM OSRT_API LEFT JOIN OSRT_CALLCHAINS ON callchainId == OSRT_CALLCHAINS.id
WHERE OSRT_API.rowid IN (SELECT rowid FROM OSRT_API ORDER BY end - start DESC LIMIT 1)
ORDER BY stackDepth LIMIT 10;
# Python equivalent for finding longest call with backtrace:
longest_call = osrt_api.loc[osrt_api['duration'].idxmax()]
backtrace = osrt_callchains[osrt_callchains['id'] == longest_call['callchainId']]
backtrace_ordered = backtrace.sort_values('stackDepth')
Results:
PID TID start end name callchainId stackDepth symbolName moduleName
---------- ---------- ---------- ---------- ---------------------- ----------- ---------- ------------------------------ ----------------------------------------
19163 19176 360897690 860966851 pthread_cond_timedwait 88 0 pthread_cond_timedwait@GLIBC_2 /lib/x86_64-linux-gnu/libpthread-2.27.so
19163 19176 360897690 860966851 pthread_cond_timedwait 88 1 0x7fbc983b7227 /usr/lib/x86_64-linux-gnu/libcuda.so.418
19163 19176 360897690 860966851 pthread_cond_timedwait 88 2 0x7fbc9835d5c7 /usr/lib/x86_64-linux-gnu/libcuda.so.418
19163 19176 360897690 860966851 pthread_cond_timedwait 88 3 0x7fbc983b64a8 /usr/lib/x86_64-linux-gnu/libcuda.so.418
19163 19176 360897690 860966851 pthread_cond_timedwait 88 4 start_thread /lib/x86_64-linux-gnu/libpthread-2.27.so
19163 19176 360897690 860966851 pthread_cond_timedwait 88 5 __clone /lib/x86_64-linux-gnu/libc-2.27.so
Profiled processes output streams.
Goal: Access stdout and stderr output from profiled processes to correlate application output with performance data.
What the code does: Resolves file paths and content from string IDs to show the captured stdout/stderr streams from profiled applications.
ALTER TABLE ProcessStreams ADD COLUMN filename TEXT;
UPDATE ProcessStreams SET filename = (SELECT value FROM StringIds WHERE ProcessStreams.filenameId = StringIds.id);
ALTER TABLE ProcessStreams ADD COLUMN content TEXT;
UPDATE ProcessStreams SET content = (SELECT value FROM StringIds WHERE ProcessStreams.contentId = StringIds.id);
Select all collected stdout and stderr streams.
select globalPid / 0x1000000 % 0x1000000 AS PID, filename, content from ProcessStreams;
Results:
PID filename content
---------- ------------------------------------------------------- --------------------------------------------------------------------------------------------------------------------
19163 /tmp/nvidia/nsight_systems/streams/pid_19163_stdout.log /home/user_name/NVIDIA_CUDA-10.1_Samples/6_Advanced/radixSortThrust/radixSortThrust Starting...
GPU Device 0: "Quadro P2000" with compute capability 6.1
Sorting 1048576 32-bit unsigned int keys and values
radixSortThrust, Throughput = 401.0872 MElements/s, Time = 0.00261 s, Size = 1048576 elements
Test passed
19163 /tmp/nvidia/nsight_systems/streams/pid_19163_stderr.log
Thread Summary
Goal: Calculate CPU utilization statistics per thread to identify which threads are consuming the most CPU resources.
Note that Nsight Systems applies additional logic during sampling events processing to work around lost events. This means that the results of the below query might differ slightly from the ones shown in “Analysis Summary” tab.
Approach 1: Using CPU Cycles (when available)
What this code does: Calculates thread CPU utilization using hardware performance counter data (CPU cycles) which provides the most accurate measurement of actual CPU usage per thread.
SELECT
globalTid / 0x1000000 % 0x1000000 AS PID,
globalTid % 0x1000000 AS TID,
ROUND(100.0 * SUM(cpuCycles) /
(
SELECT SUM(cpuCycles) FROM COMPOSITE_EVENTS
GROUP BY globalTid / 0x1000000000000 % 0x100
),
2
) as CPU_utilization,
(SELECT value FROM StringIds WHERE id =
(
SELECT nameId FROM ThreadNames
WHERE ThreadNames.globalTid = COMPOSITE_EVENTS.globalTid
)
) as thread_name
FROM COMPOSITE_EVENTS
GROUP BY globalTid
ORDER BY CPU_utilization DESC
LIMIT 10;
Results:
PID TID CPU_utilization thread_name
---------- ---------- --------------- ---------------
19163 19163 98.4 radixSortThrust
19163 19168 1.35 CUPTI worker th
19163 19166 0.25 [NS]
Approach 2: Using Scheduling Events (when PMU data not available)
What this approach does: When CPU cycle counter data is not collected, this method calculates thread CPU time based on scheduling events (when threads are scheduled in/out), then calculates utilization percentages. This approach is less precise but still useful for understanding relative thread activity.
CREATE INDEX sched_start ON SCHED_EVENTS (start);
CREATE TABLE CPU_USAGE AS
SELECT
first.globalTid as globalTid,
(SELECT nameId FROM ThreadNames WHERE ThreadNames.globalTid = first.globalTid) as nameId,
sum(second.start - first.start) as total_duration,
count() as ranges_count
FROM SCHED_EVENTS as first
LEFT JOIN SCHED_EVENTS as second
ON second.rowid =
(
SELECT rowid
FROM SCHED_EVENTS
WHERE start > first.start AND globalTid = first.globalTid
ORDER BY start ASC
LIMIT 1
)
WHERE first.isSchedIn != 0
GROUP BY first.globalTid
ORDER BY total_duration DESC;
SELECT
globalTid / 0x1000000 % 0x1000000 AS PID,
globalTid % 0x1000000 AS TID,
(SELECT value FROM StringIds where nameId == id) as thread_name,
ROUND(100.0 * total_duration / (SELECT SUM(total_duration) FROM CPU_USAGE), 2) as CPU_utilization
FROM CPU_USAGE
ORDER BY CPU_utilization DESC;
# Python equivalent for scheduling-based calculation:
def calculate_thread_cpu_time(sched_events):
cpu_usage = {}
for tid in unique_tids:
tid_events = sched_events[sched_events['globalTid'] == tid]
tid_events = tid_events[tid_events['isSchedIn'] == 1] # Only sched-in events
total_time = 0
for i in range(len(tid_events) - 1):
time_slice = tid_events.iloc[i+1]['start'] - tid_events.iloc[i]['start']
total_time += time_slice
cpu_usage[tid] = total_time
return cpu_usage
Results:
PID TID thread_name CPU_utilization
---------- ---------- --------------- ---------------
19163 19163 radixSortThrust 93.74
19163 19169 radixSortThrust 3.22
19163 19168 CUPTI worker th 2.46
19163 19166 [NS] 0.44
19163 19172 radixSortThrust 0.07
19163 19167 [NS Comms] 0.05
19163 19176 radixSortThrust 0.02
19163 19170 radixSortThrust 0.0
Function Table
Goal: Create profiler-style function tables showing flat view (total time in each function across all call stacks) and bottom-up view (time spent directly in each function).
What the code does: Processes sampling callchain data to calculate time spent in functions, providing two views commonly used in profilers.
These examples demonstrate how to calculate Flat and BottomUp (for top level only) views statistics.
To set up:
ALTER TABLE SAMPLING_CALLCHAINS ADD COLUMN symbolName TEXT;
UPDATE SAMPLING_CALLCHAINS SET symbolName = (SELECT value FROM StringIds WHERE symbol = StringIds.id);
ALTER TABLE SAMPLING_CALLCHAINS ADD COLUMN moduleName TEXT;
UPDATE SAMPLING_CALLCHAINS SET moduleName = (SELECT value FROM StringIds WHERE module = StringIds.id);
To get flat view:
Flat view: Shows total time spent in each function across all call stacks (inclusive time).
SELECT symbolName, moduleName, ROUND(100.0 * sum(cpuCycles) /
(SELECT SUM(cpuCycles) FROM COMPOSITE_EVENTS), 2) AS flatTimePercentage
FROM SAMPLING_CALLCHAINS
LEFT JOIN COMPOSITE_EVENTS ON SAMPLING_CALLCHAINS.id == COMPOSITE_EVENTS.id
GROUP BY symbol, module
ORDER BY flatTimePercentage DESC
LIMIT 5;
To get BottomUp view (top level only):
Bottom-up view: Shows time spent directly in each function (exclusive time, only leaf functions in call stacks).
SELECT symbolName, moduleName, ROUND(100.0 * sum(cpuCycles) /
(SELECT SUM(cpuCycles) FROM COMPOSITE_EVENTS), 2) AS selfTimePercentage
FROM SAMPLING_CALLCHAINS
LEFT JOIN COMPOSITE_EVENTS ON SAMPLING_CALLCHAINS.id == COMPOSITE_EVENTS.id
WHERE stackDepth == 0
GROUP BY symbol, module
ORDER BY selfTimePercentage DESC
LIMIT 5;
# Python equivalent:
# Flat view - aggregate all occurrences of each function
flat_view = callchains.groupby(['symbol', 'module'])['cpuCycles'].sum()
flat_percentages = (flat_view / total_cycles * 100).sort_values(ascending=False)
# Bottom-up view - only leaf nodes (stackDepth == 0)
leaf_functions = callchains[callchains['stackDepth'] == 0]
bottomup_view = leaf_functions.groupby(['symbol', 'module'])['cpuCycles'].sum()
Results:
symbolName moduleName flatTimePercentage
----------- ----------- ------------------
[Max depth] [Max depth] 99.92
thrust::zip /home/user_ 24.17
thrust::zip /home/user_ 24.17
thrust::det /home/user_ 24.17
thrust::det /home/user_ 24.17
symbolName moduleName selfTimePercentage
-------------- ------------------------------------------- ------------------
0x7fbc984982b6 /usr/lib/x86_64-linux-gnu/libcuda.so.418.39 5.29
0x7fbc982d0010 /usr/lib/x86_64-linux-gnu/libcuda.so.418.39 2.81
thrust::iterat /home/user_name/NVIDIA_CUDA-10.1_Samples/6_ 2.23
thrust::iterat /home/user_name/NVIDIA_CUDA-10.1_Samples/6_ 1.55
void thrust::i /home/user_name/NVIDIA_CUDA-10.1_Samples/6_ 1.55
DX12 API Frame Duration Histogram
Goal: Analyze DirectX 12 application frame timing by measuring the duration between consecutive Present calls and creating a timing histogram.
What the code does: 1. Creates a view that pairs consecutive Present calls to calculate frame durations 2. Groups frame durations into millisecond buckets 3. Counts how many frames fall into each duration bucket
The example demonstrates how to calculate DX12 CPU frames durartion and construct a histogram out of it.
CREATE INDEX DX12_API_ENDTS ON DX12_API (end);
CREATE TEMP VIEW DX12_API_FPS AS SELECT end AS start,
(SELECT end FROM DX12_API
WHERE end > outer.end AND nameId == (SELECT id FROM StringIds
WHERE value == "IDXGISwapChain::Present")
ORDER BY end ASC LIMIT 1) AS end
FROM DX12_API AS outer
WHERE nameId == (SELECT id FROM StringIds WHERE value == "IDXGISwapChain::Present")
ORDER BY end;
Number of frames with a duration of [X, X + 1] milliseconds.
SELECT
CAST((end - start) / 1000000.0 AS INT) AS duration_ms,
count(*)
FROM DX12_API_FPS
WHERE end IS NOT NULL
GROUP BY duration_ms
ORDER BY duration_ms;
# Python equivalent:
present_calls = dx12_api[dx12_api['function_name'] == 'IDXGISwapChain::Present']
present_calls = present_calls.sort_values('end')
frame_durations = []
for i in range(len(present_calls) - 1):
duration = present_calls.iloc[i+1]['end'] - present_calls.iloc[i]['end']
duration_ms = duration / 1000000.0 # Convert to milliseconds
frame_durations.append(int(duration_ms))
# Create histogram
histogram = pd.Series(frame_durations).value_counts().sort_index()
Results:
duration_ms count(*)
----------- ----------
3 1
4 2
5 7
6 153
7 19
8 116
9 16
10 8
11 2
12 2
13 1
14 4
16 3
17 2
18 1
GPU Context Switch Events Enumeration
Goal: Track GPU context switches to understand GPU scheduling behavior and identify context switch patterns.
What the code does: Filters GPU context switch events to show only BEGIN (tag=8) and END (tag=7) events, which mark the boundaries of GPU context execution periods.
GPU Context Switch Event Tags: - 7: END events (context execution ends) - 8: BEGIN events (context execution begins)
GPU context duration is between first BEGIN and a matching END event.
SELECT (CASE tag WHEN 8 THEN "BEGIN" WHEN 7 THEN "END" END) AS tag,
globalPid / 0x1000000 % 0x1000000 AS PID,
vmId, seqNo, contextId, timestamp, gpuId FROM GPU_CONTEXT_SWITCH_EVENTS
WHERE tag in (7, 8) ORDER BY seqNo LIMIT 10;
Results:
tag PID vmId seqNo contextId timestamp gpuId
---------- ---------- ---------- ---------- ---------- ---------- ----------
BEGIN 23371 0 0 1048578 56759171 0
BEGIN 23371 0 1 1048578 56927765 0
BEGIN 23371 0 3 1048578 63799379 0
END 23371 0 4 1048578 63918806 0
BEGIN 19397 0 5 1048577 64014692 0
BEGIN 19397 0 6 1048577 64250369 0
BEGIN 19397 0 8 1048577 1918310004 0
END 19397 0 9 1048577 1918521098 0
BEGIN 19397 0 10 1048577 2024164744 0
BEGIN 19397 0 11 1048577 2024358650 0
Resolve NVTX Category Name
Goal: Decode NVTX category names for NVTX markers and ranges to make the profiling data more human-readable.
What the code does: Joins NVTX events with their category definitions to resolve category IDs into meaningful category names, making it easier to understand the purpose of different NVTX annotations.
NVTX Event Types:
33: Category definition events (define new categories)
34: Mark events (instantaneous markers)
59: Push/Pop range events (nested ranges)
60: Start/End range events (paired ranges)
The example demonstrates how to resolve NVTX category name for NVTX marks and ranges.
WITH
event AS (
SELECT *
FROM NVTX_EVENTS
WHERE eventType IN (34, 59, 60) -- mark, push/pop, start/end
),
category AS (
SELECT
category,
domainId,
text AS categoryName
FROM NVTX_EVENTS
WHERE eventType == 33 -- category definition events
)
SELECT
start,
end,
globalTid,
eventType,
domainId,
category,
categoryName,
text
FROM event JOIN category USING (category, domainId)
ORDER BY start;
Results:
start end globalTid eventType domainId category categoryName text
---------- ---------- --------------- ---------- ---------- ---------- ------------------------- ----------------
18281150 18311960 281534938484214 59 0 1 FirstCategoryUnderDefault Push Pop Range A
18288187 18306674 281534938484214 59 0 2 SecondCategoryUnderDefaul Push Pop Range B
18294247 281534938484214 34 0 1 FirstCategoryUnderDefault Mark A
18300034 281534938484214 34 0 2 SecondCategoryUnderDefaul Mark B
18345546 18372595 281534938484214 60 1 1 FirstCategoryUnderMyDomai Start End Range
18352924 18378342 281534938484214 60 1 2 SecondCategoryUnderMyDoma Start End Range
18359634 281534938484214 34 1 1 FirstCategoryUnderMyDomai Mark A
18365448 281534938484214 34 1 2 SecondCategoryUnderMyDoma Mark B
Rename CUDA Kernels with NVTX
Goal: Associate CUDA kernels with their surrounding NVTX ranges to provide more meaningful names and context for kernel analysis.
What the code does: 1. Finds the innermost NVTX push-pop range that encompasses each CUDA kernel launch 2. Maps the NVTX range text to the corresponding kernel execution 3. Enables analysis of kernels by their logical function rather than just their technical names
The example demonstrates how to map innermost NVTX push-pop range to a matching CUDA kernel run.
ALTER TABLE CUPTI_ACTIVITY_KIND_KERNEL ADD COLUMN nvtxRange TEXT;
CREATE INDEX nvtx_start ON NVTX_EVENTS (start);
UPDATE CUPTI_ACTIVITY_KIND_KERNEL SET nvtxRange = (
SELECT NVTX_EVENTS.text
FROM NVTX_EVENTS JOIN CUPTI_ACTIVITY_KIND_RUNTIME ON
NVTX_EVENTS.eventType == 59 AND
NVTX_EVENTS.globalTid == CUPTI_ACTIVITY_KIND_RUNTIME.globalTid AND
NVTX_EVENTS.start <= CUPTI_ACTIVITY_KIND_RUNTIME.start AND
NVTX_EVENTS.end >= CUPTI_ACTIVITY_KIND_RUNTIME.end
WHERE
CUPTI_ACTIVITY_KIND_KERNEL.correlationId == CUPTI_ACTIVITY_KIND_RUNTIME.correlationId
ORDER BY NVTX_EVENTS.start DESC LIMIT 1
);
SELECT start, end, globalPid, StringIds.value as shortName, nvtxRange
FROM CUPTI_ACTIVITY_KIND_KERNEL JOIN StringIds ON shortName == id
ORDER BY start LIMIT 6;
# Python equivalent:
def find_innermost_nvtx_range(kernel_start, kernel_end, nvtx_ranges):
# Find NVTX ranges that completely contain the kernel
containing_ranges = []
for nvtx in nvtx_ranges:
if nvtx['start'] <= kernel_start and nvtx['end'] >= kernel_end:
containing_ranges.append(nvtx)
# Return the innermost (latest starting) range
if containing_ranges:
return max(containing_ranges, key=lambda x: x['start'])['text']
return None
Results:
start end globalPid shortName nvtxRange
---------- ---------- ----------------- ------------- ----------
526545376 526676256 72057700439031808 MatrixMulCUDA
526899648 527030368 72057700439031808 MatrixMulCUDA Add
527031648 527162272 72057700439031808 MatrixMulCUDA Add
527163584 527294176 72057700439031808 MatrixMulCUDA My Kernel
527296160 527426592 72057700439031808 MatrixMulCUDA My Range
527428096 527558656 72057700439031808 MatrixMulCUDA
Select CUDA Calls With Backtraces
Goal: Analyze CUDA API calls along with their call stacks to understand the application code paths that lead to CUDA API usage.
What the code does: Joins CUDA runtime API calls with their associated call chains to show the complete stack trace for each CUDA call, helping identify where in the application CUDA calls originate.
ALTER TABLE CUPTI_ACTIVITY_KIND_RUNTIME ADD COLUMN name TEXT;
UPDATE CUPTI_ACTIVITY_KIND_RUNTIME SET name = (SELECT value FROM StringIds WHERE CUPTI_ACTIVITY_KIND_RUNTIME.nameId = StringIds.id);
ALTER TABLE CUDA_CALLCHAINS ADD COLUMN symbolName TEXT;
UPDATE CUDA_CALLCHAINS SET symbolName = (SELECT value FROM StringIds WHERE symbol = StringIds.id);
SELECT globalTid % 0x1000000 AS TID,
start, end, name, callchainId, stackDepth, symbolName
FROM CUDA_CALLCHAINS JOIN CUPTI_ACTIVITY_KIND_RUNTIME ON callchainId == CUDA_CALLCHAINS.id
ORDER BY callchainId, stackDepth LIMIT 11;
Results:
TID start end name callchainId stackDepth symbolName
---------- ---------- ---------- ------------- ----------- ---------- --------------
11928 168976467 169077826 cuMemAlloc_v2 1 0 0x7f13c44f02ab
11928 168976467 169077826 cuMemAlloc_v2 1 1 0x7f13c44f0b8f
11928 168976467 169077826 cuMemAlloc_v2 1 2 0x7f13c44f3719
11928 168976467 169077826 cuMemAlloc_v2 1 3 cuMemAlloc_v2
11928 168976467 169077826 cuMemAlloc_v2 1 4 cudart::driver
11928 168976467 169077826 cuMemAlloc_v2 1 5 cudart::cudaAp
11928 168976467 169077826 cuMemAlloc_v2 1 6 cudaMalloc
11928 168976467 169077826 cuMemAlloc_v2 1 7 cudaError cuda
11928 168976467 169077826 cuMemAlloc_v2 1 8 main
11928 168976467 169077826 cuMemAlloc_v2 1 9 __libc_start_m
11928 168976467 169077826 cuMemAlloc_v2 1 10 _start
SLI Peer-to-Peer Query
Goal: Filter and analyze SLI (Scalable Link Interface) peer-to-peer memory transfers between GPUs based on size, timing, and other criteria.
What the code does: Demonstrates how to query SLI P2P events with filtering conditions on resource size, time range, and sorting by transfer size to identify significant GPU-to-GPU transfers.
SLI P2P Event Classes: - 62: Peer-to-peer transfer events between GPUs
The example demonstrates how to query SLI Peer-to-Peer events with resource size greater than value and within a time range sorted by resource size descending.
SELECT *
FROM SLI_P2P
WHERE resourceSize < 98304 AND start > 1568063100 AND end < 1579468901
ORDER BY resourceSize DESC;
# Python equivalent:
filtered_transfers = sli_p2p[
(sli_p2p['resourceSize'] < 98304) &
(sli_p2p['start'] > 1568063100) &
(sli_p2p['end'] < 1579468901)
].sort_values('resourceSize', ascending=False)
Results:
start end eventClass globalTid gpu frameId transferSkipped srcGpu dstGpu numSubResources resourceSize subResourceIdx smplWidth smplHeight smplDepth bytesPerElement dxgiFormat logSurfaceNames transferInfo isEarlyPushManagedByNvApi useAsyncP2pForResolve transferFuncName regimeName debugName bindType
---------- ---------- ---------- ----------------- ---------- ---------- --------------- ---------- ---------- --------------- ------------ -------------- ---------- ---------- ---------- --------------- ---------- --------------- ------------ ------------------------- --------------------- ---------------- ---------- ---------- ----------
1570351100 1570351101 62 72057698056667136 0 771 0 256 512 1 1048576 0 256 256 1 16 2 3 0 0
1570379300 1570379301 62 72057698056667136 0 771 0 256 512 1 1048576 0 64 64 64 4 31 3 0 0
1572316400 1572316401 62 72057698056667136 0 773 0 256 512 1 1048576 0 256 256 1 16 2 3 0 0
1572345400 1572345401 62 72057698056667136 0 773 0 256 512 1 1048576 0 64 64 64 4 31 3 0 0
1574734300 1574734301 62 72057698056667136 0 775 0 256 512 1 1048576 0 256 256 1 16 2 3 0 0
1574767200 1574767201 62 72057698056667136 0 775 0 256 512 1 1048576 0 64 64 64 4 31 3 0 0
Generic Events
Goal: Analyze system-level events captured through generic event collection (like ftrace) to understand system behavior and syscall patterns.
What the code does: Demonstrates how to query generic events stored in JSON format, specifically showing how to create a histogram of syscall usage by process ID. The query uses a subquery to find the specific event type ID for “raw_syscalls:sys_enter” events, then counts occurrences by process ID.
Syscall usage histogram by PID:
SELECT json_extract(data, '$.common_pid') AS PID, count(*) AS total
FROM GENERIC_EVENTS WHERE PID IS NOT NULL AND typeId = (
SELECT typeId FROM GENERIC_EVENT_TYPES
WHERE json_extract(data, '$.Name') = "raw_syscalls:sys_enter")
GROUP BY PID
ORDER BY total DESC
LIMIT 10;
# Python equivalent:
import json
# Filter for syscall enter events
syscall_events = []
for event in generic_events:
data = json.loads(event['data'])
if 'common_pid' in data:
syscall_events.append(data['common_pid'])
# Count syscalls by PID
pid_counts = pd.Series(syscall_events).value_counts().head(10)
Results:
PID total
---------- ----------
5551 32811
9680 3988
4328 1477
9564 1246
4376 1204
4377 1167
4357 656
4355 655
4356 640
4354 633
Fetching Generic Events in JSON Format
Goal: Export generic events, types, and sources in JSON format for external analysis tools or custom processing pipelines.
What the code does: Constructs JSON objects from the database tables containing generic event data, enabling export to JSON Lines format for further processing with external tools.
Use the below queries (without the LIMIT clause) to extract JSON lines representation of generic events, types, and sources.
SELECT json_insert('{}',
'$.sourceId', sourceId,
'$.data', json(data)
)
FROM GENERIC_EVENT_SOURCES LIMIT 2;
SELECT json_insert('{}',
'$.typeId', typeId,
'$.sourceId', sourceId,
'$.data', json(data)
)
FROM GENERIC_EVENT_TYPES LIMIT 2;
SELECT json_insert('{}',
'$.rawTimestamp', rawTimestamp,
'$.timestamp', timestamp,
'$.typeId', typeId,
'$.data', json(data)
)
FROM GENERIC_EVENTS LIMIT 2;
Results:
json_insert('{}',
'$.sourceId', sourceId,
'$.data', json(data)
)
---------------------------------------------------------------------------------------------------------------
{"sourceId":72057602627862528,"data":{"Name":"FTrace","TimeSource":"ClockMonotonicRaw","SourceGroup":"FTrace"}}
json_insert('{}',
'$.typeId', typeId,
'$.sourceId', sourceId,
'$.data', json(data)
)
--------------------------------------------------------------------------------------------------------------------
{"typeId":72057602627862547,"sourceId":72057602627862528,"data":{"Name":"raw_syscalls:sys_enter","Format":"\"NR %ld (%lx, %lx, %lx, %lx, %lx, %lx)\", REC->id, REC->args[0], REC->args[1], REC->args[2], REC->args[3], REC->args[4], REC->args[5]","Fields":[{"Name":"common_pid","Prefix":"int","Suffix":""},{"Name":"id","Prefix":"long","S
{"typeId":72057602627862670,"sourceId":72057602627862528,"data":{"Name":"irq:irq_handler_entry","Format":"\"irq=%d name=%s\", REC->irq, __get_str(name)","Fields":[{"Name":"common_pid","Prefix":"int","Suffix":""},{"Name":"irq","Prefix":"int","Suffix":""},{"Name":"name","Prefix":"__data_loc char[]","Suffix":""},{"Name":"common_type",
json_insert('{}',
'$.rawTimestamp', rawTimestamp,
'$.timestamp', timestamp,
'$.typeId', typeId,
'$.data', json(data)
)
--------------------------------------------------------------------------------------------------------------------
{"rawTimestamp":1183694330725221,"timestamp":6236683,"typeId":72057602627862670,"data":{"common_pid":"0","irq":"66","name":"327696","common_type":"142","common_flags":"9","common_preempt_count":"0"}}
{"rawTimestamp":1183694333695687,"timestamp":9207149,"typeId":72057602627862670,"data":{"common_pid":"0","irq":"66","name":"327696","common_type":"142","common_flags":"9","common_preempt_count":"0"}}
Arrow#
The Arrow type exported file, .arrows, uses the IPC stream format to store all tables in a
file. The tables can be read by opening the file as an arrow stream. For example one can use the
open_stream function from the arrow python package. For more information on the interfaces that
can be used to read an IPC stream file, please refer to the Apache Arrow documentation
[1, 2].
The name of each table is included in the schema metadata. Thus, while reading each table, the user
can extract the table title from the metadata. The table name metadata field has the key
table_name. The titles of all the available tables can be found in section
SQLite Schema Reference.
A sample function that reads all Arrow tables in a .arrows file is provided below in Python:
import pyarrow as pa
def read_tables(arrow_file):
with pa.input_stream(arrow_file) as source:
while source.tell() < source.size():
try:
yield pa.ipc.open_stream(arrow_file).read_all()
except:
continue
The Arrow directory exporter type, _arwdir, will create a directory with one arrow file per
table/dataset.
JSON Lines#
In the JSON Lines export format (JSON Lines Documentation), events and other report data (such as strings and processes) are serialized into JSON objects, with each object written to a new line.
Output layout:
{"id":0,"table":"StringIds","value":"[Unknown]"}
{"globalPid":284057963331584,"name":"chrome","pid":153958,"table":"PROCESSES"}
{"globalTid":281523009882942,"nameId":442,"priority":20,"table":"ThreadNames"}
{"name":"COLLECT_GPU_CTX_SW_TRACE","table":"META_DATA_CAPTURE","value":"false"}
...
Note the presence of the “table” field in each JSON object. This field allows readers to identify
the type of the event and corresponds to the table name in the sqlite export.
Common Analysis Pitfalls#
This section collects common mistakes made when interpreting Nsight Systems traces and when querying the exported SQLite database. The first subsection covers domain-level misconceptions about GPU/CPU performance; the second covers SQL traps specific to the Nsight Systems SQLite schema.
GPU Performance Analysis Pitfalls#
When interpreting GPU/CPU performance traces, a handful of misconceptions recur and lead to incorrect conclusions. Each entry below pairs a common but incorrect assumption with the correct interpretation.
CPU vs GPU cost confusion#
Resolution does not affect CPU-side API call cost#
Incorrect: “Present() / Draw() / Dispatch() / vkQueueSubmit() takes longer at higher resolutions because there are more pixels to process.”
Correct: The CPU cost of issuing these API calls is resolution-independent. The CPU records a command; it does not process pixels. GPU-side work (compositing, copying the back-buffer) may scale with resolution, but that work is asynchronous and does not appear on the CPU timeline unless explicitly synchronized via a fence or wait.
Command-buffer submission cost is not proportional to GPU workload#
Incorrect: “Submitting more complex shaders takes longer on the CPU.”
Correct: CPU-side command recording and submission cost is driven by the number and type of API calls, state changes, and descriptor updates – not by shader complexity or GPU workload size.
Synchronization and async operations#
Async compute / copy does not block the CPU#
Incorrect: “The async compute work is causing a CPU stall.”
Correct: Work submitted to async compute or copy queues does not block the CPU timeline unless the application explicitly waits on a fence tied to that work. If a CPU stall coincides with async GPU work, the stall is caused by whatever fence or event the CPU is waiting on – identify the specific synchronization point.
Fence waits vs work completion#
Incorrect: “The CPU is waiting for the GPU to finish [specific pass].”
Correct: The CPU waits on a fence value, not on a specific pass. The fence may be signaled after multiple passes complete. Identify which fence the CPU is waiting on and what GPU work signals that fence.
GPU copy duration is not host blocking time#
Incorrect: “This GPU memory copy took 2 ms, so the host was blocked for 2 ms.”
Correct: A GPU memory-copy event shows how long the transfer took, not how long the CPU thread was blocked. Inspect the associated CUDA API call on the CPU timeline and the CUDA synchronization rules.
An empty
cuda_memcpy_syncresult does not rule out host blocking. The recipe excludesAsynccalls, which can also block the CPU thread. See Asynchronous memcpy with pageable memory.
Barrier and synchronization costs#
Barriers have separate CPU and GPU costs#
Incorrect: “Barriers are expensive”.
Correct: Resource barriers have two distinct costs at different points in time:
CPU cost: recording the barrier into the command buffer (typically negligible per-barrier, but can accumulate with hundreds per frame).
GPU cost: cache flush / invalidate and possible decompression (varies enormously by resource size, transition type, split-vs-immediate).
Always check and specify which cost you mean and cite measurements for the specific barrier(s) in question.
Barrier storms: count matters, but not all barriers are equal#
Incorrect: “This frame has 200 barriers, so barriers are the bottleneck.”
Correct: Barrier count alone does not determine cost. A barrier transitioning a small buffer between shader stages costs far less than one decompressing a render target. Evaluate barrier cost by measuring GPU time consumed by barrier operations, not by counting them.
Draw calls and rendering cost#
Draw-call count is not the primary cost driver#
Incorrect: “The frame has 5000 draw calls, so draw-call overhead is the problem.”
Correct: On modern APIs (D3D12, Vulkan), per-draw CPU overhead is minimal because command recording is explicit and multi-threaded. The cost drivers are: state changes between draws (PSO switches, root signature / pipeline-layout changes, descriptor table / set updates), command-buffer structure, and GPU-side work per draw. A frame with 5000 draws but few state changes may be faster than one with 500 draws and frequent PSO switches.
Memory and bandwidth#
Theoretical peak bandwidth is not achieved bandwidth#
Incorrect: “The GPU has 1 TB/s bandwidth, so this shader should be able to read X GB in Y ms.”
Correct: Achieved memory bandwidth depends on access pattern, cache behavior, and contention. Real-world utilization is typically 60-85% of theoretical peak for well-optimized compute workloads, lower for mixed graphics workloads. Use measured bandwidth (from profiler SOL metrics) rather than theoretical calculations.
VRAM capacity vs bandwidth#
Incorrect: “The game uses 8 GB of VRAM, so memory is under pressure.”
Correct: VRAM capacity usage and bandwidth usage are independent concerns. A game can use most of VRAM without any bandwidth bottleneck (textures resident but not actively sampled), or it can have severe bandwidth pressure while using a fraction of capacity. Identify which dimension is the actual concern.
Timing and measurement#
API wall-clock duration is not active CPU time#
Incorrect: “
UpdateTileMappings/vkAllocateMemorytook 14.5 ms – that’s 14.5 ms of CPU work.”Correct: API call durations from
DX12_API/DXGI_API/VULKAN_APItables are wall-clock from call start to call return. That includes: (a) on-CPU execution, (b) kernel lock waits (e.g. DXGI Present contending with DWM), (c) user-mode waits (e.g. fence waits), (d) scheduling delays while the thread is runnable but off-CPU. A 14.5 ms call may be 2 ms of CPU work and 12.5 ms blocked. Distinguish wall-clock from active CPU time usingSCHED_EVENTSto see on-CPU vs blocked within the call window.
Present / vkQueuePresentKHR latency vs frame time#
Incorrect: “Present takes 5 ms, which is adding to frame time.”
Correct: Present latency (time from CPU call to actual flip / display) includes wait time for vsync, compositor, and display refresh. This latency is largely not the application’s GPU/CPU work. Distinguish between: (a) CPU time spent in the Present call (which may include a blocking wait), (b) GPU time for any associated copy/composite, and (c) display latency (vsync alignment, present-mode queueing).
Aggregation and ranking#
High total duration does not imply slow individual calls#
Incorrect: “The API or kernel with the highest total duration is the slowest individual operation.”
Correct: A short operation that runs often can have the largest total duration. For example, 1,000 calls taking 20 microseconds each total 20 milliseconds, more than 10 calls taking 1 millisecond each. In statistics recipes such as
cuda_api_sum,Sumis the total duration,Meanis the average per call, andMaxis the longest single call.
Kernel activity is not SM utilization#
Incorrect: “Summed kernel duration divided by the kernel time span is SM utilization.”
Correct: Kernel intervals show when recorded kernels ran, not how many streaming multiprocessors (SMs) were active. See the GPU Low Utilization rule for the distinction between time and resource utilization, including overlapping operations. Use GPU metrics such as SMs Active, or Nsight Compute measurements, to examine hardware activity.
Investigation discipline#
Reversed causation#
GPU idle, paging bursts, and fence waits during a slow window are often consequences, not causes. Always trace backward to find the initiator (the operation that pushed VRAM over budget, the thread that took the lock, the API call that blocked).
SQLite Schema Pitfalls#
Common SQL traps and misinterpretations when querying the .sqlite database
produced by nsys export. Read these alongside the SQLite Schema Reference
(table columns, ID encoding, timestamps, and query patterns) and the GPU
performance analysis pitfalls above.
Timestamp filtering#
startTime/stopTimein metadata are wall-clock times, not event-timestamp references – never use them to filter events. Filter against the trace-relativetimestampcolumn (in nanoseconds) instead. See the SQLite Schema Reference for the timestamp model.
Missing tables and incomplete data#
If a table is missing, check whether it is supported by the Nsight Systems version used to export the report. A missing or empty table does not prove that the application never used the corresponding API or feature. For collection checks, see Empty or nearly empty report.
GPU identity#
CUDA activity deviceId values are process-specific CUDA device IDs, not necessarily the IDs in TARGET_INFO_GPU. CUDA_VISIBLE_DEVICES can change device visibility and enumeration order, so do not assume the IDs match.
When TARGET_INFO_CUDA_DEVICE is present, match the activity’s process and device IDs to pid and cudaId. Use the matching row’s gpuId when available. Decode serialized process identifiers before matching them to pid.
A row in TARGET_INFO_GPU identifies a GPU known to the capture. It does not prove that the application used that GPU. Check the activity records for the mapped GPU. No recorded activity does not prove the GPU was idle if the relevant tracing was not enabled.
Process and thread identity#
Values such as globalPid and globalTid are serialized identifiers, not plain operating-system process and thread IDs. See the serialized identifier example for the encoding and an example query.
Avoid counting GPU metric samples twice#
Joining sampled GPU metrics to overlapping kernel rows can duplicate samples and distort averages or totals. Calculate sample averages from GPU_METRICS for the selected metric, device, and time window, rather than from the kernel join.
Memory-copy totals#
For CUDA memory-copy volume, sum bytes in CUPTI_ACTIVITY_KIND_MEMCPY. CUDA’s MemcpyBatchAsync APIs can batch multiple copy operations into one activity record. The copyCount field records how many copies it represents, so counting rows does not always give the number of copy operations.
SCHED_EVENTS gotchas#
No
endordurationcolumn. Compute on-CPU intervals withLEAD(start) OVER (PARTITION BY globalTid ORDER BY start).threadBlockis only meaningful whenisSchedIn = 0. Reading it on a sched-in event gives a stale value from the previous off-CPU transition.
Attribution rule (block reason vs thread state)#
Interpret the (block reason, thread state) pair, not the block reason alone – the value lists are in SQLite Schema Event Values. Before concluding lock contention, verify the stalled thread’s threadBlock shows a blocking reason (Resource, UserRequest, KeyedEvent) during the specific window. NonBlocked / Running means the thread was actively computing and got preempted – that is not lock contention.
ENUM tables: do not use value#
All ENUM_* tables have columns id, name, label – there is no value column. Filtering by b.value = '…' fails with a SQLite column error:
-- Right
JOIN ENUM_SCHEDULING_THREAD_BLOCK b ON se.threadBlock = b.id
WHERE b.name = 'Resource'
-- Wrong (no `value` column)
JOIN ENUM_SCHEDULING_THREAD_BLOCK b ON se.threadBlock = b.id
WHERE b.value = 'Resource'
cpuCycles attribution#
Always filter cpuCycles = 1 when attributing CPU time.
cpuCycles is the value of the CpuCycles PMU event carried by a composite event: a periodic CPU-cycle sample is tagged with value 1, and any callstack captured for another reason (a scheduling / context-switch transition) carries no such event and exports as 0.
This holds on both Linux and Windows. On Linux, only PERF_RECORD_SAMPLE records are stamped CpuCycles=1 (context-switch records become SCHED_EVENTS), and on Windows only single-frame stackwalks bearing a CpuCycles event count as samples (CSwitch / ReadyThread stackwalks do not).
Including cpuCycles = 0 therefore distorts hotspot results by over-counting functions that coincide with frequent scheduling events. cpuCycles = 0 callstacks are non-sample (scheduling-event) stacks – they belong with scheduling-event analysis (wake / block reasons), not with hotspot identification. Equivalently, weight by SUM(cpuCycles), which counts non-samples as 0 (this is what the shipped flat / bottom-up views do).
No periodic CPU samples in the capture#
If periodic CPU-cycle sampling was not enabled for the capture, then such samples will not be collected – a count of 0 samples is then a data-absence artifact, not confirmation of inactivity.
Graphics Schema Pitfalls#
Common SQL traps and misinterpretations when querying graphics-related tables in
the .sqlite database produced by nsys export. Read these alongside the
SQLite Schema Reference (table columns, ID encoding, timestamps, and query patterns)
and the SQLite Schema Pitfalls section.
ETW_EVENTS gotchas (Windows traces)#
Columns are
timestamp,typeId,globalTid,opcode,data. Nodurationcolumn – ETW events are point-in-time, not intervals.Event names go through
GENERIC_EVENT_TYPES.nameId->StringIds.value. Always join; comparingnameIdagainst a string literal silently matches nothing.etwEventIdis not a column onETW_EVENTS– it lives onGENERIC_EVENT_TYPES(the event-type metadata), reached via the sameETW_EVENTS.typeId->GENERIC_EVENT_TYPES.typeIdjoin as above. To filter by event id, join through the type table (WHERE t.etwEventId = 184), neverWHERE etwEventId = 184directly onETW_EVENTS.DxgKrnl
Presentevents are the event type withetwEventId = 184(the canonical CPU frame boundary on Windows). OtherPresentIDs (42 / 43) appear too – de-duplicate by sticking to 184 for frame counting.
DX12_API / VULKAN_API / OPENGL_API gotchas#
Columns are
start,end,globalTid,nameId. Duration =(end - start). Usestart/end, nottimestamp– there is notimestampcolumn.API name goes through
StringIds.value. Always join.
Graphics Glossary#
Definitions of graphics terms used in Nsight Systems reports, grouped by area.
Frames and timing
Graphics frame - one iteration of an application’s render loop
CPU-GPU frame pipelining - CPU preparing one frame while the GPU renders another
CPU-bound frame - frame limited by CPU work
GPU-bound frame - frame limited by GPU work
GPU bubble - idle gap on a GPU engine while the application still has work to do
Stutter - frame-time spike that breaks pacing even when average FPS is fine
Reflex render latency - Reflex SDK markers and the per-frame latency stages
VSync - vertical-blank synchronization
Graphics APIs
Direct3D 12 (DX12) - Direct3D 12: explicit Windows graphics and compute API
Direct3D 11 (DX11) - Direct3D 11: previous-generation Windows graphics API
Vulkan - Khronos cross-platform explicit graphics and compute API
OpenGL - Khronos legacy state-machine graphics API
Graphics pipeline - shader stages from input assembly to output merger
Command list and queue - D3D12 command lists, queues, and bundles
Vulkan command buffer - Vulkan counterpart of a D3D12 command list
Pipeline state object - pipeline state object: compiled shaders and fixed-function state
Fence - GPU and CPU synchronization object in D3D12 and Vulkan
Barrier - resource state and pipeline barriers between GPU work
Swap chain - DXGI and Vulkan back-buffer rotation
Present mode - flip and blit models, and the FIFO, Mailbox, and Immediate modes
DXGI - DirectX Graphics Infrastructure and its ETW events
Render target - render target view or framebuffer attachment
ETW, WDDM, and GPU scheduling
ETW - Event Tracing for Windows
ETW provider mask - provider keyword and level bitmask
DxgKrnl events - DirectX graphics kernel ETW events (catalog)
WDDM - Windows Display Driver Model
VidMm and VidSch - WDDM video memory manager and scheduler
GPU context switch - GPU scheduler swapping graphics contexts on an engine
GPU engine - the 3D, Copy, video, and other engines
Queue packet - DxgKrnl software-queue packet
DMA packet - GPU engine DMA packet
Hardware queue - queue that the GPU schedules (WDDM 2.7 and later)
GPU memory and resources
Memory transfer - SystemToDevice, DeviceToSystem, and other WDDM transfer kinds
Resource migration - moving an allocation between memory segments
Performance markers
Debug marker - PIX and Vulkan debug-utils labels
For NVTX primitives (domain, range, mark, payload, category), see NVTX Trace.
Graphics frame#
One iteration of an application’s render loop. On the CPU, a Present (or SwapBuffers) call to the graphics API ends it.
A frame is the unit of work that produces one image for the display.
Each iteration typically runs input sampling, simulation, render-command recording, submission to a GPU queue, and a final present.
The present call hands the just-rendered image to the display system and conceptually closes the frame.
A frame has both a CPU side (the work the application thread did) and a GPU side (the work the device actually executed), and the two are usually offset in time because the GPU runs behind the CPU.
Frames are usually numbered in order, so that analysis can line up the CPU work, GPU work, and display events of the same frame.
A consistent frame cadence matters more for perceived smoothness than a high average frame rate.
See also: FPS Overview, CPU-GPU frame pipelining, VSync
CPU-GPU frame pipelining#
The CPU prepares a newer frame while the GPU renders an earlier one, so CPU and GPU work overlap.
Flip-model swap chains queue presents. When the present queue reaches its configured limit, the application must wait for an earlier frame to leave the queue before it can submit another.
A stable pipeline overlaps CPU submission for one frame with GPU execution of another. Reducing the number of frames allowed in flight reduces latency but also reduces this CPU-GPU parallelism.
When this overlap is lost from time to time, CPU present intervals alternate between long and short, a pattern called a render queue flip-flop or CPU-GPU swap-chain desynchronization.
Further reading:
Microsoft: Reduce latency with DXGI 1.3 swap chains explains queued flips, Present blocking when the queue has no room, and the CPU-GPU parallelism tradeoff of frame-latency settings.
Microsoft: GetFrameLatencyWaitableObject documents synchronizing each new frame with completion of an earlier presentation.
Intel: D3D12 flip-model swap-chain sample describes the present-queue limit, Present blocking, and the need for multiple frames in flight to preserve CPU-GPU overlap.
See also: Swap chain, Fence, Present / vkQueuePresentKHR latency vs frame time
CPU-bound frame#
A frame whose duration is limited by CPU work, not by GPU execution.
The CPU is the bottleneck when game logic, draw-call recording, or driver submission take longer than the GPU needs to render the resulting work.
Typical signs are a fully busy main or render thread, a GPU that goes idle between frames, and frame time that scales with CPU clock or thread count, not with GPU settings.
Common causes include expensive simulation or scripting, costly state changes between draws, contention on a single thread, and synchronous readbacks that stall the CPU on the GPU. The number of draw calls alone is rarely the cost; see Draw-call count is not the primary cost driver.
Lowering graphics quality usually does not help a CPU-bound frame, because the limit is on the submission side, not the rendering side.
Mitigations include reducing state changes between draws, moving work to worker threads, reducing per-frame allocations, and avoiding API patterns that force the CPU to wait on the GPU.
A run can shift between CPU-bound and GPU-bound across scenes, so the classification applies to a frame, not to the whole application.
See also: GPU idle decomposition, GPU-bound frame, GPU bubble
GPU-bound frame#
A frame whose duration is limited by GPU execution; the CPU finishes submitting work and then waits for the GPU.
The GPU is the bottleneck when rendering, compute, or copy work for a frame takes longer than the CPU needs to record and submit it.
Typical signs are a GPU that is busy back-to-back across frames, a CPU thread that finishes early and idles, and frame time that scales with resolution, quality settings, or effects.
Common causes include heavy pixel shaders at high resolution, expensive post-processing, large geometry or overdraw, and memory bandwidth limits.
Lowering resolution, render scale, or shader quality usually helps, because the limit is on the rendering side.
A run can shift between GPU-bound and CPU-bound across scenes, so the classification applies to a frame, not to the whole application.
See also: GPU idle decomposition, GPU Metrics, CPU-bound frame
GPU bubble#
A gap in GPU execution where one or more engines sit idle with no work queued, even though the application has more work to do; visible on the timeline as empty space between GPU activity bars.
A bubble means the submission pipeline could not keep the GPU fed: the CPU was slow to record and submit, a host-side stall blocked submission, or a fence wait paused a queue.
A bubble on one engine can come from a hand-off to another: a 3D engine waiting for a Copy engine’s upload, or the other way around; cross-engine fences make this explicit.
Not every gap is a bubble. Vsync-paced workloads intentionally idle the GPU after Present; a gap inside a frame is a problem, a gap after Present is by design.
Fixes typically involve recording command buffers earlier, parallelizing recording, double-buffering work, or removing stalls in the submission path.
With hardware-accelerated GPU scheduling, the DMA packet rows miss most application work, so an engine can look idle while it is busy; see hardware-accelerated GPU scheduling.
See also: GPU idle decomposition, CPU-bound frame, GPU engine
Stutter#
A perceptible hitch in frame pacing: one or more frames take much longer than the steady state, so the image on screen jerks even if the average frame rate looks healthy.
Stutter is about variance, not average. A 60 FPS average with a single 100 ms frame is more disruptive than a steady 50 FPS, because the eye notices the sudden jump.
Frame-pacing strategies (capping at a fraction of the refresh rate, Reflex, decoupled simulation and render) smooth the delivery cadence.
A stutter shows up as a tall bar in the CPU Frame Duration row and is marked in the Stutter row (see FPS Overview). The cause is usually a long CPU range or a GPU bubble on a thread or engine in the same frame.
For common causes and how to check each one, see Common stutter causes.
See also: Stutter Analysis, CPU-bound frame, Pipeline state object
Reflex render latency#
NVIDIA Reflex splits each frame into latency stages, from input and simulation to GPU rendering, so you can see where render latency builds up.
Every stage carries a frame ID, which lets a tool tell which work belongs to which frame. Most stages have a start and an end time. Input Sample, Trigger Flash, PC Latency Ping and a few other markers are single points in time. The Driver, OS Render Queue and GPU Render stages come from the latency report of the NVIDIA driver, not from application markers.
Because the data is per stage, it shows which stage adds latency, instead of leaving you to guess from the overall frame time.
Latency is most visible in fast, aim-driven games, where even a few milliseconds change how responsive the controls feel.
Under the Frame Duration rows of a process, the Reflex Latency row shows one range per frame, from the first Reflex marker (Input Sample or Simulation) to the end of GPU Render. Its color compares that latency with the CPU frame duration: green below 2.5 frames, yellow up to 3 frames, and red above. Its child rows show each stage; FPS Overview lists the main ones.
See also: Present / vkQueuePresentKHR latency vs frame time, Graphics frame, Stutter
VSync#
Vertical-blank synchronization; the display refresh boundary at which a new scanout begins.
A display refreshes at a fixed rate, and the vertical blank is the brief interval between scanning out one image and starting the next.
VSync ties the present of a new frame to that boundary so the display never shows two frames stitched together, which would appear as tearing.
The cost is added latency and frame-time quantization: a frame that misses the boundary waits a full refresh interval before being shown.
Variable refresh rate displays relax this by letting the display wait for the GPU within a range, removing most of the wait without tearing.
Each connected display has its own VSync cadence, so multi-monitor setups can have several independent timelines.
Lining VSync events up with CPU and GPU frame timings is the standard way to diagnose stutter, dropped frames, and present-time hitches.
See also: Vertical Synchronization, Present / vkQueuePresentKHR latency vs frame time, Present mode
Direct3D 12 (DX12)#
Microsoft’s low-level explicit graphics and compute API for Windows. The application owns resource state, command recording, queue submission, and CPU and GPU synchronization, and the driver does little validation at submit time.
Work is recorded into command lists on one or more CPU threads and submitted to typed queues (Direct, Compute, Copy); see Command list and queue.
Shaders and most fixed-function state are compiled into pipeline state objects; see Pipeline state object.
Resources live in heaps, and descriptors live in descriptor heaps. Shaders reach CBV, SRV, UAV, and sampler descriptors through the root signature; render-target and depth-stencil views (RTV, DSV) are bound with
OMSetRenderTargets.The application, not the driver, transitions resource state; see Barrier.
Queues and the CPU synchronize through fences; see Fence.
See also: D3D12 API Trace, DXGI, Vulkan
Direct3D 11 (DX11)#
Microsoft’s previous-generation Windows graphics API; an immediate-mode runtime where the driver manages resource state, validates commands, and hides queue submission behind a device context.
The application talks to
ID3D11Device(creation) andID3D11DeviceContext(drawing). The immediate context submits commands directly; deferred contexts let secondary threads recordID3D11CommandListobjects that the immediate context later replays.State is set with binders on the context:
IASetVertexBuffers,VSSetShader,PSSetShaderResources,OMSetRenderTargets,RSSetState. The driver tracks which state is dirty and applies it lazily on the next draw.Resource state (read, write, depth, present) is implicit: the driver inserts barriers and waits. This shifts work from the application onto the driver and is the main source of single-threaded CPU cost in DX11.
Synchronization with the CPU goes through
ID3D11Query(event, occlusion, and timestamp queries); explicit GPU fences came later asID3D11Fence.Compute shaders, tessellation, geometry shaders, and stream output are all available. Ray tracing and mesh shaders are DX12-only.
See also: DXGI, Direct3D 12, Swap chain
Vulkan#
Khronos’s explicit, low-overhead cross-platform graphics and compute API; the application owns memory, command recording, queue submission, and synchronization, and the driver does very little on its own.
The object model centers on
VkInstance,VkPhysicalDevice,VkDevice,VkQueue,VkCommandPool,VkCommandBuffer,VkPipeline, and descriptor sets bound through aVkPipelineLayout.Work is recorded into command buffers and submitted to queues; see Vulkan command buffer.
Synchronization is fully explicit, with barriers, image layouts, semaphores, and fences; see Barrier and Fence.
Memory is allocated as
VkDeviceMemoryand sub-allocated by the application; resources (buffers, images) bind into those allocations. Most engines layer an allocator like VMA on top.Capabilities come from device features and many extensions: ray tracing, mesh shaders, dynamic rendering, descriptor indexing, sync2, and video decode and encode.
See also: Vulkan command buffer, Direct3D 12, Swap chain
OpenGL#
Khronos’s long-running cross-platform graphics API; a state-machine model where the application binds objects and calls draw functions on a single current context per thread, with the driver hiding command-buffer construction and submission.
A context (
HGLRCon Windows through WGL,GLXContexton X11,EGLContexton Android, Wayland, and embedded systems) is current on one thread at a time; all GL calls go through that current context.The API is a global state machine:
glBindBuffer,glBindTexture,glUseProgram,glBindFramebuffer. Modern OpenGL adds Vertex Array Objects (VAOs), Uniform Buffer Objects (UBOs), and Shader Storage Buffer Objects (SSBOs) to reduce the number of bind calls.Compute shaders, indirect draws (
glDrawArraysIndirect), tessellation, and transform feedback are core features of OpenGL 4.x. Bindless textures need an extension such asGL_ARB_bindless_texture.Presentation is platform-specific:
SwapBufferson Windows,glXSwapBufferson X11,eglSwapBufferselsewhere.Debug groups (
glPushDebugGroupandglPopDebugGroup) and object labels (glObjectLabel) give human-readable annotations that show up in tools.
See also: Graphics pipeline, Debug marker, Swap chain
Graphics pipeline#
The ordered sequence of fixed-function and programmable stages a GPU runs to turn vertex data into rendered pixels.
The classic graphics pipeline runs Input Assembly (IA), Vertex Shader (VS), the optional Hull, Tessellator, and Domain stages (HS, TS, DS), an optional Geometry Shader (GS), Rasterizer (RS), Pixel Shader (PS), and Output Merger (OM).
Newer mesh-shader pipelines replace IA, VS, HS, TS, DS, and GS with an Amplification Shader and a Mesh Shader that emit meshlets directly.
Compute is a sibling pipeline with a single programmable stage; it runs a kernel over a thread grid and is launched with
DispatchorvkCmdDispatchinstead ofDraw.State for the pipeline (shaders, blend, depth, raster, root signature, descriptor layout) is baked into a pipeline state object so the driver can validate and compile once.
The pipeline matters because each stage has different bottlenecks: vertex-bound, pixel-bound, and bandwidth-bound workloads need different tuning.
See also: Pipeline state object, Render target, Draw-call count is not the primary cost driver
Command list and queue#
In D3D12, a command list records GPU commands on the CPU, and a command queue runs batches of those lists on the GPU through ExecuteCommandLists.
Command lists come in types: direct (graphics, compute, and copy), compute, copy, and bundle; the type must match the queue that executes it.
A command queue is the GPU-side scheduler endpoint; multiple queues of different types can run in parallel and synchronize through fences.
Bundles are short, reusable secondary lists invoked from a primary direct list; they let the driver pre-validate a fixed sequence of commands so it can be replayed cheaply.
Recording is single-threaded per list but multiple lists can be recorded on different threads, which is the main way D3D12 scales CPU submission work.
Each command list starts with default state, even inside one
ExecuteCommandListscall, so each list must set its own pipeline state, descriptor heaps, and render targets.
See also: D3D12 API Trace, WDDM Queues, Vulkan command buffer
Vulkan command buffer#
A VkCommandBuffer records draws, dispatches, copies, and synchronization on the CPU for later execution on a VkQueue.
Command buffers are allocated from a
VkCommandPoolthat is tied to a single queue family; pools are not thread-safe, so each recording thread typically owns its own pool.Primary command buffers are submitted to a queue with
vkQueueSubmit,vkQueueSubmit2, orvkQueueSubmit2KHR; secondary command buffers can only be executed from a primary one throughvkCmdExecuteCommands.A command buffer has an explicit lifecycle (initial, recording, executable, pending, invalid) controlled by
vkBeginCommandBuffer,vkEndCommandBuffer, andvkResetCommandBuffer.vkQueueSubmittakes one or moreVkSubmitInfobatches. Each batch lists wait semaphores, command buffers, and signal semaphores, which order GPU work across queues. The optionalVkFencepassed tovkQueueSubmitis signaled when the submitted work finishes, so the CPU can wait for it.Synchronization inside a buffer is the caller’s responsibility: pipeline barriers, events, and render-pass dependencies must be inserted explicitly, unlike higher-level APIs.
See also: Command list and queue, Fence, Barrier
Pipeline state object#
A compiled, pre-validated bundle of shaders and fixed-function state that the GPU binds as a single unit.
In D3D12, a graphics
ID3D12PipelineStatepacks the vertex, pixel, hull, domain, and geometry shaders together with blend, depth-stencil, rasterizer, input layout, primitive topology type, RTV and DSV formats, and sample description.A compute PSO is much simpler: just the compute shader plus its root signature linkage.
The root signature (D3D12) or
VkPipelineLayout(Vulkan) defines the binding interface; the PSO is validated against it at creation so binding at draw time is cheap.Vulkan’s equivalent is
VkPipeline, created as graphics or compute throughvkCreateGraphicsPipelinesorvkCreateComputePipelines; pipeline libraries (VK_EXT_graphics_pipeline_library) allow incremental compilation of vertex-input, pre-rasterization, fragment, and fragment-output sub-states.Creating a PSO compiles its shaders for the GPU, and an engine that does this on the render thread when a draw first needs it stalls the frame. PSO caches and creating PSOs ahead of time avoid this.
See also: Common stutter causes, Draw-call count is not the primary cost driver, Graphics pipeline
Fence#
A synchronization object that tells the CPU or another queue when submitted GPU work has finished. D3D12 fences and Vulkan timeline semaphores use a 64-bit counter value; a Vulkan VkFence is binary.
In D3D12,
ID3D12Fenceholds a 64-bit value;ID3D12CommandQueue::Signalwrites a value when prior work on that queue completes, andWaitmakes a queue stall until the fence reaches a target value.In Vulkan,
VkFenceis a binary GPU-to-CPU signal, while timeline semaphores (VkSemaphorecreated withVK_SEMAPHORE_TYPE_TIMELINE) provide the equivalent monotonic-counter behavior for GPU-to-GPU and GPU-to-CPU ordering.CPU-side waits use
ID3D12Fence::SetEventOnCompletionpaired withWaitForSingleObject, orvkWaitSemaphoresorvkWaitForFences; these block the host thread until the GPU reaches the requested value.Cross-queue ordering is expressed by signaling a fence on one queue and waiting on it from another, letting compute, copy, and direct queues coordinate without CPU involvement.
In WDDM, the graphics kernel tracks monitored fences, so the scheduler can wait on them directly instead of going through user mode for queue-to-queue dependencies.
See also: D3D12 API Trace, GPU bubble, Command list and queue
Barrier#
Explicit synchronization that orders GPU work and transitions resource state when multiple operations touch the same resource.
In D3D12,
ID3D12GraphicsCommandList::ResourceBarrierissues transitions betweenD3D12_RESOURCE_STATE_*values such asRENDER_TARGET,PIXEL_SHADER_RESOURCE,COPY_DEST, andCOMMON, plus UAV and aliasing barriers.Enhanced barriers (
D3D12_BARRIER_*) decouple sync scope, access type, and layout into separate fields, allowing finer-grained ordering than legacy state transitions.Vulkan uses
vkCmdPipelineBarrierandvkCmdPipelineBarrier2to express execution barriers (stage masks), memory barriers (access masks), and image layout transitions in a single call.Image layouts (
VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL,VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL,VK_IMAGE_LAYOUT_TRANSFER_DST_OPTIMAL) describe how a resource is currently laid out in memory and must match the access about to occur.Common bubble causes include barriers inserted between back-to-back draws that did not need them, redundant transitions to
COMMON, and overly broad stage masks that drain the pipeline.
See also: GPU bubble, Vulkan command buffer, Graphics pipeline
Swap chain#
The ring of back buffers that the windowing system rotates each frame so the application can render to one image while another is displayed.
On Windows, swap chains are owned by DXGI (
IDXGISwapChain) and shared across D3D11, D3D12, and other clients; on Vulkan the equivalent isVkSwapchainKHRprovided by the WSI extension.IDXGISwapChain::Present,Present1, andvkQueuePresentKHRhand the current back buffer over for display and advance to the next image.Flip-model swap chains avoid the extra copy of the legacy blit model, and they are required for HDR and for tearing support with variable refresh rate (VRR) displays; see Present mode.
Sync interval and present flags control V-sync behavior and tearing (for example,
ALLOW_TEARING), determining whether Present blocks and how frame pacing behaves.Waitable swap chains are set up at creation time with
DXGI_SWAP_CHAIN_FLAG_FRAME_LATENCY_WAITABLE_OBJECT, not through present flags.ResizeBuffersreallocates the back buffer ring when the window changes size or format; all outstanding references must be released first.
See also: CPU-GPU frame pipelining, DXGI, Vulkan
Present mode#
How a Present call interacts with the swap chain and the display, controlling buffering, tearing, and pacing.
DXGI offers a legacy blit model and a newer flip model. With the flip model, the compositor (DWM) reads the back buffer without an extra copy, which lowers latency and makes Independent Flip possible.
DXGI swap effects include
DXGI_SWAP_EFFECT_FLIP_DISCARDandDXGI_SWAP_EFFECT_FLIP_SEQUENTIAL(flip model) and the olderDXGI_SWAP_EFFECT_DISCARDandDXGI_SWAP_EFFECT_SEQUENTIAL(blit model).Vulkan present modes include
VK_PRESENT_MODE_FIFO_KHR(vsync, no tearing),VK_PRESENT_MODE_MAILBOX_KHR(replace pending image, no tearing),VK_PRESENT_MODE_IMMEDIATE_KHR(tearing allowed), andVK_PRESENT_MODE_FIFO_RELAXED_KHR(tear only on late frames).Independent Flip bypasses the Desktop Window Manager so the swap chain image is scanned out directly, and Multi-Plane Overlay (MPO) lets the display engine composite multiple planes in hardware.
A presented frame is first queued by the runtime, then by the driver and the graphics kernel, and is then shown on the display. For how present latency differs from frame time, see Present / vkQueuePresentKHR latency vs frame time.
See also: Vertical Synchronization, Swap chain, VSync
DXGI#
DirectX Graphics Infrastructure, the Windows layer that owns adapters, outputs, factories, and swap chains, and handles presentation for Direct3D 11 and Direct3D 12.
DXGI enumerates GPUs and displays (
IDXGIFactory,IDXGIAdapter,IDXGIOutput), manages fullscreen state, and owns the swap chain thatPresentandPresent1put on screen.Nsight Systems collects the
Microsoft-Windows-DXGIETW provider whenever WDDM trace is enabled. It shows the events on the ETW rows of the process and its threads, and stores them as generic ETW events inETW_EVENTS.Present_StartandPresent_Stopbracket eachPresentcall, and the timeline pairs them into onePresentrange.PresentMultiplaneOverlay_StartandPresentMultiplaneOverlay_Stopbracket multiplane overlay presents.Events of the
SwapChaintask report swap chain creation.DXGI events are logged on the thread that calls the API, so they line up with that thread’s activity. GPU-side work shows up later as DxgKrnl queue and DMA packets and finally as a flip on the display.
See also: Swap chain, Present mode, DxgKrnl events
Render target#
The image, or set of images, that the output-merger stage writes to during a draw.
In D3D12 a render target is bound through a render-target view (RTV) with
OMSetRenderTargets, paired with an optional depth-stencil view (DSV) for the depth buffer.In Vulkan a render target is a framebuffer attachment described either by a
VkRenderPassandVkFramebufferor by the dynamic-rendering extension (vkCmdBeginRendering) which takes attachments inline.Render targets can be the swap chain back buffer, an offscreen color texture for post-processing, a depth or stencil image, or a multi-sampled image that is later resolved.
Clearing happens with
ClearRenderTargetViewandClearDepthStencilViewon D3D12, or through load operations (VK_ATTACHMENT_LOAD_OP_CLEAR) on Vulkan; load and store operations are important for bandwidth on tiled GPUs.Multiple Render Targets (MRT) lets a single pixel shader write several outputs at once, which is the basis for deferred shading G-buffers.
See also: Graphics pipeline, Swap chain
ETW#
Event Tracing for Windows is the OS-wide, high-throughput event-streaming facility built into Windows for kernel and user-mode components.
ETW follows a three-role model: controllers start and stop sessions, providers emit events, and consumers read the stream.
A session can be real-time (events delivered live to a consumer) or file-backed (events written to an ETL file for later replay).
Each provider is identified by a GUID and exposes a schema of event IDs; controllers enable it with a level and a keyword bitmask to filter what is emitted.
Keywords are bitflags that group related events by category; levels (critical, error, warning, informational, verbose) gate by severity.
Sessions own ring buffers in kernel memory and have a lifetime independent of the controller that created them, so a session can outlive its starter and must be explicitly stopped.
ETW is designed for low overhead, so it is the standard transport for OS, graphics, and driver telemetry on Windows.
See also: Custom ETW Trace, ETW provider mask, DxgKrnl events
ETW provider mask#
The provider mask is the keyword bitmask plus level that a controller passes when enabling an ETW provider; together they tell the provider which event categories and severities to emit into the session.
Each ETW provider defines its own keyword bits, where every bit names a category of events (for example, scheduler, memory, synchronization, or resource events).
A controller passes a MatchAnyKeyword mask, and the provider emits an event when at least one of its keyword bits is also set in that mask.
A MatchAllKeyword mask can further require that every set bit in the filter is present on the event; most controllers leave this zero.
The level value (critical, error, warning, informational, verbose) gates by severity: the provider emits events whose declared level is at or below the requested level.
Event-ID filters and stack-walk filters can be layered on top of the keyword and level to further narrow what the session captures.
Keyword definitions are provider-specific, so the same bit value means different things across providers; values come from the provider’s manifest or header.
See also: Custom ETW Trace, ETW
DxgKrnl events#
ETW events from Microsoft-Windows-DxgKrnl, the provider of the DirectX graphics kernel (dxgkrnl.sys). They come from the OS, not from the application, so they show what the GPU scheduler and the video memory manager did even when the application has no instrumentation.
See also: WDDM Queues, WDDM HW Scheduler, Windows GPU Memory Utilization
Collection#
Which events Nsight Systems collects depends on the WDDM trace mode; see WDDM trace modes.
WDDM trace needs administrator rights or membership in the Performance Log Users group.
With WDDM trace, Nsight Systems also collects the
Microsoft-Windows-DXGIprovider; see DXGI.Most events without a dedicated table are kept as generic ETW events (
ETW_EVENTS); some high-volume tasks are dropped.The provider GUID is
{802EC45A-1E99-4B83-9920-87C98277BA9D}.
The lists below give the ID of each event and, if it has one, its table in the SQLite export (see SQLite Schema Reference) or its timeline row.
Scheduling#
Context_StartandContext_DCStart(IDs 30 and 32) give the engine of each context, andNodeMetadata(ID 250) gives the type of each engine and the friendly name that the display driver reports. Nsight Systems exports the result toTARGET_INFO_WDDM_CONTEXTSand uses it to label engine rows (see GPU engine).QueuePacket_Start(ID 178) andQueuePacket_Stop(ID 180): a packet enters and leaves the software queue of a context, with optionalQueuePacket_Infoevents (ID 179) in between (see Queue packet). TablesWDDM_QUEUE_PACKET_START_EVENTS,WDDM_QUEUE_PACKET_INFO_EVENTS, andWDDM_QUEUE_PACKET_STOP_EVENTS. Wait and signal packets start with IDs 244 and 245 instead, which Nsight Systems collects only with the extensive trace; their start rows have packet type 4294967295, which is not a packet type.DmaPacket_Start(ID 175) andDmaPacket_Stop(ID 176): a packet starts and completes on a GPU engine, withDmaPacket_Infoevents (ID 177) that report interrupts such as preemption or a page fault (see DMA packet and SQLite Schema Event Values). TablesWDDM_DMA_PACKET_START_EVENTS,WDDM_DMA_PACKET_INFO_EVENTS, andWDDM_DMA_PACKET_STOP_EVENTS.HwQueue_Startand its rundown eventHwQueue_DCStart(IDs 422 and 424): a hardware queue was created for a context (see Hardware queue). TableWDDM_HW_QUEUE_EVENTS.HwSchedDmaPacket_BeginandHwSchedDmaPacket_End(IDs 450 and 451): a packet starts and ends on a hardware queue. Nsight Systems collects these events but does not show or export them.SchedulingLog(ID 432): a batch of entries from the hardware scheduler’s log, such as context state changes. Nsight Systems shows it as the WDDM HW Scheduler rows (see WDDM HW Scheduler).
Presentation#
Present(ID 184): a present call reached the graphics kernel. The graphics recipes use it to build frames for a process that has no Streamline or Reflex present markers.VSyncDPC(ID 17): logged once per refresh of each display output, from the vsync deferred procedure call (DPC). Nsight Systems requests it only with--wddm-memory-trace=falseand builds the VSYNC rows from it (see Vertical Synchronization).
Video memory (VidMm)#
VidMmProcessBudgetChangeandVidMmProcessUsageChange(IDs 366 and 367): the OS changed how much video memory a process may use, or the usage of the process changed. They draw the budget and usage graphs of the GPU VRAM and GPU WDDM SYSMEM rows (see Windows GPU Memory Utilization).VidMmProcessCommitmentChange(ID 371): the amount of video memory that a process has committed changed. It feeds the Committed VRAM row.VidMmProcessDemotedCommitmentChange(ID 370): the amount of process memory that was demoted from GPU local memory to non-local memory changed. It feeds the Demoted Memory row.TotalBytesResidentInSegment(ID 274): how many bytes are resident in each memory segment. It feeds the Resident Memory rows.PagingQueuePacket_Start(IDs 322 and 323),PagingQueuePacket_Info(ID 324), andPagingQueuePacket_Stop(ID 325): a packet on a VidMm paging queue. TablesWDDM_PAGING_QUEUE_PACKET_START_EVENTS,WDDM_PAGING_QUEUE_PACKET_INFO_EVENTS, andWDDM_PAGING_QUEUE_PACKET_STOP_EVENTS.The
PagingOpevents (IDs 306 to 314): single paging operations. Their names say what each one does, for examplePagingOpVirtualFill,PagingOpUpdatePageTable,PagingOpFlushTlb, andPagingOpSysmemCommit.
Allocations#
AdapterAllocation_Start,AdapterAllocation_Stop, andAdapterAllocation_DCStart(IDs 33, 34, and 35): the adapter-level record of each allocation, in any memory segment. The Resource Allocations row shows the start events (see Windows GPU Memory Utilization).DeviceAllocation_Start,DeviceAllocation_Stop, andDeviceAllocation_DCStart(IDs 36, 37, and 38): the per-device record of each allocation, which links it to the device and the process.MemoryTransfer(ID 50): a copy, fill, or discard of allocation memory. TableMEMORY_TRANSFER_EVENTS; see Memory transfer.EvictAllocation(ID 74): an allocation was evicted from GPU memory. TableWDDM_EVICT_ALLOCATION_EVENTS, and marks in the Allocation Evictions row of the process. In an extensive capture, it also shows as a mark on the allocation rows under Resource Migrations, together with the other VidMm and paging events of that allocation.AllocationFault,MarkAllocation, andPageInAllocation(IDs 71, 72, and 73): VidMm residency events for one allocation.
Practical caveats#
Volume: DxgKrnl produces many events on a busy system, and the scheduling log, paging events, and the extensive trace add many more.
Rundown:
_DCStartevents report objects that already exist when tracing starts, such as contexts, devices, hardware queues, and allocations. Nsight Systems treats them as start events. A tool that only watchesStartandStopevents sees stop events without a matching start.
WDDM#
Windows Display Driver Model is the architecture Windows uses for graphics drivers. It splits each driver between user mode and kernel mode and controls GPU access through the OS.
A WDDM driver has two halves: a user-mode driver (UMD) loaded into each graphics process, and a kernel-mode driver (KMD) loaded by the OS graphics kernel.
The OS graphics kernel (dxgkrnl) sits above the KMD and owns the video scheduler, the video memory manager, synchronization primitives, and adapter and display enumeration.
Applications submit work as command buffers to per-context queues; see WDDM Queues.
WDDM introduces the concepts of adapter, device, context, allocation, and synchronization object, all referenced by OS-level handles.
WDDM is the boundary where graphics APIs like D3D and Vulkan meet the OS, so most system-level GPU telemetry is described in WDDM terms.
See also: WDDM HW Scheduler, DxgKrnl events, VidMm and VidSch
VidMm and VidSch#
VidMm is the video memory manager and VidSch is the video scheduler, two cooperating components inside the Windows graphics kernel that manage GPU memory and GPU work execution.
VidMm tracks GPU allocations across video memory segments, system memory, and shared pools, and decides what is resident on the GPU at any moment.
VidMm enforces per-process memory budgets, evicts allocations under pressure, and issues paging operations to move data between segments.
VidSch maintains per-context queues of GPU work, picks the next packet to run on each GPU engine, and handles preemption, priorities, and quanta.
VidSch produces DMA packets from queued command buffers and signals fences when work completes.
With hardware-accelerated GPU scheduling, much of the scheduling work moves to a scheduler on the GPU; see Hardware queue.
Both components emit detailed telemetry that is the primary source of system-level GPU traces: scheduling packets, paging operations, residency changes, and memory budget changes.
See also: Windows GPU Memory Utilization, WDDM Queues, DxgKrnl events
GPU context switch#
A GPU context switch is the point at which the GPU scheduler stops running one graphics context on a GPU engine and starts another, much like a CPU thread context switch but per engine.
A graphics context represents one client’s view of a GPU engine: its command stream, residency set, and synchronization state. Many contexts compete for time on the same engine.
With hardware scheduling, the hardware scheduler records each context state change (idle, running, ready, ready-standby) in its scheduling log, which DxgKrnl reports as the
SchedulingLogevent. Nsight Systems shows the running periods as the WDDM HW Scheduler rows (see WDDM HW Scheduler).Without hardware scheduling there is no scheduling log, and context switches show only through the DMA packets that each context runs.
Frequent or long switches on the 3D engine usually mean contention between an application, the compositor, and background GPU work. To check whether another context took GPU time from the application in a slow frame, see GPU idle decomposition.
This WDDM concept is not the same as the GPU Context Switch trace in the User Guide (
--gpuctxsw; see CLI Profile Command Switch Options).
See also: Hardware queue, GPU engine, VidMm and VidSch
GPU engine#
A GPU engine is one of the hardware units that a GPU exposes for a class of work, such as 3D, Copy, Video Decode, or Video Encode.
Each engine takes its own command stream and runs largely in parallel, so a frame can rasterize on the 3D engine while a Copy engine moves a texture and Video Decode produces frames for a media surface.
Each engine has an engine type and a node ordinal. The engine types are
Other,3D,Video decode,Video encode,Video processing,Scene assembly,Copy,Overlay, andCrypto; there is no separate compute engine type. The node ordinal tells engines apart, including engines of the same type.The OS scheduler tracks utilization and preemption per engine, so a profiler can show engines that sit idle while others are busy.
D3D12 and Vulkan queues map onto engines: a Direct queue targets 3D and a Copy queue targets Copy. A Compute queue maps to whatever engine the driver reports for it. The mapping is set by the driver.
Nsight Systems labels engine rows with the engine type, such as 3D or Copy. For engines of type Other, it uses the friendly name that the display driver reports, such as Compute or Graphics_1.
See also: WDDM Queues, SQLite Schema Event Values, DMA packet
Queue packet#
A queue packet is a software-queue entry that the OS graphics scheduler creates when an application submits work to a GPU context, before the work runs on a GPU engine.
The packet sits on a per-context software queue inside the OS graphics kernel. For the packet types, see WDDM Queues.
On Windows it is bracketed by
QueuePacket_Start, optionalQueuePacket_Info, andQueuePacket_StopETW events and identified by a submission sequence number and anhContext.The time from Start to Stop is how long the packet stayed in the software queue, from submission until it retired; the matching DMA packet shows the actual engine time.
On the timeline, overlapping queue packets of a context stack up, so the stack height shows how many submissions are queued, including the one the GPU is running. With hardware scheduling, the HW Context rows show the same idea as bars: the bar height, scaled to the row’s maximum, shows the number of packets of the context that overlap at that time, and the Queue Depth value in the tooltip gives the exact count.
See also: DMA packet, Hardware queue, Command list and queue
DMA packet#
A DMA packet is a single unit of GPU work that the scheduler hands to a specific GPU engine; it is the smallest unit for which the OS reports on-GPU execution time.
The GPU scheduler turns a submitted command buffer into one or more DMA packets and hands each to an engine; the packet’s lifetime is the engine’s execution lifetime.
On Windows, each packet is bracketed by
DmaPacket_StartandDmaPacket_StopETW events, andDmaPacket_Infoevents report interrupt and page-fault status.The packet’s
ulQueueSubmitSequencematches theSubmitSequenceof the queue packet that produced it, which ties on-GPU execution back to the CPU-side submission.The DMA events do not carry an engine type. Nsight Systems tags each packet with the engine of its context and shows the packets per engine, so you can see how busy each engine really is.
DMA packets have their own packet types, which differ from the queue-packet types.
A DMA packet can be preempted or hit a fault or a page fault; the GUI shows these states in the packet caption.
With hardware-accelerated GPU scheduling, most application work is reported by other events, which Nsight Systems does not show; see Hardware queue.
See also: WDDM Queues, Queue packet, GPU engine
Hardware queue#
A hardware queue is a GPU-scheduled submission queue introduced with WDDM 2.7 hardware scheduling, in which the GPU itself rather than the OS decides when each queued packet runs on an engine.
Each hardware queue belongs to a context. Queue packet events still appear in the trace when hardware-accelerated GPU scheduling (HWS) is on.
Nsight Systems uses the
HwQueuestart and rundown events, which record the context that each hardware queue belongs to. With HWS on, queue packets name the hardware queue instead of the context, and these events map them back to the context.GPU work on a hardware queue is reported by
HwSchedDmaPacket_BeginandHwSchedDmaPacket_Endevents instead ofDmaPacket_StartandDmaPacket_Stop. Nsight Systems collects these events but does not show or export them.As a result, on systems with hardware scheduling the DMA packet tables hold mostly paging and copy work. For what this means for GPU idle checks, see hardware-accelerated GPU scheduling.
HWS moves scheduling work from the CPU to the GPU. Context state changes are then reported in the scheduling log; see GPU context switch.
See also: WDDM HW Scheduler, DMA packet, Queue packet
Memory transfer#
A WDDM copy, fill, or discard of allocation memory, mostly between system (CPU) memory and device (GPU) memory, as reported by DxgKrnl MemoryTransfer events.
In the SQLite export, the
memoryTransferTypecolumn ofMEMORY_TRANSFER_EVENTSgives the kind of transfer. Its values, listed inENUM_ETW_MEMORY_TRANSFER_TYPE, areUnknownTransfer,SystemToDevice,DeviceToSystem,AgpToDevice,DeviceToAgp,EvictToAlternateva,RestoreFromAlternateva, andDiscard. See SQLite Schema Reference.Fills are reported as
SystemToDevice. There is no device-to-device transfer kind.Transfers usually run on a copy engine, independently of the 3D engine, so they can overlap with draw work.
A transfer’s duration depends on its size and on the bandwidth of the link between system memory and the GPU.
The Memory Transfer row in the timeline shows these events by kind; see Windows GPU Memory Utilization.
See also: GPU Metrics, Resource migration, DxgKrnl events
Resource migration#
The movement of a GPU allocation from one memory segment to another, typically between dedicated VRAM and shared system memory, while the allocation stays logically alive.
On Windows, the video memory manager moves allocations from local VRAM to non-local system memory (paging across PCIe) when the working set exceeds the budget. An evicted allocation is paged back in before the GPU uses it again. A demoted allocation can stay in system memory, and the GPU then accesses it there.
Migration is not free. The bytes cross PCIe twice over an eviction-restore cycle, and the consuming draw call stalls until the resource is resident again.
In a trace, migration appears as a gap or latency spike near a draw call, often correlated with a bandwidth burst on the link and with paging activity from VidMm.
The Resource Migrations row shows each resource’s movement between VRAM and system memory; see Windows GPU Memory Utilization.
Frequent migration is a sign of budget overcommit. Fixes include reducing working-set size or adjusting priorities so that frequently used resources stay resident.
See also: Common stutter causes, Memory transfer, VidMm and VidSch
Debug marker#
A named region or label that an application records into a graphics command stream so tools can show what each block of GPU work represents.
A begin and end pair brackets a run of commands:
PIXBeginEventandPIXEndEventin D3D11 and D3D12, andvkCmdDebugMarkerBeginEXTandvkCmdDebugMarkerEndEXT(or the newervkCmdBeginDebugUtilsLabelEXTandvkCmdEndDebugUtilsLabelEXT) in Vulkan.A point marker has no duration and is used for one-off annotations, exposed as
PIXSetMarkerandvkCmdInsertDebugUtilsLabelEXT.Markers carry a short string and sometimes a color. Queues can be labeled too, to mark submissions rather than recorded commands: the PIX calls also accept a D3D12 command queue, and Vulkan has separate calls such as
vkQueueBeginDebugUtilsLabelEXT.Begin and end must nest correctly. In Vulkan, a label region can begin in one command buffer and end in a later one on the same queue.
They complement NVTX: NVTX describes CPU-side phases, while debug markers describe GPU-side regions in the command stream.
Profilers match begin and end by nesting depth and render a labeled hierarchy aligned with draw calls, dispatches, and submissions.
See also: D3D12 API Trace, NVTX Trace, Vulkan command buffer