User Guide

The user guide for NVIDIA Nsight Graphics.

Introduction to NVIDIA Nsight Graphics

Nsight Graphics™ is a standalone application for the debugging, profiling, and analysis of graphics applications. Nsight Graphics supports applications built with DirectCompute, Direct3D (11, 12), OpenGL, Vulkan, Oculus SDK, and OpenVR.

This documentation is separated up into different sections to help you understand how to get started using the software, understand activities, and offer a reference on the user interface.

  • Getting Started - Offers a brief introduction on how to use the tools.

  • Activities - Nsight Graphics Supports multiple activities to target your workload to the need of your work at a particular point in time. This section documents each of these activities in detail.

  • User Interface Reference - Provides a deep view of all of the user interface elements and views that Nsight Graphics offers.

  • Appendix - Contains a selection of topics on concerns not covered by any other section.

Getting Started

This section describes an approach to using the Nsight Graphics tools.

Expected Workflow

When debugging or profiling, it is important to narrow your investigation to the path that provides the most impactful and actionable data for you to make conclusions and solve problems. Nsight Graphics provides a number of tools to fit each of these workflow scenarios.

When debugging a rendering problem, Nsight Graphics’s Frame Debugger is the tool of choice. This tool enables the inspection of events, API state, resource values, and dependencies to understand where your application might have issues. For more information on the Frame Debugger, see Frame Debugger.

When profiling a graphical application, the first step is to determine if you are CPU or GPU bound. If you are CPU bound, you will not be able to issue enough work to the GPU to take full advantage of its full processing power. If you are GPU bound, the GPU is not able to process the work it is issued fast enough and your engine may stall. One way of making the determination of which aspect is limiting you is to use Nsight Systems™. Nsight Systems is a system-wide performance analysis tool designed to visualize an application’s algorithms, help you select the largest opportunities to optimize, and tune to scale efficiently across any quantity of CPUs and GPUs in your computer. NVIDIA also provides a system analysis and trace tool within Nsight Visual Studio Edition; for more information on that tool see this site.

If you have determined that you are CPU bound, you will want to use a CPU profiling tool to discover how you can eliminate inefficiencies to issue work faster to the GPU. You may also want to look into the overhead of the API constructs you are using and determine if there are more lighter weight constructs that can offer the same effect at less cost. The Frame Debugger tool is an excellent resource while you are making these adjustments to your engine.

The GPU Trace activity within Nsight Graphics allows for analysis of a few different GPU bound scenarios. GPU Trace offers a deep analysis of your SM’s performance by tracing the execution of your shaders on the SM across a series of frames. Another key technique in optimizing performance is to take advantage of the GPUs ability to process parallel work by using techniques to achieve simultaneous compute and graphics (SCG), also known as async compute. GPU Trace allows you to both see opportunities for async compute as well as to confirm and measure the impact of async compute on your frame.

How to Launch and Connect to Your Application

To analyze an application, Nsight Graphics requires the launching of applications through its launching facilities. The sections below describe creating a project, launching the application, and connecting to it so that you can perform your analysis.

Upon starting Nsight Graphics, you are presented with the option to create a project. If you are using Nsight Graphics for the first time, skip project creation by selecting continue under Start Activity.

../_images/nsight_graphics_welcome_document.png

Once selected, you will be presented with a target-specific dialog that allows you to configure the application to launch. Browse and select the activity you wish to run and then proceed to the target-specific instructions below to configure the application to analyze.

../_images/frame_debugger_connect.01.png

The target-specific sections below describe how to launch and connect on each specific platform. While the process may be different on different targets, there are many commonalities between all systems. In particular, once a process is launched, the Nsight host must attach to that process in order to analyze it. This logical separation of launch and attach facilities allows for complex use cases including remote targets, launching though command lines, reattaching to previous sessions, etc. The Nsight host does simplify many common cases, however, by supporting user-controlled automatic connection to processes that were just launched. The sections below cover these uses cases and more, in turn.

Process Launch and Connection on Windows Targets

On the Windows platform, Nsight Graphics supports debugging and profiling of x64 native applications as well as x86 (32-bit) launchers (e.g. Steam, Origin) that start x64 (64-bit) applications.

Launching an Application with Automatic Attach

Nsight Graphics supports automatic attach to processes of interest. It accomplishes this by identifying the processes in a process hierarchy that perform graphics work, signaling that these are of interest.

To launch your application, perform the following steps.

  1. Set the application executable to the path of your application. This path may be any executable or batch file.

  2. If your application requires a working directory that is different from your application’s directory, adjust it now.

  3. Adjust the environment (if necessary).

  4. Leave Automatically Connect as Yes.

  5. Click Launch.

Once launched, you will be presented with a dialog that notes the launching and attaching of your application. After the launch completes, you are ready to begin your analysis.

Connecting to an Application with Manual Attach

There may be some cases where a manual attach to an application is desired. These situations include:

  • Using the command line launcher to launch applications (see Process Launch from a Command Line)

  • Automatic attach is attaching to an application other than the one you want

  • Connecting to an application that has previously been detached and reattach to the analysis session is desired

If the application is already launched, perform the following steps:

  1. Click the Connect button.

  2. Select the Attach tab.

  3. Select the application you wish to analyze in the attach tab and click Attach.

After the launch completes you are ready to begin your analysis.

../_images/frame_debugger_attach.02.png

In the example image above, VRFunhouse.exe is a child process of the UE4Game.exe launcher. Selecting VRFunhouse.exe and clicking Attach would allow you to analyze the primary application.

Remote Launching

Remote debugging is supported on Nsight Graphics on Windows through use of the Nsight Remote Monitor. This is a process that runs on a target machine to allow connections to be started on that machine.

To run the remote monitor, install Nsight Graphics on the target machine. Then, launch the remote monitor on that machine by Start > NVIDIA Corporation > Nsight Remote Monitor.

Once the monitor is launched on the remote machine, you need to add the remote monitor as a connection in Nsight Graphics. By default, launches will be done on the localhost machine. To add another machine, click the + button.

../_images/remote_debugging_configure_connection.png

This brings up a dialog in which you can add a machine name or IP address.

../_images/remote_debugging_add_remote_connection.png

Enter the machine name in IP/Host Name. Click Add to add the connection. The machine you just added will be listed as the target connection at this time.

../_images/remote_debugging_configure_connection_added.png

Any number of connections may be added; connections can be removed by clicking - on the selected connection. The connections may be switched between any of the added connections before launch or attach. Connections are globally persisted and may be applied to any project once they are added.

Process Launch and Connection on Linux Targets

Remote debugging on Linux is supported through SSH connections. Enter your SSH information when establishing the connection to connect to the target machine.

Process Launch from a Command Line

Nsight Graphics offers a command line interface (CLI) to facilitate launching on applications for which the environment setup can be complex to transfer to the host application. Currently it also provides a non-interactive way to Generate C++ Capture from a Command Line or Generate GPU Trace Capture from a Command Line).

This executable is located in the host application folder:

Windows

<install directory>/host/windows-desktop-nomad-x64/ngfx.exe

Linux

<install directory>/host/linux-desktop-nomad-x64/ngfx.bin

Note

The original command line launcher (nv-nsight-launcher) has been replaced by this CLI (ngfx), because this CLI extends the original launcher’s capabilities.

CLI Arguments Details

To understand how to launch, start by launching the CLI with the --help argument. This will display what a general options the CLI has.

Several of these arguments are optional, while some are required. The full argument list is the following:

Table 1. CLI General Options

Option

Description

--help

Display all available general arguments.

--help-all

Display all available arguments, including all activity specific arguments.

--activity arg

Select the target activity to use.

This argument is always required.

Note: “activity name” must be exact name in “Activity” section of connection dialog. The available activity names can also be found in the help message (--help).

--platform arg

Select the target platform to use.

This argument is recommended to be set, otherwise it’ll use the local platform as a default.

Note: “platform name” must be exact name in “Target Platform” section of connection dialog. The available platform names can also be found in the help message (--help).

--hostname arg

Select the device on which to launch the application.

Default value is “localhost”.

This argument is required if trying to launch application in a machine with nv-nsight-remote-monitor running.

--exe arg

Set the executable path on target device.

NOTE: This argument is usually required, but can be implicitly deduced from project settings if a project has been loaded (see --project).

--dir arg

Set the working directory of the application to be launched.

--env arg

Set the additional environments of the application to be launched.

These arguments should be in the form of “FOO=1; BAR=0;”.

--args arg

Set the arguments passed to the application to be launched.

--project arg

Select an Nsight Graphics project to load.

If a project has been successfully loaded, some arguments (e.g., --exe) can be implicitly deduced from the loaded project settings if they are not specified.

If there’s a dedicated project for a certain application, and there are changed and saved options for adjusting the activity, it is preferred to run the CLI with this argument.

--output-dir arg

Set output folder to export/write data to.

If not specified, it’ll use the default document folder on Nsight Graphics GUI

--verbose

Enable verbose mode to display more messages.

--no-timeout

By default operations (e.g., launch) are bound to proper timeouts, disable timeouts if some applications can take a long time to perform operations.

NOTE: This argument is not used for simply launching the target application.

--launch-detached

Run the CLI as a command line launcher; the CLI exits after launching the target application. You may attach to the application with the Nsight Host after it has been launched.

There are also activity-specific options beyond the general options above. For examples on launching with a specific activity, as well as referencing these activity specific options, see the follow sections:

If you wish to simply launch an application, without automatically performing a capture, the CLI can launch an application with commands in the form of the below command.

ngfx.exe --launch-detached [general_options]

Examples:

  • ngfx.exe --activity="Frame Debugger" --platform="Windows"
    --project="D:\Projects\Bloom.ngfx-proj" --launch-detached
    

    Launch an application in the local host, with using the launch options and activity options read from a Nsight Graphics project.

  • ngfx.exe --activity="Frame Debugger" --platform="Windows"
    --hostname=192.168.1.10 --exe=D:\Bloom\bloom.exe --launch-detached
    

    Launch an application on a remote machine.

Using Nsight Graphics With WebGL

Nsight Graphics has the ability to debug and profile WebGL applications running in a browser. However, there are some setup steps that must be taken to ensure compatiblity and provide as much API contextual information as possible. This is important because many browsers, especially on Microsoft Windows, will use graphics APIs other than OpenGL for the backend rendering. Because of this, when you attempt to debug your application, you may see some DirectX calls that won’t map easily to the WebGL calls you originally made. Using the correct settings can force an OpenGL backend, which will look more similar to what you are expecting and aid in debugability.

To get started, make sure you close out all browser windows. This is necessary due to the method that Nsight Graphics uses for injecting code into your application. Specifically, Chrome will tend to launch a child process from one of the other, already running processes. If that initial process is not injected, then the child process will not be either, even if you launched it from the Nsight Graphics Connection dialog. So, close out any browser windows and use Task Manager to ensure that no Google Chrome processes (or other browser processes) remain.

Next, you will want to set the browser to use the OpenGL backend. In Chrome, you need to:

  1. Type chrome://flags/ in the address bar to bring up the settings

  2. Search the settings for OpenGL

  3. For the ANGLE graphics backend, choose OpenGL

Finally, browsers typically need additional settings to ensure tool compatibility. For Google Chrome, you need so specify the following additional command line options when launching the browser:

Table 2. Chrome Options For WebGL

Option

Description

–no-sandbox

Disable some of the Chrome security checks to allow process injection to work.

–disable-gpu-watchdog

Disable the Chrome GPU activity check. This allows for the application to be paused live and not have Chrome exit.

–disable-features=RendererCodeIntegrity

Disables the renderer code integrity feature that interferes with graphics overrides from Nsight Graphics.

–gpu-startup-dialog

Optional: This flag will cause a dialog to display on launch of the graphics process, to help find the process you want to debug. Note that Nsight Graphics can typically find the process without resorting to manual intervention.

Automatic Cleanup of Launcher Processes

There are some processes that have the potential of interfering with the launch of an analysis session of your application. These processes are typically long-lived application launchers – they often perform a coordinated launch between child process and the parent launcher. In some cases, this coordinated launch can interfere the process by which Nsight Graphics injects itself with the approach analysis settings.

To mitigate this problem, Nsight Graphics attempts to detect processes that are known to interfere and to offer the user an opportunity (dialog shown below) to terminate the processes before launch, thereby allowing a launch without interference.

../_images/launcher_detection.PNG

The buttons on the dialog perform the following actions:

  • Yes - terminate the processes and continue launching

  • No - do not terminate the processes, but continue to launch the application

  • Abort - cancel the launch entirely

To edit the list of processes that are detected, add an entry to the list in Tools > Options > Injection.

../_images/launch_interfering_processes.PNG

After a Process is Connected

After a process is connected, it is ready to be analyzed. For many activities, a default set of windows will come up that offer an impactful set of tools for analysis that pertains to the activity. You can also add additional windows to the application by selecting a view from the menu bar. See the User Interface Reference for a detailed discussion of each view and tool window.

For the Frame Debugger activity that was started above, there are both live analysis and capture utilities. When capturing from this activity, done though the Target application capture hotkey or “Capture for Live Analysis”, a number of views will open. On the target application, the HUD will appear with the toolbar and scrubber. This UI allows you to view an exhaustive amount of information on the state, resources, and synchronization of your application.

../_images/perf_dashboard_capture_frame.01.png ../_images/frame_debugger_capture_sidebyside.02.png

With such an expansive set of information available, debugging a rendering problem is made easier.

Choose Window To Debug

If your application has multiple windows, each with a graphics context, you can choose the window that you want to focus your debugging and profiling efforts on. For instance, if your application has 3 different windows, each with it’s own view of the scene, you can easily select the window of interest to capture. When multiple windows are detected, there will be a new “Select Window” control on the top toolbar. The combo box to the right will be populated with the detected windows, including the window name and size to aid in determing the correct window.

../_images/choose_window_to_debug.png

Any time that you are not currently in an active Capture/Replay, you can use the control to specify the window to capture. If you select “Default”, the most recent window to present the back buffer will be selected as the window to capture. This option is available with the Frame Debugger and Generate C++ Capture activities.

Target application capture hotkey

Activities support triggering capture from the Nsight Graphics UI or directly from the target application. The default capture hotkey is F11. This may be configured also in Tools > Options.

../_images/nomad_capture_hotkey.png

Configuring Your System for Optimal Analysis

In order for your system to work well with the analysis tools provided by Nsight Graphics, there are a number of details you should consider.

Configure Developer Mode

If using a Windows machine, we recommend running under Developer Mode.

Reasons to use Developer mode include:

  • Preview versions of the D3D12 Agility SDK require developer mode.

  • Some applications can only replay with the D3D12 preview Agility SDK.

Please follow the instructions at https://learn.microsoft.com/en-us/windows/apps/get-started/developer-mode-features-and-debugging for setting up developer mode.

Configuring Your Application for Optimal Analysis

In order for your application to work well with the analysis tools provided by Nsight Graphics, there are a number of details you should consider when configuring your application.

Using Performance Markers

Performance markers are integral to nearly all workflows. We recommend that your application always run with perf markers when running under tools analysis.

Performance markers are most commonly used to delineate sections of events and note where in your application they begin and end. They can also be nested to show sub-sections of events. Perf markers are generally used to measure the amount of time that an inner portion of algorithm will take.

There are multiple different types of perf markers that are supported in Nsight Graphics:

  • D3D9 perf markers are supported for all D3D applications.

  • Vulkan applications may use either VK_EXT_debug_utils or VK_EXT_debug_marker.

  • OpenGL applications use the KHR_debug group, glPushDebugGroup and glPopDebugGroup.

Shader Compilation

Nsight Graphics works best when you have the full shader source available for debugging. Follow the steps below to set up your application for optimal configuration.

D3D Configuration

Nsight Graphics works best with access to the original HLSL source code of your shaders. There are a few ways to accomplish this task. The first is to precompile the shaders into binary format using the using one of the legacy D3DCompile functions or the latest IDxcCompiler interfaces and saving the results out to a file.

Alternatively, you can use the offline compiler, fxc.exe or dxc.exe, provided by the DirectX SDK.

For each of these methods, you need to specify some flags in order for the HLSL debug information to be embedded in the binary output, outlined below:

Compile type

Required actions

Shaders compiled offline via dxc.exe or fxc.exe

Add "-Zi" option to the command line to get full debug info, such as linetable info required for shader profiling.

For dynamic shader editing, you can retrieve a minimal set of debug info using the "-Zs" option to the pArguments parameter.

Shaders compiled online via IDxcCompiler3::Compile

Add "-Zi" option to the pArguments parameter to get full debug info, such as linetable info required for shader profiling.

For dynamic shader editing, you can retrieve a minimal set of debug info using the "-Zs" option to the pArguments parameter.

Note that earlier versions of the Compile function, including IDxcCompiler::Compile and IDxcCompiler2::CompileWithDebug, are deprecated. Microsoft recommends use of IDxcCompiler3::Compile as the alternative.

Shaders compiled online via D3DCompile, D3DCompile2, D3DCompileFromFile

Add the D3DCOMPILE_DEBUG flag to the Flags1 parameter

Nsight Graphics also supports reading debug info from files that have been generated using the dxc.exe -Fd option. To load these external files, the user will need to set the appropriate path(s) in the Compiled Shader Symbol Paths section of the <xref href=”global_options_search_paths.dita”/>

Vulkan Configuration

Nsight Graphics works best with access to the original high-level source code of your shaders. To accomplish this, shaders need to be compiled with debug information in order for the original high-level source code to be embedded in the SPIR-V binary modules.

When using the glslangValidator tool, add -g flag to the shader compilation command line. For example:

glslangValidator -V shader.vert -o shader.spv -g

When using the dxc tool, add -Zi flag to the shader compilation command line. For example:

dxc -spirv -T ps_6_5 -E PSMain shader.frag -Fo shader.spv -Zi

Naming Objects and Threads

Many of Nsight Graphics’s views and analysis benefits from naming API objects and threads. Similar to perf markers, these names can help offer increased context for your analysis. The tables below list the supported methods for naming objects and threads.

Naming Objects

D3D11

No programmatic method; use Nsight-generated names

D3D12

ID3D12Object::SetName

OpenGL

glObjectLabel

Vulkan

vkDebugMarkerSetObjectNameEXT or vkSetDebugUtilsObjectNameEXT

Naming Threads

Platform

Method

Windows

SetThreadNameDescription

Linux

Not yet supported

How To Setup and Inspect GPU Crash Dumps

This section describes how to use the NVIDIA Nsight Aftermath Monitor to generate GPU crash dumps for applications using the Direct3D 12 or Vulkan API, and how to open and inspect those GPU crash dumps with the crash dump inspector plug-in in Nsight Graphics.

Alternatively, developers can also add GPU crash dump collection support into their graphics application using the NVIDIA Nsight Aftermath SDK.

Workflow

The general workflow for working with Nsight Aftermath GPU crash dumps is to:

  1. Run the NVIDIA Nsight Aftermath GPU Crash Dump Monitor.

  2. Configure the desired GPU crash dump features.

  3. Optional: if you want to collect additional information via event markers, you can optionally instrument the graphics application using the Aftermath SDK.

  4. Run the graphics application for which to capture GPU crash dumps and reproduce the GPU crash or hang, allowing the monitor to collect the GPU crash dump.

  5. Open the GPU crash dump in Nsight Graphics.

  6. Configure GPU Crash Dump Inspector settings.

  7. Inspect the crash dump data using the Nsight Graphics crash dump inspector.

See the sections below for details on each step of this process.

The GPU Crash Dump Monitor

The NVIDIA Nsight Aftermath Crash Dump Monitor provides the means to capture GPU crash dump files for GPU crashes or GPU hangs, and to modify the driver configuration settings related to crash dump generation.

Running the GPU Crash Dump Monitor

The NVIDIA Nsight Aftermath Monitor nv-aftermath-monitor.exe is installed to the Nsight Graphics host directory. Typically this is:

<install directory>\host\windows-desktop-nomad-x64

The crash dump monitor application will by default start in the background. Its user interface is accessible through the NVIDA Nsight Aftermath Monitor icon in the Microsoft Windows system notification area (system tray).

Configuring the GPU Crash Dump Monitor

All configuration options related to GPU crash dump creation are available through the GPU Crash Dump Monitor Settings dialog.

  • Set up directory where crash dump files are stored.

  • Set up directory where shader debug information files are stored.

  • Enable Aftermath GPU Crash Dump collection. Either set Aftermath mode to Global to enable crash dumps for all applications using the D3D12 or Vulkan API or selectively enable it for one or more applications by managing an application Whitelist.

  • Enable the desired Aftermath graphics driver features:

    • Generate Shader Debug Information to generate shader debug information (line tables for mapping from the shader IL passed to the NVIDIA graphics driver to the shader microcode executed by the GPU) for all shaders loaded by the application.

      The GPU Crash Dump Monitor will store the debug information into files with the .nvdbg extension in the Debug Info Dump Directory directory configured in the General Settings Tab of the GPU Crash Dump Monitor Settings.

      The shader debug information is required for mapping shader microcode instructions of active or faulted shader warps to shader IL or shader source lines.

      See also the section about Source Shader Debug Information for details on how to compile shader source with source-level debug information.

    • Enable Resource Tracking to enable additional driver-side tracking of live and recently destroyed resources.

      This allows Aftermath to identify resources related to GPU virtual addresses seen in the case of a crash due to a GPU page fault. The resource information being tracked includes details about the size of the resource, its format, and the current deletion status of the resource object. D3D12 developers may also consider instrumenting the application using the GFSDK_Aftermath_DX12_RegisterResource function to register the D3D12 resources the application creates. That will allow Aftermath to track additional information, such as the resources debug names set by the application. For Vulkan applications, the resources debug names set via vkSetDebugUtilsObjectNameEXT will be captured, too.

    • Enable Call Stack Capturing to enable automatic generation of Aftermath event markers for tracking the origin of all draw calls, compute and ray tracing dispatches, ray tracing acceleration structure build operations, or resource copies initiated by the application.

      The automatic event markers are added into the command stream right after the corresponding commands with the CPU call stacks of the functions recording the commands as the data payloads.

      Note

      Enabling this feature will cause considerable driver overhead for gathering the necessary information.

      Note

      When this feature is enabled the GPU crash dump file may contain the file path for the crashing application’s executable as well as the file paths for all DLLs or DSOs it has loaded.

    • Enable Shader Error Reporting to enable a special mode that allows the GPU to report additional runtime shader errors. This may provide additional information when debugging hangs, crashes or unexpected behavior related to shader execution.

      Enabling this feature may result in additional crash dumps reporting issues in shaders that exhibit undefined behavior or have hidden bugs, which so far went unnoticed because by default the HW silently ignores them. The additional error checks that are enabled when using this option will cause GPU exceptions for the following situations:

      • Accessing memory using misaligned addresses, such as reading or writing a byte address that is not a multiple of the access size.

      • Accessing memory out-of-bounds, such as reading or writing beyond the declared bounds of (group) shared or thread local memory or reading from an out-of-bounds constant buffer address.

      • Accessing a texture with incompatible format or memory layout.

      • Hitting call stack limits.

        Note

        This feature is only supported with NVIDIA graphics driver R515 or later.

      Note

      On Windows, modifying Aftermath graphics driver settings requires Windows Administrator privileges. Therefore, when any of these settings are modified and applied, a User Account Control confirmation window may pop up asking for permission to modify system settings.

  • Enable the desired Aftermath systemwide features:

    • Enable SM Register Data Collection to collect SM register values when faults happen inside SMs. This can provide additional information when debugging GPU crashes related to shader execution.

      Since this is a system setting, modifying it might also affect other tools such as Nsight VSE CUDA debugger and may result in unexpected behavior! On Linux, this feature is always enabled without incompability with other tools.

      Note

      This feature is only supported for the D3D12 and Vulkan APIs with NVIDIA graphics driver R535 or later and requires Nsight Graphics Pro to visualize the data.

    Note

    On Windows, modifying Aftermath system settings requires Windows Administrator privileges. Therefore, when any of these settings are modified and applied, a User Account Control confirmation window may pop up asking for permission to modify system settings.

The GPU Crash Dump Inspector

The NVIDIA Nsight Aftermath Crash Dump Inspector provides the means to open, inspect, and analyze GPU crash dump files created by the NVIDIA Nsight Aftermath Monitor.

Loading GPU Crash Dump Files

GPU crash dump files use the .nv-gpudmp file extension and can be loaded through File > Open File... This will bring up a GPU Crash Dump Inspector window displaying the crash dump file’s content.

Configuring the GPU Crash Dump Inspector

In order to use all functionality provided by the GPU crash dump inspector, the following configuration settings should be made in the Search Paths Settings.

  • Add the directories where binary shader files (DXIL or SPIR-V shader files) are stored to Shader Binaries. If the binary shaders cannot be found, the Shader View will not be able to display intermediate shader assembly code or shader source code.

    For more information on how to generate these files, see Source Shader Debug Information.

  • Add the directories where the separate shader debug information files (.lld or .pdb files generated by dxc.exe for instance) are stored to Separate Shader Debug Information. If the shader debug information cannot be found, the Shader View will not be able to map GPU PC addresses of active or faulted warps to intermediate shader assembly or shader source code locations.

    For more information on how to generate these files, see Source Shader Debug Information.

  • Add the directories where the NVIDIA shader debug information files generated by the GPU crash dump monitor are stored to NVIDIA Shader Debug Information. If the NVIDIA shader debug information cannot be found, the Shader View will not be able to map GPU PC addresses of active or faulted warps to intermediate shader assembly or shader source code locations.

  • Optionally, add the directories where shader source files are stored to Shader Source. Usually, this is not required as the shader debug information already includes the shader source. Only if a shader was compiled from source that contains references to other source files, for example via #line directives, may it be necessary to specify additional source directories so that the Shader View will be able to find the correct shader source.

  • Add the directories where to find the symbol files for the graphics application for which the GPU crash dump has been captured to Application Debug Information. This allows the Aftermath Marker Call Stack View to resolve addresses to functions and source locations.

Inspecting GPU Crash Dump Files

Use the GPU Crash Dump Inspector to analyze crash reasons. This is not an exhaustive tutorial on how to analyze GPU crash dumps, because every crash or hang is different, but it should provide some hints to get started.

After loading a crash dump file, it is usually a good start to check the Exception Summary on the Dump Info tab. This will show a high-level fault reason, e.g. whether the graphics device was hung or an error like a page fault has occurred. If there was a page fault or shader fault, this section will contain an analysis which mentions potential causes and provides links to relevant information in the Crash Info tab and Shader View.

In case of a hang it makes sense to check if there is an Active Warps section on the Dump Info tab showing shader activity. This could point towards an issue with very long running shader warps or shader warps being stuck in an infinite loop. In that case the Shader View may help to root cause the problem.

If the device state indicates there was a memory fault, the next step would be to look for a Page Fault section on the Dump Info tab. This may help to pin point problems with out-of-bounds resource access or accessing an already deleted resource.

If the application was instrumented with Aftermath Event markers, a Aftermath Markers section should be available on the Dump Info tab. This may help to pin point the draw or dispatch call that caused problems.

If Call Stack Capturing was enabled when capturing the GPU crash dump, Call Stack links should be available in the Aftermath Markers section, pointing to the draw, dispatch, or copy call that may be related to the problem.

Last, the GPU State section on the Dump Info tab may provide some hints about which parts of the graphics pipeline were active or have faulted when the crash occurred.

Instrumenting Applications With The Aftermath API

The NVIDIA Nsight Aftermath SDK provides the Aftermath API that can be used by developers to instrument their applications. The latest version can be downloaded from https://developer.nvidia.com/nsight-aftermath.

By default, the the latest version of the SDK package available at the time of a Nsight Graphics release is installed together with Nsight Graphics in:

<install directory>\SDKs\NsightAftermathSDK

Detailed information about the functionality provided by the library and how to use it in an application can be found in the Readme.md that comes with the SDK package and the header files.

Aftermath Event Markers

In D3D applications, the Aftermath event marker API (GFSDK_Aftermath_SetEventMarker) can be used to inject event markers with user defined data directly into the graphics command stream. If the application is instrumented with event markers, information about the last event markers that were processed by the GPU for each command stream will be captured into the GPU crash dump, including the user provided event data.

NOTE: Using event markers should be considered carefully. Injecting markers in high-frequency code paths can introduce high CPU overhead. Therefore, on some driver versions, the event marker feature is only available if the Nsight Aftermath GPU Crash Dump Monitor is running on the system. This requirement applies to R495 to R530 drivers for DX12 and R495+ drivers for DX11. No Aftermath configuration needs to be made in the Monitor. It serves only as a dongle to ensure Aftermath event markers do not impact application performance on end user systems.

Similar functionality is available for Vulkan applications with the VK_NV_device_diagnostic_checkpoints extension.

Source Shader Debug Information

For mapping shader instruction addresses for active or faulted shader warps to high-level shader source, shaders need to be compiled with debug information. Since shader compilation is a two-step process - compilation from shader source, such as HLSL or GLSL, to an intermediate shader language representation, such as DXIL or SPIR-V, and graphics driver-level compilation of the shader IL to the actual microcode executed by the GPU - there are two levels of debug information required to accomplish such a mapping. This section describes how to compile shader source code with debug information suitable for consumption by the Aftermath GPU Crash Dump Inspector using the Microsoft DirectX Shader Compiler or the Vulkan SDK toolchain for shader compilation.

The generation of shader debug information for the microcode level needs to be enabled either through the Nsight Aftermath GPU Crash Dump Monitor settings or Aftermath feature flags when using the Nsight Aftermath SDK.

To enable shader instruction mapping when analyzing crash dumps in Nsight Graphics, the debug information must be made available by setting the Search Path Settings as described in Configuring the GPU Crash Dump Inspector.

For D3D12, the following variants of compiling shaders with debug information using the Microsoft DirectX Shader Compiler (dxc.exe) are supported by Aftermath:

  1. Compile and use full shader blobs: Compile the shaders with debug information. Use the full (i.e., not stripped) shader binaries when running the application and make them accessible to Nsight Graphics when inspecting GPU crash dumps by adding the disk location where the compilation results are stored to the Shader Binaries search paths.

    An example command line may look like this:

    dxc -Zi [..] -Fo shader.bin shader.hlsl
    
  2. Compile and strip: Compile the shaders with debug information, then strip off the debug information. Use the stripped shader binaries when running the application and make both stripped and not stripped files accessible to Nsight Graphics when inspecting GPU crash dumps. Add the disk location of the stripped files to the Shader Binaries search path and add the disk location of the not stripped files to the Separate Shader Debug Information search paths.

    An example command line may look like this:

    dxc -Zi [..] -Fo full_shader.bin shader.hlsl
    dxc -dumpbin -Qstrip_debug -Fo shader.bin full_shader.bin
    
  3. Compile with separate debug information: Compile the shaders with debug information and instruct the compiler to store the debug meta data in separate shader debug information files (shader PDB files). Make both the shader binaries and the shader debug information files accessible to Nsight Graphics when inspecting GPU crash dumps. Add the disk location of the shader binaries to the Shader Binaries search path and add the disk location of the shader debug information files to the Separate Shader Debug Information search paths.

    An example command line may look like this:

    dxc -Zi [..] -Fo shader.bin -Fd debugInfo\ shader.hlsl
    

If the application compiles shaders on-the-fly, it needs to store the shader binary blobs to disk in a similar fashion so that they are accessible to Nsight Graphics when inspecting GPU crash dumps.

Note

No IL-level or source-level shader mapping is supported for DX bytecode shaders generated by the legacy Microsoft DirectX fxc.exe shader compiler.

For Vulkan, the following variants of generating SPIR-V shader code with debug information are supported by Aftermath:

  1. Compilation using the glslangValidator tool of the Vulkan SDK’s shader compilation toolchain. An example command line may look like this:

    glslangValidator -V -g -o shader.spv shader.vert
    
  2. Compilation using the Microsoft DirectX Shader Compiler. An example command line may look like this:

    dxc -spirv -Zi [..] -Fo shader.spv shader.hlsl
    

Use the full (i.e., not stripped) SPIR-V shader binaries when running the application and make them accessible to Nsight Graphics when inspecting GPU crash dumps by adding the disk location where they are stored to the Shader Binaries search paths.

Note

No source-level shader mapping is supported for pairs of stripped and not stripped SPIR-V files. Users interested in shader source mapping for applications shipping with stripped SPIR-V shaders may use the GPU crash dump decoding functionality provided by the Nsight Aftermath SDK and implement their own crash dump decoding tool.

Samples

Nsight Graphics ships with a selection of samples that can help you understand how to use the tool. To access these samples, select Help > Samples and choose a sample to open. Upon selection, a sample project will be created and the connection dialog will come up, allowing you to work with and example the sample. Some samples also include example reports for you to study.

Note that samples are also an effective tool for disambiguating Nsight Graphics problems from system problems. If the samples do not run for you without issue, there is likely a system problem that needs diagnosing.

Activities

Nsight Graphics Supports multiple activities to target your workload to the need of your work at a particular point in your development process.

  • Frame Debugger - allows you debug a frame by each draw call. You can view vertex shaders, pixel shaders, and pipeline states.

  • Generate C++ Capture - The C++ Capture activity allows you to export an application frame as C++ code to be compiled and run as a self-contained application for later analysis, debugging, profiling, regression testing, and edit-and-compile experimentation a frame by each draw call. You can view vertex shaders, pixel shaders, and pipeline states.

Frame Debugger

The Frame Debugger activity allows for:

  • Real-time examination of rendering calls;

  • Interactive examination of GPU pipeline state, including visualization of bound textures, geometry and unordered access views;

  • Pixel History shows all operations that affect a given pixel;

  • Shader profiling to explore shader performance;

  • C++ Capture exports for offline collaboration and analysis.

When to use the Frame Debugger Activity

The Frame Debugger activity offers a comprehensive set of tools for discovering problems with your application’s rendering or general operation. This activity enables the inspection of events, API state, resource values, and dependencies to understand where your application might have issues. Use this activity when:

  • You have a render-accuracy issue

  • You expect that you may have a synchronization issue

  • You want to explore the performance of your application shaders (DX12 and Vulkan)

The Frame Debugger activity supports all APIs that are generally supported by Nsight Graphics.

Basic Workflow

To start this activity, select Frame Debugger from the connection dialog.

../_images/activity_list_frame_debugger.png

The basic workflow for the Frame Debugger activity is to capture an application and then navigate the events, data, and resources that your application is submitting/using to identify your issue.

Whether you are debugging on the CPU or GPU, the first step of any debugging process is to narrow in on the set of data that you need to analyze to understand your problem. Generally, this means that you will want to scrub to a particular event of interest in either the Scrubber or the Event Viewer. Because Nsight Graphics™ will show you the rendering contribution of every draw call, looking at either the HUD or the Current Target View will give you an indication of where your rendering might be going wrong. Another alternative is to use the Pixel History experiment to automatically identify the draw calls that relate to a particular texture update.

From there, you will want to use your knowledge of the graphics pipeline to try to understand what might be causing a problem. Some questions to ask yourself:

  • Is this a geometry problem? If so, is it a pre-transform or post-transform problem?

  • Is this a blending problem?

  • Is this a synchronization problem?

In some cases, there may be a combination of problems that exacerbate a given problem. Isolating the symptoms can be challenging, but an effective use of the tools can offer increased confidence that you are heading in the right direction.

Generate C++ Capture

The C++ Capture activity allows you to export an application frame as C++ code to be compiled and run as a self-contained application for later analysis, debugging, profiling, regression testing, and edit-and-compile experimentation.

When to Use the Generate C++ Capture Activity

While C++ captures can be collected in while Frame Debugging, the C++ capture activity provides a focused activity to streamline the creation of captures. Non-necessary analysis subsystems are turned off in order to allow for the quickest and more robust application capture. This activity is an excellent way to save a snapshot of your application, frozen in time. Use this activity when:

  • You want to save a deterministic application for follow-up performance analysis.

  • You want to save a reference point for how your application is working.

  • You want to share a minimal reproducible with the developer tools or driver teams at NVIDIA to facilitate bug reporting.

The Generate C++ Capture activity supports all APIs that are generally supported by Nsight Graphics.

Basic Workflow

To start this activity, select Generate C++ Capture from the connection dialog.

../_images/activity_list_cpp_capture.png

Once the application is running, the Generate C++ Capture button will be available on the main toolbar.

../_images/generate_capture_toolbar.01.png

Once a capture is started, the target application will temporarily pause, and a progress dialog will be shown detailing the steps of the export to C++ process. When complete, the C++ project is written to the disk and the application will resume.

../_images/c_capture_complete_dialog.01.png

By default, the save directory is co-located beside the current project. If no project is currently loaded the default save directory is used (see Options > Environment > Default Documents Folder).

In addition to the C++ project, the code generation process also produces an ngfx-cppcap file with additional information and utilities. These ngfx-cppcap files are automatically associated with the current project and can be reopened later.

../_images/generated_capture_numbered.01.png

The additional features of an ngfx-cppcap file include:

  1. Screenshot of the capture taken from the original application.

  2. Information about the captured application and its original system.

  3. Statistics about the captured API stream.

  4. Utilities to build the C++ capture without opening the generated Visual Studio project.

  5. Utilities to launch the compiled application:

    1. The Execute button will launch the compiled executable.

    2. The Connect… button will populate a new connection dialog that allows you to run a specific activity on the generated capture.

  6. User comments that are persisted within this file.

Using a Saved Capture

  1. To use the saved capture, use Visual Studio’s Open Folder capability on the directory that was generated. After doing this, Visual Studio will read the CMakeLists.txt and allow you to build and run the executable . Alternatively, if you are using a version of Visual Studio that is earlier than 2017, and it does not support native CMake loading, you can use a standalone CMake tool to generate the projects for your version of Visual Studio.

  2. These solution files contain a number of generated source files.

    1. Main.cpp — This is where all of the initialization code is called, resources are created, and each frame portion is called in a message loop.

    2. Resources NN .cpp — Depending on the number of resources to be created, there will be multiple Resources NN.cpp files, each with a CreateResourcesNN call in them, that will construct all of the resources (device(s), textures, shaders, etc.) that are used in the scene. These are called in Main.cpp before replaying the frame in the message loop.

    3. FrameSetup.cpp — This file contains all of the state setting calls to set the API state to the proper values for the beginning of the frame, including what buffers are bound, which shaders are enabled, etc.

    4. FrameNPartMM.cpp — In Direct3D and single-threaded OpenGL captures, these files contain the API functions, each named RunFrameNPartMM(), to replay the frame. It is split into multiple files so generated code is easier to work with. These functions are called sequentially in the message loop in Main.cpp.

      Note

      In this scenario, both N and MM are placeholders for numbers in the multiple files generated. Frame N will typically be Frame0 since only a single frame is captured, and Part MM will typically be in the 00-05 range, depending on how many API calls are in the frame.

    5. ThreadLLFrameNPartMM.cpp — In multi-threaded OpenGL captures, these files contain the API functions, each named ThreadLLRunFrameNPartMM(), to replay the frame. The functions correspond to the work done by each thread during the frame. These functions are called by their respective threads and synchronized to replay the saved events in the same order as captured.

    6. ReadOnlyDatabase.cpp — This is a helper class to access resource data that is stored in the data.bin file. It is accessed throughout the code via the GetResource() call.

    7. Helpers.cpp — These functions are used throughout the replayer for various conversions and access to the ReadOnlyDatabase.

    8. Threading.cpp — This file contains helper functions and classes to manage threads used in the project.

  3. Build and run the project.

Building a Saved Capture for a Different Platform

For Vulkan applications, Nsight Graphics currently supports building and running the saved capture for a different platform, e.g., you may have a capture saved on Windows but would like to run it on Embedded Linux.

Nsight Graphics tries to support the cross-platform converting on a best-efforts basis, but there are some platform-specific considerations and mechanisms outside Vulkan, and some platform-specific extensions and functions may not be able to be converted properly for a different platform. Accordingly, the runtime compatibility of the cross-platform converted capture cannot be ensured.

To use a capture saved on a different platform, you can simply open an ngfx-cppcap file in Nsight Graphics through File > Open File…, and click Build button. We will automatically determine the default compile options for you.

Alternatively, you can manually build a saved capture from a command line, but you need to explicitly specify the compile options:

  • NV_TARGET_PLATFORM: select target platform to build for

  • NV_WINSYS: select window system type to interact with graphics API

  • NV_SDKDIR: specify the path to SDK directory (only required when building for Embedded Linux)

Examples:

  • Build a saved capture for Embedded Linux and use x11 as the window system:

    mkdir -p int &&
    cd int &&
    cmake -G "Unix Makefiles" -DNV_INSTALL_FOLDER=bin -DNV_TARGET_PLATFORM=LINUX_EMBEDDED -DNV_WINSYS=x11 -DNV_SDKDIR=<SDK_PATH> .. &&
    cmake --build . --config RelWithDebInfo --target install -- -j13
    
  • Build a saved capture for Windows:

    MD int &&
    CD int &&
    cmake -G "Visual Studio 16 2019" -A x64 -DCMAKE_GENERATOR_INSTANCE="C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional" -DNV_INSTALL_FOLDER=bin .. &&
    cmake --build . --config RelWithDebInfo --target install
    
  • Build a saved capture for Linux Desktop and use xcb as the window system:

    mkdir -p int &&
    cd int &&
    cmake -G "Unix Makefiles" -DNV_INSTALL_FOLDER=bin -DNV_TARGET_PLATFORM=LINUX_DESKTOP -DNV_WINSYS=xcb .. &&
    cmake --build . --config RelWithDebInfo --target install -- -j13
    

Changing a Resource

If you want to change a resource (for example, to swap in a different texture), you can change the parameters for the construction by looking within the ResourcesNN.cpp files for the texture in question. Textures can be matched by size and/or format. Once you find the variable for the texture, look for that name in the FrameSetup.cpp file. This will contain source lines to lock the texture, call GetResource() to retrieve the data from the ReadOnlyDatabase, and then call memcpy(…) to link the data to the texture. You can substitute the call to the ReadOnlyDatabase with a call to read from a file of choice to load the alternate texture.

Changing a Draw Call

If you want to change the state for a given draw call, you can locate the draw call by replaying the capture within Nsight Graphics and scrubbing to find the call you want to examine. Search in the FrameNPartMM.cpp files for Draw NN, where NN is the 0-based draw call index that Nsight Graphics displayed on the scrubber. Doing this will bring you to the source line for that draw call, and from here, you can add any state changes before that call. Alternatively, you can also disable that specific call by commenting out the source call containing the draw call.

Parameters

  • -repeatN — This setting enables Nsight Graphics to use serialized captures in the normal arch workflow. The N setting indicates the number of times to repeat the entire capture; the default setting is -1, which keeps the capture running on an infinite loop.

  • -noreset — This setting controls whether context state and all resources are reset to their beginning of frame value. When this setting is specified, all frame restoration operations will be skipped, avoiding the performance cost associated with them. Note that this may introduce rendering errors if the rendered frame has a data dependency on the results of a previous frame. Additionally, note that, while uncommon, skipping frame restoration does have the opportunity to lead to application crashes.

Generate C++ Capture from a Command Line

To understand how to generate C++ capture, start by launching the CLI with the --help-all argument. This will display all available options the CLI has.

The CLI can launch an application for generating C++ capture in the form:

ngfx.exe --activity="Generate C++ Capture" [general_options] [Generate_C++_Capture_activity_options]

See CLI Arguments Details for the general options details, the Generate C++ Capture activity options:

Table 5. CLI Generate C++ Capture Activity Options

Option

Description

–wait-seconds arg

Wait time (in seconds) before capturing a frame.

–wait-hotkey

The capture is expected to be triggered by pressing Target application capture hotkey on the running application.

If enabled, the option about waiting in seconds would be ignored.

Examples:

  • ngfx.exe --activity="Generate C++ Capture" --platform="Windows"
    --exe=D:\Bloom\bloom.exe --wait-seconds=10
    

    Launch an application for automatically generating C++ capture after waiting the specified count of seconds.

  • ngfx.exe --activity="Generate C++ Capture" --platform="Windows"
    --exe=D:\Bloom\bloom.exe --wait-hotkey
    

    Launch an application for manually triggering capture so as to generate C++ capture. CLI is waiting for the capture triggered from the target side (pressing Target application capture hotkey on the running application).

  • ngfx.exe --activity="Generate C++ Capture" --platform="Windows"
    --project="D:\Projects\Bloom.ngfx-proj" --wait-seconds=10 --no-timeout
    

    Launch an application for automatically generating C++ capture after waiting the specified count of seconds, but with using the launch options and activity options read from a Nsight Graphics project. In addition, --no-timeout disables all timeouts in case this application may take a long time to launch/capture/.

GPU Trace Profiler

The GPU Trace Profiler activity runs a low-level profiler that can be used for developers to optimize application for NVIDIA Turing and above Hardware. It runs on live applications and captures GPU Units’ utilization throughout frame execution. The GPU Trace captured report may help to detect bottlenecks in the GPU Pipeline, as well as areas where your application is under utilizing the GPU.

When to Use the GPU Trace Profiler Activity

The GPU Trace activity provides detailed performance information for various GPU Units.

Use this activity when:

  • You wish to understand the GPU Units’ utilization and search for throughput bottlenecks.

  • You wish to understand how synchronization objects across queues are being executed.

  • You would like to search for opportunities where your application is under-utilizing the GPU.

  • You suspect your engine will benefit from asynchronous compute.

The GPU Trace activity currently supports profiling applications on NVIDIA Turing architecture and above.

System Setup

The target system must be configured to allow performance metrics to be captured by GPU Trace. Please see documentation at this link for a guide on how to do this.

Basic Workflow

To start the activity, select GPU Trace from the connection dialog.

../_images/activity_list_gpu_trace.png
  1. Set up your application for connection (see How to Launch and Connect to Your Application for more information.).

../_images/gpu_trace_launch.01.png
  1. Specify a Start After condition. This parameter defines how and when the trace is started.

    • Manual Trigger: Specifies that the trace will be manually triggered by the user through the host application or the Target application capture hotkey on the running application.

    • Frame Count: The trace will automatically start after a select number of frames have elapsed.

    • Submit Count: The trace will automatically start after a select number of submits have been performed. Specifying 0 will capture all submits.

    • Elapsed Time: The trace will automatically start after a select amount of time has elapsed.

  2. Enable Real-Time Shader Profiler if you want source-level shader performance to be revealed per ~10 usec interval in the Top-Down Calls table and other shader profiler views. Shader performance is collected via a high-speed sampling profiler in the SM hardware, which incurs no shader execution overhead at runtime, and only consumes PCIe TX bandwidth (along with perf counters). When disabled, a more detailed list of SM and L1TEX perf counters will be collected. This setting is compatible with Timeline Metrics with a ‘fire’ icon. Requires ‘Collect Shader Pipelines’ to be enabled. This feature is available on NVIDIA Ampere GA10x and newer architectures.

  1. Specify a Max Duration for the trace. This parameter defines maximum duration of the trace on the target application.

  2. Specify a Limited To condition. This parameter specifies if the trace should be limited to other events than the maximum duration. This parameter also has impact on when the trace starts after the Start After condition has been met.

    • Max Frames: The trace will be limited to a set number of frames in addition to the max duration. Specifying this will cause the trace to contain frame delimiters on the timeline. The trace will start at the beginning of the next frame once the Start After condition has been met.

    • Max Submits: The trace will be limited to a set number of submits in addition to the max duration. The trace will start on the the next submit once the Start After condition has been met.

    • None: The trace will only be limited by the max duration. The trace will start on the the next submit once the Start After condition has been met.

Note

GPU Trace consumes a lot of memory, especially in complex frames. You need to make sure that by capturing a large duration, there is enough memory to consume it all.

Example:

Starting the trace after a manual trigger may be useful when profiling interactive applications. It makes it possible to interact with the app between traces in addition to allowing traces to be triggered at arbitrary points in time.

Example:

Automatically starting the trace after a select number of submits may be useful when profiling non-interactive applications. The trace will be automatically triggered without any manual action required by the user.

  1. Launch or attach to your application. (See How to Launch and Connect to Your Application for more information.)

  2. Once the application is running, various indicators will appear inside the application window, in the GPU Trace target HUD.

    ../_images/gpu_trace_target_hud.png
    1. Data Capture: Indicates the current state of the data capture:

    • Initializing: The capture infrastructure is initializing.

    • Not Available: The capture is unavailable, possibly due to an error, or the application is not running on an NVIDIA GPU.

    • Ready: The target is ready for capture, waiting for the user to trigger it.

    • Waiting: The target is ready for capture, waiting for the start conditions to be met.

    1. Independent Flip (Window only): Indicates whether the optimal presentation path on Windows (I-Flip) is currently taken or not. I-Flip is available in full-screen-exclusive mode, or in windowed mode when the GPU+driver supports Multi-Plane Overlay (MPO).

    2. Cycle Position: Indicates the current hotkey that cycles the position of the HUD. Can be used to change the position of the HUD if it hides important parts of the application.

    d. Background Compiles: Indicates the current state of background shader compilation in the graphics driver. If you’re attempting to profile a stable scene (unmoving camera), waiting for background compiles to reach an Inactive state will yield more stable and representative performance numbers.

    • Inactive: No background shader compiles are in progress.

    • Active: Background shader compilation is in progress.

  3. If the application successfully connected, the process name will appear in the lower right corner of the window. If you configured the trace to start after a Manual Trigger then you can generate a new trace by clicking the Generate GPU Trace Capture button or by pressing the Target application capture hotkey on the running application. If you configured the trace to start automatically after a given condition is met, then just wait for the trace to complete.

../_images/gpu_trace_capture.01.png

Note

GPU Trace captures all GPU activity. Therefore, it is preferred that you run the application on a remote machine and/or turn off all other applications while capturing.

Note

For best accuracy, it is recommended that you run your application in full-screen mode, wait for Independent Flip to be engaged (Windows only), and turn off V-Sync. You can turn V-Sync off from your application or set V-Sync Mode to Off in the activity dialog (this is the default option).

Note

By default, GPU Trace will lock the GPU clock to base before capturing. This methodology is recommended so consecutive captures will be comparable.

  1. After a capture is taken, a popup will appear that allows you to open it and optionally terminate the application.

../_images/gpu_trace_capture_dialog.01.png

Note

It is recommended that you close the application after capturing, in order to free up your system’s memory while exploring the captured file.

How to Interpret a Report

When interpreting a report, reference the GPU Trace UI section for information on how to interpret each of the pieces of information that is provided. Things to consider:

  • Am I GPU bound?

  • Am I using asynchronous compute?

  • Do I have opportunities for asynchronous compute?

  • What workloads are taking the most time?

  • Is my occupancy low for these workloads?

If you determine that you have opportunities for asynchronous compute and you are not currently using (or achieving) async compute, you may want to investigate your engine to understand where or how you can achieve it.

If you determine that you have expensive workloads with low occupancy, you will want to analyze your shader for opportunities to reduce work or reduce register/memory usage to allow for more occupancy.

Generate GPU Trace Capture from a Command Line

To understand how to generate GPU Trace capture, start by launching the CLI with the --help-all argument. This will display all available options the CLI has.

The CLI can launch an application for generating GPU Trace capture in the form:

ngfx.exe --activity="GPU Trace" [general_options] [GPU_Trace_activity_options]

See CLI Arguments Details for the general options details, the GPU Trace activity options:

Table 6. CLI GPU Trace Activity Options

Option

Description

--wait-frames arg

Wait in frames before generating GPU Trace capture.

--wait-seconds arg

Wait time (in seconds) before generating GPU Trace capture.

--wait-hotkey

The capture is expected to be triggered by pressing the Target application capture hotkey on the running application.

If enabled, the options about waiting in frames/seconds would be ignored.

--auto-export

Automatically export metrics data after generating GPU Trace capture.

--num-frames arg

How many frames (1-15) to capture (may be limited by memory availability).

--architecture arg

Selects which architecture the arch-specific options configure.

NOTE: The available architectures can be found in the help message (--help-all).

--metric-set-id arg

Select metric set index to use with the given architecture.

NOTE: The available metric set indices (and the corresponding metric set names) can be found in the help message (--help-all).

--metric-set-name arg

Select metric set name to use with the given architecture.

NOTE: The available metric set names (and the corresponding metric set indices) can be found in the help message (--help-all).

--multi-pass-metrics

Enable Multi-Pass Metrics, which will collect additional data over multiple passes.

--per-arch-config-path arg

Path to a json file that contains per architecture configuration.

The file should be structured as a top level array containing objects that specify “architecture”, “metric-set-name” or “metric-set-id”, and “multi-pass-metrics”, for example:

[
    {
        "architecture": "Turing",
        "metric-set-name": "Throughput Metrics",
        "multi-pass-metrics": "true"
    },
    {
        "architecture": "Ampere GA10x",
        "metric-set-id": "0"
    }
]

NOTE: The available architectures and metric sets can be found in the help message (--help-all).

Examples:

  • ngfx.exe --activity="GPU Trace" --platform="Windows"
    --exe=D:\Bloom\bloom.exe --wait-seconds=10 --architecture=Ada --metric-set-id=1
    

    Launch an application for automatically generating GPU Trace capture after waiting the specified count of seconds, with using the metric set index of 1 on the Ada GPU architecture.

  • ngfx.exe --activity="GPU Trace" --platform="Windows"
    --exe=D:\Bloom\bloom.exe --wait-hotkey --auto-export
    

    Launch an application for manually triggering generating GPU Trace capture. CLI is waiting for the capture triggered from the target side (pressing Target application capture hotkey on the running application). After the capture is finished, CLI also opens the generated GPU Trace capture and exports the metrics data.

  • ngfx.exe --activity="GPU Trace" --platform="Windows"
    --project="D:\Projects\Bloom.ngfx-proj" --wait-seconds=10
    

    Launch an application for automatically generating GPU Trace capture after waiting the specified count of seconds, but with using the launch options and activity options read from a Nsight Graphics project.

System Trace

The System Trace activity is a special activity that connects to the Nsight Systems™ tool. See the Expected Workflow section to understand how Nsight Systems can fit into your profiling and optimization workflow.

When a compatible version of Nsight Systems is present on the system, the System Trace activity offers a connection to Nsight Systems to automatically populate a system trace activity on the external tool with settings from the Nsight Graphics connection dialog. This allows for easy saving of project properties in a singular location.

When a compatible version of Nsight Systems is not present on the system, the System Trace activity offers a convenient link to download the tool. Nsight Graphics must be restarted to discover any newly installed versions of Nsight Systems.

User Interface Reference

This section provides a deep view of all of the user interface elements and views that Nsight Graphics offers.

App Configuration and Activity Selection UI

Launch Tab

The Launch tab enables launching applications for analysis. This is where you will add the basic process information to launch and subsequently connect to the application you wish to analyze.

../_images/connection_dialog.PNG

This tab has the following controls:

  • Application Executable - Specifies the root application to launch. Note that this may not be the final application that you wish to analyze. Reference this field using $(ApplicationExecutable), or its parent directory using $(ApplicationDir).

  • Working Directory - The directory in which the application will be launched. By default the working directory will be set to the application directory. Reference this field using $(WorkingDir).

  • Command Line Arguments - Specify the arguments to pass the application executable.

  • Environment - The environment variables to set in the launched application.

  • Automatically Connect - Specifies whether the launched application should be automatically connected to. If the launched application is a launcher that creates the process that you ultimately wish to analyze, set this to ‘No’.

The following variables can be used in any of the Launch tab fields:

  • $(ProjectDir) - Refers to directory in which the current project is saved.

  • $(ApplicationExecutable) - Refers to the value in the Application Executable field.

  • $(ApplicationDir) - Refers to parent folder of the Application Executable.

  • $(WorkingDir) - Refers to the value in the Working Directory field.

Note

Several fields have a selector to allow you to cycle through recently used entries. This is a useful capability for cycling through common configurations.

../_images/graphics_hud_framedebugger_mode.png

Attach Tab

To attach to an application, it must have previously been launched through the launch tab. This page will list the launched application as well as any children that the application has launched.

Note

If the host disconnects for any reason, and the target happens to still be running, you can reattach to the previously launched or even captured application by using the attach facility. The process does not have to be newly relaunched.

Activities Options

Nsight Graphics allows for adjusting the activity with a large set of options. Options are available in the Connect window under the Additional Options section. These options are saved per-project, and per-activity, because the options for one activity may not relate to the other. Note that you may need to apply them to multiple activities if your needs for each activity are the same.

Table 7. General Options

Option

Description

Capture Mode

Select the capture mode on how to trigger a capture:

  • Manual - capture is manually triggered by a user.

  • By Frame Index - capture is automatically triggered for a given frame index.

    The “Capture Frame Index” option is used to specify the frame index to capture (the minimum value is 1). If the specified frame index is very small, the target HUD would be force disabled.

Enable Target HUD

Enables the HUD on the target application, which enables:

Force Repaint

Enables a periodic trigger of window invalidation, which causes applications that lazily present to repaint, such as many professional visualization applications. This is useful for providing a consistent stream of frames with which Nsight Graphics can perform its analysis.

Table 8. OpenGL Options

Option

Description

Frame Delimiter

Select the API call used to delimit frame boundaries for OpenGL applications. This setting is useful for applications that do not necessarily present to a screen, such as offscreen rendering applications or benchmark applications.

Table 9. D3D Options

Option

Description

Synchronous Shader Collection

Controls the extent of information that is collected for D3D11 shaders. Synchronous collection is necessary for some shader related statistics but may introduce increased application loading time. Synchronous collection also requires that application has been started with administrative privileges

D3D12 Replay Fence Behavior

Choose the behavior when encountering a sync point during D3D12 replay.

Modern APIs, such as D3D12, have fine-grained, application control of synchronization. Tools must infer what the expectations of the application when identifying application syncs, and must do it in a way that allows for high performance while still respecting data hazards.

There are several possible synchronization points, such as when the application calls GetCompletedValue, when an application calls a member of the WaitFor*Object family, when a Signal is observed to have been emitted, etc. This setting controls the approach that is used by Nsight Graphics in reflecting the application synchronization behavior.

  • Default - synchronizes on GetCompletedValue and Wait events

  • Never Sync - never performs synchronization. This option instructs replay to be free running, potentially leading to the highest frame rate. Note that this is extremely likely to run into data hazards, so use with caution.

  • Always sync - performance synchronization at every possible synchronization opportunity (see above list of synchronization points). This will lead to the lowest frame rate, but introduces the most safety in replay. Use this setting as a debugging option if you suspect that there are synchronization options in the application replay. If turning this option on does lead to render-accuracy, please contact support to report this as a bug.

  • No sync on GetCompletedValue - applies all default settings, but turns off synchronization on GetCompletedValue. GetCompletedValue can be used as both a determination of what the current fence value is as well as an input into a control flow decision. Accordingly, because it may lead to control flow, it is synchronized on by default. You may use this setting if you are certain your application never uses GetCompletedValue as a control flow decision.

  • No Sync On Wait Corresponding To SetEventOnCompletion - This options turns off Synchronization on Win32 Wait calls. Note that this is extremely likely to run into data hazards, so use with caution.

DXGI SyncInterval

Controls the SyncInterval value passed to the DXGI Present method. The default is to disable V-Sync to allow the debugger to collect valid real-time counters.

Enable Revision Zero Data Collection

Controls the collection of revision zero (e.g. pre-capture) data during capture. This is potentially an expensive operation, in both memory and processing time, and some applications can replay a single frame without explicitly storing these revisions.

  • Auto - let Nsight decide. Typically this will default to Memory.

  • Memory - Save revision zero data in memory. This is the fastest, data correct option but can incur a large cost.

  • Tempfile - Save revision zero data to temporary files, which Nsight will attempt to clean up no later than process termination. This may avoid memory limits, but comes at a speed cost.

  • Disabled - do not save any revision zero data. This is not generally correct but may work for some applications.

Replay Captured ExecuteIndirect Buffer

When enabled, replays the application’s captured ExecuteIndirect buffer instead of a replay-generated buffer. Consider this option if your application has rendering issues in replay that derive from a non-deterministic ExecuteIndirect buffer, for example one that is generated based off of atomic operations that can vary from frame-to-frame.

Report Force-Failed Query Interfaces

Controls whether failed query interfaces are reported to a user with a blocking message box.

Nsight Graphics is an API debugger, and there may be some APIs that it does not yet support or does not yet know about. When such an interface is queried, the interception will force the failure of the operation with an E_NOINTERFACE return code.

If this operation interferes with normal operation, and otherwise would result in no issues, it may be disabled for the project.

Report Unknown Objects

Controls whether unknown objects are reported to a user with a blocking message box.

Some applications pass objects that are unknown to Nsight Graphics. These objects may be indicative of an application bug, lack of support in the product’s interception, or they may ultimately be innocuous. In many cases, such an unknown object may result in an analysis crash. To mitigate this issue, Nsight Graphics warns about this concern with a blocking message box.

If this operation interferes with normal operation, and otherwise would result in no issues, it may be disabled for the project.

Support Cached Pipeline State

Controls whether cached pipeline state is supported.

By default, Nsight Graphics will reject calls to create or load a cached pipeline state object. Setting this option to true will enable support for these objects.

Force Minimal Shader Bind Tables

Controls whether minimal shader bind tables are forced.

By default, Nsight Graphics will create replay shader bind tables matching the size specified. in the source application. If this option is enabled, it will use the last valid, non-null record in the SBT to override the replay buffer size. This is not universally safe since indexing null records is a valid usage.

Table 10. Vulkan Options

Option

Description

Force Validation

Force the Vulkan validation layers to be enabled. This requires the LunarG Vulkan SDK to be installed.

Validation Layers

Layers used when force enabling validation. This option is only visible when ‘Force Validation’ is turned on.

Device Address C++ Support

Enables buffer device address capture/replay support so that buffers with VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT have the same address between capture and C++ replay. Enabling this option may lead to instability due to different allocation methods used by the driver. This option is only needed to when generating C++ captures of applications that indirectly access a buffer address within a shader.

Force Capture/Replay Device Memory

Forces the VK_MEMORY_ALLOCATE_DEVICE_ADDRESS_BIT and VK_MEMORY_ALLOCATE_DEVICE_ADDRESS_CAPTURE_REPLAY_BIT bits to be be set on all device memory allocations. This is necessary if the application is later binding addressable buffers but incorrectly excluded the flag on the associated memory.

Capture All Device Memory for C++ Captures

Includes the entire contents of device memory for C++ captures, as opposed to just the bound contents. This results in larger capture but might address issues with out-of-bounds memory access.

Enable Coherent Buffer Collection

Controls the monitoring and collection of mapped coherent buffer updates during capture. This is potentially an expensive operation and many applications can replay a single frame without actively monitoring these changes. Use this option if your capture takes a long time but you do not straddle frames with coherent updates.

Enable Revision Zero Data Collection

Controls the collection of revision zero (e.g. pre-capture) data during capture. This is potentially an expensive operation, in both memory and processing time, and some applications can replay a single frame without explicitly storing these revisions.

  • Auto - Let Nsight Graphics decide. Typically this will default to Memory.

  • Memory - Save revision zero data in memory. This is the fastest, data correct option but can incur a large cost.

  • Disabled - Do not save any revision zero data. This is not generally correct but may work for some applications.

Allow Unsafe pNext Values

Allows the inspection of Vulkan structures with potentially dangerous pNext values. By default structures with no known extensions are skipped.

Use Safe Object Lookup

Controls how objects are stored internally by the tool.

Safe lookup are slower but may improve stability when using an unsupported extension.

  • Auto - Fallback to safe mode when an unsupported extension is seen.

  • Enable - Always use safe object lookup.

  • Disable - Never use safe object lookup.

C++ Capture Object Set

This option controls which objects are exported as part of a Vulkan C++ capture.

By default we limit the object set to only objects used in the capture but in some cases a user might want to see all objects used in the application. This typically isn’t necessary and can lead to a very large C++ project.

This might also help WAR a bug where the tool incorrectly prunes an object it shouldn’t have.

  • Only Active - Only include objects actively used in capture

  • All Resources - All active capture objects plus all buffers, images, pipelines, and shaders

  • Full - The entire object set

Reserve Heap Space

Amount of physical device heap space (MB) to automatically reserve for the frame debugger.

Unweave Threads

For multi-threaded applications, attempts to remove excessive context switching by grouping thread events together. May improve C++ capture replay performance of heavily threaded applications.

Ignore DirectX/OpenGL Wrapper Libraries

To capture an application that uses wrapper libraries atop Vulkan, for example DXVK, set this setting to ‘Yes’ to ignore the wrapper library and capture the underlying Vulkan calls. When set to ‘Auto’, Nsight will attempt to auto-detect whether wrapper libraries should be ignored.

Table 11. Ray Tracing Options

Option

Description

Acceleration Structure Geometry Tracking

This option controls how geometry data is tracked for acceleration structures. There are tradeoffs between performance, accuracy, and robustness of any given approach. The default setting of ‘Auto’ is most often implemented in terms of ‘Deep Geometry Copy’, which tries to match the most common application behavior whereby a deep copy is needed. For example, after building an acceleration structure, it is legal for an application to update or destroy the geometry buffers that were used in construction. In this situation, without deep copies of the original data, the tool cannot guarantee full function of the ray traincg inspector, or of C++ capture. If you know that your application does not update or destroy buffers after construction, consider a ‘Shallow Geometry Reference’ option.

Auto

This option generally implies ‘Deep Geometry Copy’ except in cases where Nsight will override the behavior with known application profiles.

Shallow Geometry Reference with Integrity Check

This option is deprecated and will effectively override to Shallow Geometry Reference.

Shallow Geometry Reference

Uses the original geometry buffers as specified by the application. This option has the fastest performance, but only works correctly when the application doesn’t destroy or modify the geometry buffers after construction. Instance buffers are unconditionally deep copied.

Deep Geometry Copy

Deep-copies all builds and refits, collecting full data. This is an exhaustive collection, but it is has the most overhead and increases the memory burden of the tool, leading to risks for out-of-memory situations if the application is already tightly memory constrained.

No Tracking

Neither geometry nor instance buffers are copied. As such it’s unlikely that any component that depends on deep copied data (e.g the ray tracing inspector and C++ captures) will work. This option can be helpful to quantify the overhead of deep copies, to troubleshoot instability, or to remove overhead when profiling or debugging problems not related to ray tracing.

Track Acceleration Structure Refits

Controls whether acceleration structure refits should be tracked in addition to builds. This will require additional memory consumption but may result in a C++ capture with performance that is more representative of the original application.

Report Shallow Report Warnings

Controls whether warnings are issued for possible shallow reference validity issues. If an expert user knows that the original acceleration structure input data remains undisturbed they may silence warnings with this setting.

Collect Geometry In GPU Memory

By default acceleration structure deep copy data is collected in system memory, for stability reasons. Performance may be somewhat better doing the collection into GPU memory, but this puts pressure on the application’s video memory budget.

Table 12. Troubleshooting Options

Option

Description

Enable Driver Instrumentation

Controls the enablement of capabilities that require driver support. This effectively disables:

  • Hardware performance metrics

  • Native shaders collection

  • Other underlying mechanisms for timing

Disabling this option is the first and best option to try if you run into capture errors as it disambiguates problems quickly given the number of subsystems it turns off.

Collect Shader Reflection

Controls the collection of all information reflected from shader objects. This includes source code, disassembly, input attributes, resource associations, etc… Note, dynamic shader editing is not available when this option is disabled. This option is useful if you suspect an error or incompatibility with a shader reflection tool (such as D3DCompiler.dll or SPIRV-Cross).

Collect SASS

Enable fetch of SASS shaders which can be used to collect shader performance stats.

Collect Line Tables

Enable creation of shader-to-PC line tables used by the shader profiler for source correlation.

Collect Hardware Performance Metrics

Enables the collection of performance metrics from the hardware.

Ignore Incompatibilities

Nsight Graphics uses an incompatibility system to detect and report problems that are likely to interfere with the analysis of your application. By default, these incompatibilities are reported and the user is given the option of capturing despite them (with an associated warning of the possibility of issues). Some applications may have innocuous incompatibilities, however, and having to view this warning every time might be undesired.

When this option is enabled, the frame will attempt to capture despite any incompatibilities. Use this option only when you are certain that the incompatibility will not impact your analysis.

Block on First Incompatibility

Nsight Graphics uses an incompatibility system to detect and report problems that are likely to interfere with the analysis of your application. In some cases, these incompatibilities may be the first sign of an impending failure. Accordingly, being able to block on such a reported failure may aid in triaging and understanding a crash when running under Nsight Graphics . This option defaults to ‘Auto’ such that it only reports critical incompatibilities, allowing lesser incompatibilities so as not to interfere with expected operation. It may be useful to toggle to ‘Enable’ if you encounter an application crash under Nsight Graphics to force an opportunity to investigate the crash.

Enable Crash Reporting

Enables the collection and reporting of crash data to help identify issues with the frame debugger.

While a user is always prompted before a crash report is sent, this option is available to suppress these facilities entirely.

Enable C/C++ Serialization

Enables the ability to serialize a capture to C/C++.

By default, applications are available to create a C++ capture, but there are some cases where extra data is collected in support of this feature before it is invoked. This option allows that collection to be disabled entirely.

Force Single-Threaded Capture

Controls whether capture proceeds with concurrent threads or with serialized threads.

Use this option if you suspect your application’s multi-threading may be interfering with the capture process.

Replay Thread Pause Strategy

Controls the strategy used in live analysis for pausing threads.

  • Auto - Use the default strategy, which may disable the Aggressive strategy for some applications.

  • Aggressive - Pause all non-Nsight threads.

  • RenderOnly - Only pause rendering threads.

Disable All Interception

Turns off the interception mechanisms that Nsight Graphics uses to analyze the application. This is used to disambiguate launch failures from failures caused by interception. It is useful for validating that the application settings are correct. When enabled, the associated activity is not expect to work in any way. Please disable this option once you have confirmed that your application launches successfully.

Frame Debugging UI

The Frame Debugger activity is a capture-based activity. There are two classes of views in these activities – pre-capture views and post-capture views. Pre-capture views generally report real-time information on the application as it is running. Post-capture views show information related to the captured frame and are only available after an application has been captured for live analysis. For an example of how to capture, follow the example walkthrough in How to Launch and Connect to Your Application.

All Resources View

The All Resources View allows you to see all of the available resources in the scene.

To access this view, go to Frame Debugger > All Resources.

This view shows a grid of all of the resources used by the application. For graphical resources, these resources will be displayed graphically. For others, an icon is used to denote its type. When a resource is selected, a row of revisions will be shown for that resource. Clicking on any revision will change the frame debugger event to the closest event that generated or had the potential of consuming that revision.

Clicking the link below a resource, or double-clicking on the resource thumbnail, will open a Resource Viewer or Ray Tracing Inspector on that resource.

There are a number of additional capabilities in this view. At the top of the All Resources view, you’ll find a toolbar:

../_images/resources_toolbar.01.png
  • Clone — makes a copy of the current view, so that you can open another instance.

  • Lock — freezes the current view so that changing the current event does not update this view. This is helpful when trying to compare the state or a resource at two different actions.

  • Save — saves the captured resources to disk.

  • Red, Green, and Blue — toggles on and off specific colors.

  • Alpha — enables alpha visualization. In the neighboring drop-down, you can select one of the following two options:

    • Blend — blends the alpha with a checkerboard background.

    • Grayscale — alpha values are displayed as grayscale.

  • Flip Image — inverts the image of the resource displayed.

Below the toolbar is a set of buttons for high-level filtering of the resources based on type. Next to that, there is a drop-down menu that allows you to select how you wish to view the resources: thumbnails, small thumbnails, tiles, or details.

../_images/resources_view_size.01.png

If you select the Details view, you can sort the resources by the available column headings (type, name, size, etc.).

Filtering

There are three ways to filter the available resources.

  1. For high-level filtering, there are color coded buttons to filter based on resource type. All resource types are visible by default, and you can filter the resource list by de-selecting the button for the type you don’t want to see. For example, if you’d like to see only textures, you can click the other buttons to de-select them and remove them from the list of resources.

../_images/resources_filter.01.png
  1. You can manually type in a search string to filter the list of resources.

  2. You can choose from the drop-down of predefined filters to view only large resources, depth resources, unused resources, or resources that change in the frame. Selecting one of these will fill in the JavaScript string necessary for the requested filter, which is also useful as a basis to construct custom filters.

../_images/resouces_filter_predefined.01.png

Application HUD

The Application HUD is a heads-up display which overlays directly on your application. You can use the HUD to capture a frame and subsequently scrub through its constituent draw calls on either the HUD or an attached host.

All actions that occur either in the HUD or on the host — such as capturing a frame or scrubbing to a specific draw call — are automatically synchronized between the HUD and the host, and thus you can switch between using the HUD and host UI seamlessly as needed.

The HUD has two (2) modes:

  1. Running: Interact with your game or application normally, while the HUD shows an FPS counter. When you first start your application with Nsight Graphics, the HUD is in Running mode. This mode is most useful for viewing coarse GPU frame time in real-time while you run your application.

  2. Frame Debugger: Once you have captured a frame, you can debug the frame directly in the Nsight Graphics HUD (as well as from the host). The HUD allows you to scrub through the constituent draw calls of a frame, to view render targets with panning and zooming, and to examine specific values in those render targets.

Running Mode

In this mode, your application can interact with the game or application normally, and the HUD shows frame-time overlaid on the scene. When you first start your application with Nsight Graphics, the HUD is in Running mode.

../_images/graphics_hud_running_mode.png

Frame Debugger Mode

There are two different methods to pause the application, which causes it to enter Frame Debugger mode.

Once you have captured a frame, you can debug the frame directly in the HUD. While you can also debug the frame on the host, the HUD allows you to scrub through the constituent draw calls of a frame, to view render targets with panning and zooming, and to examine specific values in those render targets.

../_images/graphics_hud_framedebugger_mode.png

The HUD scrubber can be clicked to navigate between events. Additionally, the view has several controls to aid in your resource investigation.

Hot Keys

Action

Left-click + drag on the scrubber bar

Navigate to a particular draw call in your frame. When the desired draw call is active, release the left mouse button. The geometry for the currently active draw call will be highlighted, as long as it is on screen.

Home

Navigate to the first event

End

Navigate to the last event

CTRL + Left

Navigate to the next event

CTRL + Right

Navigate to the previous event

CTRL + Plus (+)

Zoom in

CTRL + Minus (-)

Zoom out

CTRL + Zero (0)

Makes the current texture fit to screen.

Left-click + drag on a render target

Pans and zooms the currently displayed render target. Use the mouse wheel to zoom in to a particular portion of the render target.

N

Cycles between the currently available render targets, depth targets, and stencil targets.

W

Cycles between wireframe modes (off, red, animated).

To switch the display to another active render target:

  • Click the Select Render Target button on the HUD toolbar.

  • A drop-down menu will appear, showing all valid choices for the current draw call. Select the desired render target.

  • Note that if a selected render target is not still active for a different draw call, the display will automatically switch to an active render target.

  • You can also toggle between available render targets using the Ctrl+N hotkey.

HUD ICON

DEFINITION

../_images/graphics_hud_toolbar_previous.png

Navigate to the next event.

../_images/graphics_hud_toolbar_next.png

Navigate to the previous event.

../_images/graphics_hud_toolbar_layer.png

Cycles between the currently available render targets, depth targets, and stencil targets.

../_images/graphics_hud_toolbar_wireframe.png

Cycles between wireframe modes (off, red, animated).

API Inspector

The API inspector is a common view to all supported APIs that offers an exhaustive look at all of the state that is relevant to a particular event to which the capture analysis is scrubbed.

To access this view, go to Frame Debugger > API Inspector.

While the view is common, the state within it is particular to each API. See the section below that relates to your API of interest.

Search

With API Inspector pages, there is a search bar that offers a quick way of finding the information you need on a particular page. The bar will indicate the number of matches in each page, and forward and back navigation buttons are provided for navigating between each match. The buttons also support keybindings, with F3 for next and Shift+F3 for previous.

../_images/api_inspector_search.png

Expand/Collapse

Within an API Inspector page, there are many sections that can be expanded or collapsed to help narrow the information that is displayed to only the information you wish to see at that point in time. While each section can be individually collapsed, the UI has buttons that allow for expanding or collapsing all elements in one click.

../_images/api_inspector_expand_collapse.png

Export

Each page has the ability to be exported to structured data in a json format. This json data will include key value pairs of the data elements, as well as indirections that indicate the relationships between different kinds of data.

This data is useful in cases where you may want to export data for persistence, or perhaps to run a diff between the data of different events.

../_images/api_inspector_export.png

The view also has the ability to export data from all pages. The information in each will be exported to a large, combined file in a structured json format. To accomplish this, use the Export All Pages to Json button.

../_images/api_inspector_export_all.png
D3D11 API Inspector

The API Inspector view has an API-specific pipeline navigator that allows you to select a particular group of state within the GPU pipeline. From here, you can inspect the API state for each stage, including what textures and render targets are bound, or which shaders are in use in the related constants. Note that if a stage is not active (either there is nothing bound to that stage or it doesn’t apply for the current action) it will be greyed out, but you you can still click on it to inspect the state.

Pipeline Stages

The following table shows the stages that are available for inspection:

  • IA —The Input Assembler shows the layout of your vertex buffers and index buffers.

  • VS — Shows all of the shader resource views and constant buffers bound to the Vertex Shader stage, as well as links to the HLSL source code and other shader information.

  • HS — This shows all of the shader resource views and constant buffers bound to the Hull Shader stage, as well as links to the HLSL source code and other shader information.

  • DS — This shows all of the shader resource views and constant buffer bound to the Domain Shader stage, as well as links to the HLSL source code and other shader information.

  • GS — Shows all of the shader resource views and constant buffers bound to the Geometry Shader stage, as well as links to the HLSL source code and other shader information.

  • SO — Shows the resources bound for Stream Output.

  • RS — Shows the Rasterizer State parameters, including culling mode, scissor and viewport rectangles, etc.

  • PS — Shows all of the shader resource views, constant buffers, and render target views bound to the Pixel Shader stage, as well as links to the HLSL source code and other shader information.

  • OM — Shows the Output Merger parameters, including blending setup, depth, stencil, render target views, etc.

  • CS — This shows all of the shader resource and unordered access views and constant buffers bound to the Compute Shader stage, as well as links to the HLSL source code and other shader information.

Input Assembler (IA)

The Input Assembler page shows the details of your vertex buffers and index buffers, the input layout of the vertices.

../_images/api_inspector_d3d11_ia.01.png
Shaders (VS, HS, DS, GS, PS, CS)

The various shader pages display all of the constant buffers, shader resource views, and input/output parameters, as well as links to the HLSL source code and other shader information.

In the constant buffer list, you can expand the buffer to see which HLSL variables are mapped to each entry, as well as the current values.

../_images/api_inspector_d3d11_ps.png

To enable resolution of HLSL variables, you must enable debug info when compiling the shader. See Shader Compilation for a discussion of the parameters required to prepare your shaders for optimal usage within Nsight Graphics.

Rasterizer State (RS)

The Rasterizer State page displays parameters including culling mode, scissor and viewport rectangles, etc.

../_images/api_inspector_d3d11_rs.01.png
Output Merger (OM)

The Output Merger page shows parameters including blending setup, depth, stencil, currently bound render target views, etc.

../_images/api_inspector_d3d11_om.01.png
D3D12 API Inspector

The API Inspector view has an API-specific pipeline navigator that allows you to select a particular group of state within the GPU pipeline. From here, you can inspect the API state for each stage, including what textures and render targets are bound, or which shaders are in use in the related constants. Note that if a stage is not active (either there is nothing bound to that stage or it doesn’t apply for the current action) it will be greyed out, but you can still click on it to inspect the state.

Pipeline Stages

The following table shows the stages that are available for inspection:

  • IA — The Input Assembler shows the layout of your vertex buffers and index buffers.

  • VS — Shows all of the shader resource views and constant buffers bound to the Vertex Shader stage, as well as links to the HLSL source code and other shader information.

  • HS — This shows all of the shader resource views and constant buffers bound to the Hull Shader stage, as well as links to the HLSL source code and other shader information.

  • DS — This shows all of the shader resource views and constant buffer bound to the Domain Shader stage, as well as links to the HLSL source code and other shader information.

  • GS — Shows all of the shader resource views and constant buffers bound to the Geometry Shader stage, as well as links to the HLSL source code and other shader information.

  • SO — Shows the resources bound for Stream Output.

  • RS — Shows the Rasterizer State parameters, including culling mode, scissor and viewport rectangles, etc.

  • PS — Shows all of the shader resource views, constant buffers, and render target views bound to the Pixel Shader stage, as well as links to the HLSL source code and other shader information.

  • OM — Shows the Output Merger parameters, including blending setup, depth, stencil, render target views, etc.

  • CS — This shows all of the shader resource and unordered access views and constant buffers bound to the Compute Shader stage, as well as links to the HLSL source code and other shader information.

Input Assembler (IA)

The Input Assembler page shows the layout of your vertex buffers and index buffers, as well as the vertex declaration information.

../_images/api_inspector_d3d12_ia.01.png
Shaders (VS, HS, DS, GS, PS, CS)

The various shader pages display all of the constant buffers, shader resource views, and input/output parameters, as well as links to the HLSL source code and other shader information.

In the constant buffer list, you can expand the buffer to see which HLSL variables are mapped to each entry, as well as the current values.

../_images/api_inspector_d3d12_cs.png

To enable resolution of HLSL variables, you must enable debug info when compiling the shader. See Shader Compilation for a discussion of the parameters required to prepare your shaders for optimal usage within Nsight Graphics.

Rasterizer State (RS)

The Rasterizer page displays render state settings, texture wrapping modes, and viewport information.

../_images/api_inspector_d3d12_rs.01.png ../_images/api_inspector_d3d12_om.01.png
Output Merger (OM)

The Output Merger page displays parameters such as blending setup, depth, and stencil states.

../_images/api_inspector_d3d12_om.01.png
Device

The Device page displays details about the architecture that was used.

../_images/api_inspector_d3d12_device.01.png
Present

The Present page displays information about back buffers that were used.

../_images/api_inspector_d3d12_present.01.png
OpenGL API Inspector

When using the Frame Debugger feature of Nsight Graphics, you may wish to do a deep dive into the specific draw calls in order to analyze your application further. There are three different categories of API Inspector navigation.

../_images/api_inspector_opengl_framebuffer.png
Pipeline Stages

The first category is laid out like a “virtual GPU pipeline.” This pipeline section of the API Inspector consists of the following:

  • Vtx Spec (Vertex Specification) — State information associated with your vertex attributes, vertex array object state, element array buffer, and draw indirect buffer.

  • VS (Vertex Shader) — Vertex shader state, including attributes, samplers, uniforms, etc.

  • TCS (Tessellation Control Shader) — Tessellation control shader state, including attributes, samplers, uniforms, control state, etc.

  • TES (Tessellation Evaluation Shader) — Tessellation evaluation shader state, including attributes, samplers, uniforms, evaluation state, etc.

  • GS (Geometry Shader) — Geometry shader state, including attributes, samplers, uniforms, geometry state, etc.

  • XFB (Transform Feedback) — Transform feedback state, including object state and bound buffers.

  • Raster (Rasterizer) — Rasterizer state, including point, line, and polygon state, culling state, multisampling state, etc.

  • FS (Fragment Shader) — Fragment shader state, including attributes, samplers, uniforms, etc.

  • Pix Ops (Pixel Operations) — State information for pixel operations, including blend settings, depth and stencil state, etc.

  • FB (Framebuffer) — State of the currently drawn framebuffer, including the default framebuffer, read buffer, draw buffer, etc.

Object and Pixel State Inspectors

The object and pixel state inspectors section of the API Inspector consists of the following:

  • Textures — Details about all of the currently bound textures and samplers, including texture and sampler parameters.

  • Images — Details about all of the images currently bound to the image units.

  • Buffers — Details about all of the bound buffer objects, including size, usage, etc.

  • Program — Information about the currently bound program object and/or pipeline program pipeline object, including shaders, active uniforms, etc.

  • Pixels — Current settings for pixel pack and unpack state.

Miscellaneous

The miscellaneous screen contains additional information such as shader limits, implementation dependent values, transform feedback limits, and various minimum/maximum values.

Vulkan API Inspector

The API Inspector view has an API-specific pipeline navigator that allows you to select a particular group of state within the GPU pipeline. From here, you can inspect the API state for each stage, including what textures and render targets are bound, or which shaders are in use in the related constants. Note that if a stage is not active (either there is nothing bound to that stage or it doesn’t apply for the current action) it will be greyed out, but you you can still click on it to inspect the state.

Pipeline Stages

The following table shows the stages that are available for inspection:

  • Pipeline — Shows information about the currently bound pipeline object.

  • Render Pass — Shows information about the current render pass object.

  • FBO  — Shows information related to the Frame Buffer Object that is associated with the current render pass.

  • IA — The Input Assembler shows the layout of your vertex buffers and index buffers.

  • Viewport — Shows the current viewport and scissor information.

  • VS — Shows all of the shader resource views and constant buffers bound to the Vertex Shader stage.

  • TCS — Shows all of the shader resources associated with the Tessellation Control Shader stage.

  • TES — Shows all of the shader resources associated with the Tessellation Evaluation Shader stage.

  • GS — Shows all of the shader resource views and constant buffers bound to the Geometry Shader stage.

  • SO — Shows the resources bound for Stream Output.

  • Raster — Shows the Rasterizer State parameters, including culling mode, scissor and viewport rectangles, etc.

  • FS — Shows all of the shader resources associated with the Fragment Shader stage.

  • Pix Ops — Shows the Pixel Operations parameters, including depth/stencil, multi-sample, and blending states.

  • Compute — This shows all of the shader resource and unordered access views and constant buffers bound to the Compute Shader stage.

  • Misc - Shows miscellaneous information associated with the instance, physical devices, and logical devices.

Pipeline

The Pipeline page shows information about the currently bound pipeline object including: create info, pipeline layout, and push constant ranges.

../_images/api_inspector_vulkan_pipeline.01.png
Render Pass

The Render Pass page shows information about the current render pass including: clear values, attachments operations, and sub-pass dependencies.

../_images/api_inspector_vulkan_renderpass.png
Input Assembler (IA)

The Input Assembler page shows the layout of your vertex buffers and index buffers, as well as the vertex bindings and attribute information.

../_images/api_inspector_vulkan_ia.png
Shaders (VS, TCS, TES, GS, FS, CS)

The various shader pages display all of the shader modules, including: creation information, human readable SPIR-V source, current push constants, current bound descriptor sets, associated buffers, associated images and samples, and associated texel buffer views for this stage.

../_images/api_inspector_vulkan_vs.png
Raster

The Raster page shows all rasterization information associated with pipeline object include: polygons modes, cull modes, depth bias, and line widths.

../_images/api_inspector_vulkan_raster.01.png
Pixel Operations (Pix Ops)

The Pixel Operations page displays information associated with the current pixel state including: depth/stencil state, multi-sample state, and blending state.

../_images/api_inspector_vulkan_pix_ops.01.png
Miscellaneous Information (Misc)

The Miscellaneous Information page shows information related to the instance, physical device(s), logical device(s), and queue(s)

../_images/api_inspector_vulkan_misc.png

API Statistics View

The API Statistics View is a high-level view of important API calls, and includes information to help you see where GPU and CPU time is spent.

To access this view, go to Frame Debugger > API Statistics.

../_images/api_statistics.01.png

Batch Histogram View

The Batch Histogram view provides an intuitive way for the user to inspect the primitive’s distribution across the draws. The draws can be configurably divided into buckets and allow for disabling or enabling. This can be useful for the user who want to know which draw is heavy and how it affects the render target.

To access this view, go to Frame Debugger > Batch Histogram.

Batch Histogram will display a histogram chart that contains divided buckets and can be configured by a few options.

Click Configure to open the configuration pane:

  • Bucketing Mode — Determines how to divide the draws into buckets.

    • Logarithmic: Bucketing by logarithmic scale.

    • Linear: Bucketing by linear scale.

  • Bucketing Min/Max — Specifies the range that works for bucketing, the out-of-range events will be put into bookend buckets.

  • Bucket Count — Specifies the bucket count.

Click Buckets then the corresponding events show in the table view. You can disable or enable the events by clicking Disable All or Enable All, also can be achieved by the check-box or right-click on the table items. Besides, the linkers on the table is directed to corresponding event in Events List.

../_images/batch_histogram.png

Current Target View

The Current Target view is used to show the currently bound output targets. This can be useful because it focuses in on the bound output resources, rather than having to search for them in the All Resources view.

To access this view, go to Frame Debugger > Current Target.

Current Target will display thumbnails along the left pane for all currently bound color, depth, and stencil targets. This view will change as you scrub from event to event. All of the thumbnails on the left can be selected to show a larger image on the right. You can also click the link below each to open the target in the Resource Viewer.

../_images/current_target.01.png

Event Viewer

The Events view shows all API calls in a captured frame. It also displays both CPU and GPU activity, as a measurement of how much each call “costs.”

To access this view, go to Frame Debugger > Events.

../_images/events.01.png

To add context to each API call, the thread ID and object/context that made that call are offered. Nsight also supports application-generated object and thread names in these columns; see Naming Objects and Threads for guidance on the supported methods for setting these names.

Clicking a hyperlink in the Events column will bring you to the API Inspector page for that draw call.

You can select whether to view the events in a hierarchical or flat view. If multiple performance marker types are used, you can select the correct one, as well as varying levels of verbosity for the call display (variable + value, value, or none). You can also sort the events by clicking on any of the available column headers.

The visibility of columns can be toggled by right-clicking on the table’s header. By default some columns will be hidden if they offer no unique data (e.g. single thread) for the captured frame.

Table 13. Events View Column Descriptions

Column

Description

(Indicator Column)

Points to the currently scrubbed event

(Booksmarks column)

When bookmarks are used, this column will have a visual indication of whether the row in question is bookmarked. When no bookmarks are used, this column is hidden.

Event

The event ID of the API call, operation, or comment for this event. On the left-hand-side, when perfmarkers are used, a perfmarker stack is visually indicated and the ID indicates the range of events contained within this hierarchy level. Hovering over this cell will show the full perfmarker hierarchy. When the event in question is an action, the right-hand-side will contain a link to open the event in the API Inspector.

Description

Describes the API call, operation, or comment. This column hierarchically lists the event hierarchy as controlled by the usage of Marker APIs.

Object

The API object (context, queue, resource, etc.) for which the event was operated on or by.

CPU ms

CPU timing for the event in question. When this is a perfmarker range, this indicates an aggregate summation. These timings are provided for reference, as they are not fully accurate due to the impact of capture operations on their timing. Note that tracing tools like Nsight Systems™ are targeted for higher resolution timing.

GPU ms

GPU timing for the event in question. Not all events will have GPU timings. For perfmarker ranges, a delta time is indicated, noting that this is an approximated aggregation of the timing of the events within it, not an absolute timing.

Thread

The thread that performed the API call, operation, or comment.

Tag

A list of meta-data tags that relate to the call in question. These typically indicate special characteristics of the call that are not evident in their API name or parameters. When no columns have tags, this column is hidden.

Issue

For APIs that support issue tracking (OpenGL), this column indicates whether a particular API call or operation has issues/warnings of interest.

Filtering Events

The events view can be filtered with both a quick filtering expression as well as a detailed configuration dialog.

../_images/eventview_filter_activate.PNG

The filter input box offers a quick, regex-based match against events to find events of interest. Once entered, the view is automatically updated to match against the specified filter.

The Configure button brings up a dialog for more advanced, as well as persistent, filtering of the events in the view.

../_images/eventview_filter_dialog.PNG

Changes within this dialog will take immediate effect. There are three major classes of filters:

  • Event Type Filters - these filters allow filtering to happen on the classification of the event. For example, you may want to hide all non-action events to quickly filter to just draws, clears, and other actions.

  • Other Filters - these filters allow matching against events with a particular characteristics. Additionally, the Advanced Filters tab allows for javascript-based filtering on particular columns.

  • Method Filters - these filters hide methods that match against method names or object types. To add method filters, right click on an event that you wish to hide and select one of the hiding capabilities within the view.

../_images/eventview_method_filter_context_menu.PNG
Filter Persistence

Filters set by the filter configuration dialog will persist from session to session. Additionally, if multiple filter configurations are desired, you may save different named versions and recall them quickly by name.

../_images/eventview_filter_configurations.PNG

Filters entered into the main filter-input box are not persisted, as these filters are meant for quick filtering of the event data.

Regex Syntax

For entries that support regex syntax, the syntax is implemented with a perl-compatible regular expression language. Here are some examples of common tasks and the expressions that achieve them:

Table 14. Example regex filtering expressions

Task

Expression

Search for a draw call

Draw

or use the predefined filter)

Match OpenGL binding calls

glBind

Match D3D AddRef or Release calls

AddRef|Release

Search for D3D methods that set constant buffers

[A-Z]{2,2}SetConstantBuffers

JavaScript Syntax

The Advanced Filters configuration dialog supports JavaScript syntax. This enables complex evaluation of filtering expressions. The basic approach for JavaScript expressions is to match a particular column of data against an expression. Columns are “accessed” via a $('ColumnName') expression. For example, a column titled “Description” is accessed via $('Description'). From there, you can perform mathematical, logical, and text-matching expressions. See some examples below to demonstrate the power and usage of these expressions:

Table 15. Example JavaScript filtering expressions

Task

Expression

Match against the description column for draw

/::Draw/.test($('Description'))

Find events with non-zero GPU time

$('GPU ms') > 0

Find odd events

($('Event') % 2) == 1

Find non-draw events with non-zero GPU time

/::Draw/.test($('Description')) != 1 && $('GPU ms') > 0

Bookmarking

While filtering, it is often desired to keep the context of certain items while you find others. To prevent an event from being filtered, Right Click the event and select Toggle Bookmark.

Alternatively, you can Double Click or use Ctrl + B to bookmark the currently selected event. To navigate between bookmarks, use Alt + Up and Alt + Down.

If you wish to see the filtered results on the scrubber, you can select the tag button to the right of the filter toolbar, and a new row will appear in the Scrubber that displays your filtered events, allowing you to navigate those events in isolation.

../_images/events_filter_scrubber_button.01.png ../_images/events_scrubber_row.01.png
Perf Markers

On the Events page, you can use the hierarchical view to see a tree view of performance markers. The items listed in the drop-downs correspond with the nested child perf markers on the Scrubber.

../_images/events_view_hierarchical.PNG

If you use the flat view on the Events page, the perf marker won’t be nested, but you can hover your mouse over the color-coded field in the far left column, which allows you to view the details about that perf marker.

When an application uses multiple kinds of perf markers, the Marker API allows selecting the API to use for the display. This situation may arise if the application uses a middleware, for example, or mixes components with different marker strategies.

../_images/perf_markers_events.01.png
Event Breadcrumbs

To assist in navigation for an application using perf markers, the Events page shows a breadcrumb trail of the current perf marker stack. Each of these sections, including the current event, are clickable and will navigate back to that location in the Event page.

../_images/event_viewer_breakcrumbs.png
Hotkeys

Navigation

CTRL + Home

Go to the first event in the list.

CTRL + End

Go the last event in the list.

Up Arrow

Go to the previous action.

Down Arrow

Go to the next action.

Left Arrow

Collapse the current perfmarker level.

Right Arrow

Expand the current perfmarker level.

Alt+Up Arrow

Go to the next bookmark.

Alt+Down Arrow

Go to the previous bookmark.

]

Go to the next perfmarker on the same level of the perfmarker stack.

[

Go to the previous perfmarker on the same level of the perfmarker stack.

}

Go to the next perfmarker that is one level (or greater) up the stack for the current selection.

{

Go to the previous perfmarker that is one level up the stack for the current selection.

>

Go down the perfmarker stack to the next perfmarker that is one level below the current selection.

<

Go down the perfmarker stack to the previous perfmarker that is one level below the current selection.

'

Focus the current event to the center of the view.

:

Focus the current event to the top of the view.

;

Focus the current event to the bottom of the view.

&

Focus the current selection by collapsing all other perfmarkers. If the current selection is a perfmarker itself, it will be expanded.

Escape

Clear selection.

Ctrl + A

Select all events.

Ctrl + C

Copy selected events.

Event Details

The Event Details view shows all parameters for the current event in a hierarchical tree structure that allows for searching.

To access this view, go to Frame Debugger > Event Details.

../_images/eventdetails.png

Because this window shows parameters for the current event, it will change as you navigate the scene. If you wish to keep the parameters for comparison against another call, the view supports Clone and Lock capabilities.

Table 16. Event Details Column Descriptions

Column

Description

Name

The name of the parameter or child of a parameter structure.

Value

The value passed in to the API call for this parameter.

Type

The type of the parameter in question.

For events that reference API objects, the event details view provides a link to examine more information to that object in the Object Browser.

../_images/eventdetails_links.png

Geometry View

The Geometry view takes the state of the Direct3D, OpenGL, or Vulkan machine, along with the parameters for the current draw call, and shows pre-transformed geometry.

To access this View, go to Frame Debugger > Geometry.

There are two views into this data: a graphical view and a memory view.

Graphical Tab

../_images/geometry.01.png

Mouse Events

  • Hover — Hint the elements of the hovered primitive.

  • Left Click — Select the primitive or reset the selection if clicking at nothing. When selecting in the graphical viewer, the correlated rows in the memory table are also selected at the same time.

Attribute Options

  • Position — Specifies the vertex attribute to use for positional geometry data.

  • Color — Specifies how to color the geometry. If Diffuse Color is selected, the selected diffuse color swatch will be used for coloring. If a vertex attribute is selected, the selected attribute will be used for per-vertex coloring.

    ../_images/geometry_color.01.png
  • Normal — Specifies the per-vertex normal. This selection applies when using a shade mode that specifies Normal Attribute or when rendering normal vectors.

Rendering Options

Clicking Configure in the bottom right corner of the Geometry View will open up the rendering options menu.

../_images/geometry_configure.01.png
  • Reset Camera — Resets the camera to its default orientation. By default, the viewer bounds all geometry with a bounding sphere for optimal orientation.

  • Zoom To Selected — Zoom the camera to the selected primitive.

  • Render Mode — Determines how to render and raster geometry.

    • Solid: renders filled geometry.

    • Points: renders a vertex point cloud.

      ../_images/geometry_render_mode_points.01.png
    • Wireframe: renders a wireframe of the geometry.

    • Wireframe + Solid: renders filled geometry with a wireframe on top of it.

  • Shade Mode — Specifies the lighting mode of the rendered image.

    • Selected Color Attribute: Shades with the specified color attribute

    • Flat Shading Using Generated Normals: Renders the geometry using flat shading with calculated normals

    • Flat Sharing Using Normal Attribute: Renders the geometry using flat shading with the specified Normal Attribute.

    • Smooth Shading Using Normal Attribute: Renders the geometry using smooth shading with the specified Normal Attribute.

  • Render Normal Vectors — Renders the specified normal attribute as a vector pointing from each vertex. The vector may be colored by the Normal Color selection and may be scaled by the Normal Scale selection.

    ../_images/geometry_render_normal_vectors.01.png

Memory Tab

The Memory tab of the Geometry View shows the contents of the vertex buffer, as interpreted by the current vertex or input attribute specification. This view is useful for seeing the raw data of your draw call. An additional capability of this view is that it highlights invalid or corrupt vertices to streamline finding problematic data. Another useful feature is that the selection linkage to the graphical viewer, where selecting a memory row also selects the associated primitive.

There are two modes of display for the geometry data:

  1. Index Buffer Order shows the vertices as indexed by the current index buffer and current draw call.

    ../_images/geometry_memory_ibo.01.png
  2. Vertex Buffer Order shows the vertices as linearly laid out from the start of the vertex buffer and draw call specification.

    ../_images/geometry_memory_vbo.01.png

Object Browser

The Object Browser view provides a list of all objects tracked for your frame, listed by name and by type. Beneath each object is a list of the properties and other metadata that Nsight tracks. This view is useful for finding objects that utilize a particular kind of property, for example a memory buffer with a particular flag.

To access this view, go to Frame Debugger > Object Browser. This view is also a destination for links provided by the Event Details and Event Viewer views.

../_images/objectbrowser.png

This view supports Clone capabilities. Note, however, that this view captures fixed properties and metadata for each object at the end of frame. For APIs with mutable object properties, such as OpenGL, those properties will not be updated in coordination with scrubbing. As such, Lock capabilities are not applicable to this view.

This view provides two panes side-by-side. The left-hand Objects pane provides the object list as well as their properties; the right-hand pane is context-sensitive and provides additional information about the object that is selected on the left-hand side.

Objects Pane

The objects pane (left-hand side) provides several capabilities for filtering objects:

  • Object names and properties can be filtered via the filter box.

  • Object types can be filtered via the types combo box (default is All Types).

  • Objects can be filtered as to whether they report usages (default is Any # of Usages).

  • The Filter Properties checkbox determines whether the properties of an object are considered when filtering items in the object tree view.

Object Name

This section shows the name of the selected object.

When the selected object has a specific viewer for viewing additional information about that type, a link to that specific viewer will be provided. For example, texture resources will provide a link to opening the selected texture in the Resource Viewer.

Object Usage

This section lists a table for the events in which an object is used. Each event will be tagged to indicate the Usage of that object (READ or WRITE).

Resource Viewer

The Resource Viewer allows you to see all of the available resources in the scene. This view is brought up by clicking resource links in any frame debugging view.

The Resource Viewer is opened through links from resource thumbnails or entries in many Nsight views, for example from the API Inspector or All Resources View.

Once opened, there are two tabs available:

  1. Graphical

  2. Memory

Graphical Tab

The Graphical tab allows you to inspect the resource, pan using the left mouse button to click and drag, zoom using the mouse wheel, and inspect pixel values. Also, this is where you can save the resource to disk. If supported on your GPU and API, this is also where you can initiate a Pixel History session to get all of the contributing fragments for a given pixel.

../_images/resources_graphical.01.png

When you have selected a buffer from the left pane, the Show Histogram button will be available on the right side of the Graphical tab, which allows for remapping the color channels for the resource being viewed.

../_images/resources_histogram.01.png

To modify the histogram view, the following options are available:

../_images/resources_histogram_toolbar.01.png
  • You can set the minimum and maximum cutoff values via the sliders under the histograms, or by typing in values in the Minimum and Maximum boxes.

  • You can change the scale by using the Log button.

  • The Luminance button allows you to visualize luminance instead of color values.

  • The Normalize button can preset the minimum and maximum values to the extents of the data in the resource.

Memory Tab

The Memory tab shows a dump of the resource data.

../_images/resources_memory.01.png

You can use multiple options to configure how this memory is displayed:

  • The Axis drop-down changes between address (memory offset) and index (array element) views.

  • The Offset entry limits the view to an offset within the given resource.

  • The Extent entry limits the view to a maximum extent within the given resource.

  • The Precision spin box controls the number of decimal places to show for floating point entries.

  • The Hex Display toggles between decimal (base-10) and hex (base-8) display formats.

  • Hash shows a hash value representative of the given memory resource within the current offset and extent. This is useful for comparing memory objects or sub-regions.

  • The Transpose button swaps the rows and columns of the data representation.

  • The Configure button opens the Structured Memory Configuration dialog.

Additional Capabilities

There are a number of additional capabilities in this view. At the top of the viewer, you’ll find a toolbar:

../_images/resourceviewer_toolbar.png
  • Clone — makes a copy of the current view, so that you can open another instance.

  • Lock — freezes the current view so that changing the current event does not update this view. This is helpful when trying to compare the state or a resource at two different actions.

  • Save — saves the captured resources to disk.

  • Red, Green, and Blue — toggles on and off specific colors.

  • Alpha — enables alpha visualization. In the neighboring drop-down, you can select one of the following two options:

    • Blend — blends the alpha with a checkerboard background.

    • Grayscale — alpha values are displayed as grayscale.

  • Flip Image — inverts the image of the resource displayed.

Pixel History

Pixel history enables the automatic detection of the draw, clear, and data-update events that contributed to the change in a pixel’s value. In addition, pixel history can identify the fragments that failed to modify a particular texture target, allowing you to understand why a draw might be failing, such as whether you may have misconfigured API state in setting up your pipeline.

To run a pixel history test, click the image75 button and select a pixel to run the experiment on. The Pixel History view will come up with a loading bar and present the results once they are complete.

../_images/resources_histogram.01.png
Structured Memory Configuration

The Structured Memory Configuration dialog allows the user to specify a data layout to interpret the raw data backing the selected resource. For example, a texture may be represented by its colors channels or a uniform buffer may be represented by the various types packed within that buffer.

../_images/resources_structured_memory_config.01.png

Typing in a valid structure definition will automatically update the viewer to respect the configuration.

../_images/resources_memory_modified.01.png

New columns can be created using a simple C-like syntax.

int;      // creates a column with an anonymous int
int x;    // creates a second column with an int named x
float y;  // creates a third column with a float named y

Where additional user types can be defined like the following:

struct MyType{ int x; float y;};
struct MyOtherType{ MyType z; double u; };

Many common sized, unsized, and normalized types are permitted as valid types. Vector and matrix types are provided in a similar syntax to HLSL and GLSL. The full list of supported types can be browsed and searched by clicking on the expandable “Defined Types” sub-section of the configuration dialog.

../_images/resources_memory_defined_types.01.png

As some additional notes on the parser:

  • Full C/C++ grammar is not supported.

  • Single line comments are accepted; c-style block comments (/* */) are not.

  • Macros are not currently supported.

  • Alignments are not considered; all types are considered packed.

  • To add explicit padding, use padN where N is a multiple of 8.

  • Members can be selectively hidden as well, which can be useful for narrowing your data.

When clicking on a texture resource, the configuration is automatically populated to interpret the channels of that format.

../_images/resources_memory_texture_config.01.png

Similarly, buffers are defaulted to a generic byte configuration. A user can typically interpret this buffer data by examining the specific use case. For example, the layout of a vertex buffer can be seen in the Input Assembler section of the API Inspector view, or a uniform buffer can be interpreted by looking at the data layout specified within the shader source.

To persist a configuration, you can click on the Save… button to assign a name to this configuration.

../_images/resources_structured_memory_config_save_dialog.001.png

Later, you can restore this configuration by clicking on the Load… button.

../_images/resources_structured_memory_load_saved.001.png

Scrubber

The Nsight Graphics Frame Debugger has two parts. One part appears as the Frame Debugger window on the host. The other part appears as a Heads-Up Display (HUD) on the target application.

To access this view, go to Frame Debugger > Scrubber.

The part of the Frame Debugger that appears as a HUD on the target machine is comprised of the following:

  • HUD Toolbar — controls the frame capture, along with a number of other options (help, etc.).

  • Frame Scrubber — indicates the current draw event. There is a scrubber view in the Frame Debugger on the host, as well as a frame scrubber on the HUD. The frame scrubber controls stay in synch with each other, meaning that when you move the controls on one, it affects the other. For example, if you move the frame scrubber on the HUD to highlight a new draw event, the scrubber on the Frame Debugger moves in synch to do likewise.

../_images/scrubber_hud_sidebyside.02.png

Understanding the Frame Scrubber

For the sake of discussion when it comes to graphics debugging, it helps to note some common terminology.

  • An event is a single call to the API. It could be a triangle draw call, or backbuffer clear, or a less obvious call, like configuring buffers. A snapshot is a sequence of events.

  • An action is a subset of the event types. It can be one of the following: (1) Draw Call, (2) Clear, or (3) Dispatch. Actions are interesting since they explicitly change data which may result in visual changes.

    Note

    For Direct3D frame debugging: The Direct3D runtime documentation states that, “the return values of AddRef & Release may be unstable and should not be relied upon.” The Nsight Graphics Frame Debugger will also take additional references on objects so any code that relies on an exact reference count at a particular time may fail. In general, users should not expect an exact reference count to be returned from the Direct3D runtime. For more information, see Microsoft’s Rules for Managing Reference Counts.

When you debug your graphics project, the Scrubber window shows the perf markers you implemented. When working with user-defined markers, the Scrubber window will use the color and label that you defined for the perf marker.

../_images/perf_markers_scrubber.01.png

On the Scrubber, you can select one performance marker and it will automatically create a range of all of the draw calls that occurred within that time frame. Clicking on it again will cause the scrubber to automatically zoom to that range of events. You can zoom in on a nested/child marker the same way.

To zoom out, click the parent performance marker, or use CTRL + mouse wheel.

Performance markers are also displayed on the HUD, color-coded the same way that they are on the Scrubber. However, on the HUD, the information is condensed, and you must hover your mouse over the selected performance marker to get its details.

../_images/scrubber.01.png

The default view will show the events in your application, in addition to any performance markers you have defined. Clicking the Add… button will open a dialog that allows you to select what type of range you want to add.

../_images/scrubber_row_add_dialog.01.png
  • Program Ranges — Actions that use the same shader program.

  • Viewport — Actions that render to the same viewport rectangle.

  • Alpha Blending Enabled — Actions that have alpha blending enabled.

  • Alpha Test Enabled — Actions that have alpha test enabled.

  • Back Face Cull Enabled — Actions that have back face cull enabled.

  • User — A range defined by you on the fly. Use SHIFT + left-click and drag the scrubber on the created “User” row to create a new range.

Right-clicking on a specific action in the Scrubber will allow you to open the API Inspector for that action, change your view settings, or initiate a shader profile.

Scrubber View Options

../_images/scrubber_view_mode.01.png

From the Mode drop-down menu, choose one of the following:

  • Unit Scale is the default view, which simply shows the actions and events on the timeline.

  • GPU Time Scale displays the GPU activity and how much each event or action cost the GPU.

  • CPU Time Scale displays the CPU activity and how much each event or action cost the CPU.

  • X by CPU, Y by GPU displays the CPU time scale on a horizontal X-axis, and the GPU time scale on a vertical Y-axis.

Depending on which mode you select, you can also select whether you want to view the ruler relative to the capture, viewport, or cursor.

../_images/scrubber_view_ruler_relative.01.png

From the Hierarchy drop-down, Queue Centric sorts the events by queue, while Thread Centric sorts the events by the thread.

../_images/scrubber_view_hierarchy.01.png

Using Hotkeys to Scrub Through a Frame

When the scrubber has focus, you can use the following hotkeys to move the scrubber cursor from one event to another.

Navigation

CTRL + Home

Go to the first event.

CTRL + End

Go the last event.

CTRL + Left Arrow

Go to the previous event.

CTRL + Right Arrow

Go to the next event.

Up Arrow

Expand the current event group (HUD only).

Down Arrow

Collapse the current event group (HUD only).

F2

Current event: show less information (HUD only).

F3

Current event: show more information (HUD only).

Zooming and Panning

CTRL + Scroll mouse wheel up, or

CTRL + NumPadPlus

Zoom in X-axis

CTRL + Scroll mouse wheel down, or

CTRL + NumPadMinus

Zoom out X-axis

CTRL + 0

Reset zoom

CTRL + SHIFT + Scroll mouse wheel up

Increase row height (all rows)

CTRL + SHIFT + Scroll mouse wheel do

Decrease row height (all rows)

CTRL + Left mouse click and drag

Pan

ALT + mouse move

View zoom window

Cursor and Selection

Left mouse click on desired cursor location

Set cursor(Places cursor at closest point to the start of a range.)

Left mouse click on desired row

Select row (The selected row is highlighted in orange.)

SHIFT + Left mouse click and drag

Make range selection

Left mouse click on selection

Zoom to range

Left double-click on event action, or

Right-click menu, Open API Inspector

Open API Inspector

CTRL + A

Select all events

For the purpose of moving the scrubber cursor, the following are considered action events:

  • Draw methods

  • Clear methods

  • Dispatch methods

  • Present methods

For example, if you are looking for the next draw method that was called, you can press the CTRL + RIGHT ARROW on the keyboard to skip over events that are not typically of interest, and only stop on events that are considered action events.

Shader Profiler

The Shader Profiler is a tool for analyzing the performance of SM-limited workloads. It helps you, as a developer, identify the reasons that your shader is stalling and thus lowering performance. With the data that the shader profiler provides, you can investigate, at both a high- and low-level, how to get more performance out of your shaders.

To access this view, go to Frame Debugger > Shader Profiler.

You can alternatively perform shader profiling through targeted actions through several controls within the UI:

  • From the Linked Programs View, when right-clicking a shader.

  • From the API Inspector, when navigating shader pipeline state

  • From the Event List View, when right-clicking an event or range

Note

Shader profiler reports can be exported and saved for future reference or sharing with colleagues. To save the report, click the save icon in the upper-left-hand side of the report.

Sections

The Shader profiler has the following tabbed sections:

  • Summary: this section shows a top-level summary of information about the profiling run. It is the place from which you can understand how your shaders within the profiled range performed.

  • Source: this section reports a per-line breakdown of the performance of your shaders. It includes several visualization modes, including high-level source reports as well as side-by-side high-level source to lower-level correlation.

  • Session Info: reports information on the profiling session including the events that were profiled, how many passes were required, information about the application that was sampled, and what convergence was seen of the profiling session.

Summary Tab

The summary tab shows a top-level summary of information about the profiling run. It contains the following sections:

  • Shaders: this section provides a hierarchical breakdown of the shaders used by the range that was profiled. It is the primary jumping-off point for understanding how your shaders performed.

  • Top-Down Calls: this section provides a hierarchical breakdown of the high-level functions (HLSL / GLSL) and their called sub-functions. It is context-sensitive to the selection in the Shaders.

  • Hot Spots: this section reports the source-lines for which the Top Stalls are discovered. It is context-sensitive to the selection in the Shaders.

  • Ray Tracing Live State: this section reports the ray tracing live state information for ray tracing applications. It is context-sensitive to the selection in the Shaders.

  • Events: this section reports the events for which the Top Stalls are discovered, sorted by GPU time. It is context-sensitive to the selection in the Shaders.

  • Instruction Mix: this section provides a breakdown of the instruction types, input dependencies and output stall locations in your shader(s). It is context-sensitive to the selection in the Shaders and Source Tab views.

Shaders

This section provides a hierarchical breakdown of the shaders used by the range that was profiled. It is the primary jumping-off point for understanding how your shaders performed.

../_images/uireference_shaderprofiler_summary_functions.png

This section presents a table of all of the shaders within a range alongside how each performed. At the root, there is a ‘Session’ element that represents all samples for the entire range. Below it, the shaders will be grouped depending on the Group By setting of the view. Grouping by Shader will flatten the tree; grouping by Pipeline will hierarchically layout all samples according to the pipeline to which they belong.

The left-hand side of the information panel presents information by which you can classify each shader.

  • Type: indicates the kind of hierarchy for elements below.

  • Name: the name of the shader / pipeline for which results were collected.

  • Hash: displays a unique hash of application level shader information (e.g., bytecode).

  • # Warp: indicates the maximum theoretical warp occupancy. At the shader instance level, this is the maximum number of warps that can fit concurrently in an SM, if this shader was run in isolation. At the pipeline level, it shows the range of values across all shader stages’ shader instances. Hover over this cell for a tooltip that reports which resource(s) limited the theoretical warp occupancy. When dynamic conditions (such as the # of primitives) affect the calculation, a range of values is displayed for the two extreme cases.

  • # Reg: indicates the registers per thread needed by a shader.

  • Smem: indicates the bytes of attributes needed by a 3D shader per warp, or the shared memory allocated for a compute shader per thread group.

  • CTA Dim: indicates the thread group size used by a shader (i.e., HLSL numthreads, GLSL local_size, Hull / Tessellation Control patch_size).

  • Correlation: reports the success or failure of reading debug and correlation information for the shader that was profiled. Hover over this cell for a tooltip that reports detailed information about the operation, including the files in which debug information was read.

  • File Name: the file from which this shader was generated.

The center-right side presents the performance of the row in question, including the top stalls for each row and all stall reasons.

See Stall Reasons for a full listing and description of stall reasons.

When sampling, some samples may be reported as Unattributed. Samples in this row either failed correlation, or represent internal operation of the GPU that this sampler does not report. These samples can generally be ignored, except in the case where the results are of a high quantity, in which case we recommend you save this report this issue to communicate this issue to the Nsight Graphics team.

  • Samples: reports the total summation of samples collected for this row.

  • Top Stall #1: reports the stall reason with the highest incidence for the row in question.

  • Top Stall #2: reports the stall reason with the 2nd highest incidence for the row in question.

  • Top Stall #3: reports the stall reason with the 3rd highest incidence for the row in question.

The rightmost side presents the execution counters of the row in question, including instruction executed counts, thread-instruction executed counts and thread divergence.

  • Instructions Executed: reports number of instructions executed.

  • Thread Instructions Executed: reports sum of instructions executed of all threads (this is up to 32 * “Instruction Executed” as each warp has 32 threads).

  • Thread Instructions Executed Pred On: same as “Thread Instructions Executed” but only count the predicated-on threads (the predicated-on threads execute the current instruction while the predicated-off threads skip it).

  • Active Threads Per Warp: reports thread divergence that the average predicated-on threads are executed per instruction.

    • Active Threads Per Warp = Thread Instructions Executed Pred On / (Instructions Executed * 32)

Note

By default some columns are hidden. The visibility of columns can be toggled by right-clicking on the table’s header.

Top-Down Calls

This section provides a hierarchical breakdown of the high-level (HLSL / GLSL) functions and their called sub-functions. It’s only available when the input shader bytecode has debug information. (e.g. adding “/Zi” option for dxc compiling DXIL bytecode) Samples / execution counters are aggregated to the function level. Links can be clicked to navigate to the source line of function definition or where a function is called within the Source section.

../_images/shaderprofiler_topdown_calls.png
  • Function: the name of the function. Expand to see other functions that called by it.

  • Call Location: the file and line that this function is called.

Hot Spots

This section reports the source-lines for which the Top Stalls are discovered. Lines within this table can be double-clicked to navigate to the source line in question within the Source section. The display can also be toggled between High-Level and Intermediate/Lower level by changing the Type selection.

../_images/shaderprofiler_hotspots.png
  • Shader: the shader to which this line belongs.

  • Source Location: the file and line of this hot spot. Expand to see the call path.

  • Function: the name of the function to which this hot spot belongs.

  • Source: the source for the particular hot spot.

Ray Tracing Live State

This section reports the ray tracing live state information for ray tracing applications. Source/IL location links within this table can be clicked to navigate to the source/IL line in question within the Source section.

Variables initialized before an (HLSL) TraceRay or (GLSL) traceRayEXT call, and used after it, are Live State that need to be maintained across the call while invoking hit and miss shaders. For improved performance, we recommend trying to minimize the amount of live state.

../_images/shaderprofiler_raytracinglivestate.png
  • Name: shows the callsite / calling context / live value names (identifiers).

  • Source Location: reports the source location from which this live state information comes.

  • Live State Bytes: reports the size of this live state information.

  • Live State Values: reports the number of live values reloaded / defined at the particular location.

  • Samples: reports the cost of the trace ray call in question, including live state loads, but it does not include the latency penalty on the consumer of the live state loads.

  • Source Preview: shows the source preview for the particular live state inforamtion.

  • IL Preview: shows the IL preview for the particular live state information.

Events

This section reports the events for which the Top Stalls are discovered, sorted by GPU time. It is context-sensitive to the selection in the Shaders. Within each event row is a link to an event that relates to the selection. This allows you to make associations between selected shaders and the resources, state, and commands that utilized the shader in question.

../_images/shaderprofiler_summarypage_events.png
Instruction Mix

The Instruction Mix view provides a breakdown of the instruction types, input dependencies and output stall locations in your shader(s). It is context-sensitive to the selection in the Shaders and Source views.

The Instruction Mix view is useful for navigating through instruction level dependencies. Variable latency instructions – such as memory accesses, transcendental math functions, and warp level primitives – are called scoreboard producers. Instructions that use the results of scoreboard producers, or that reuse the registers from scoreboard producers, are called scoreboard consumers. Stall reasons such as “long scoreboard” appear on the scoreboard consumer instructions, but are actually caused by the corresponding scoreboard producers instructions.

The section contains three tables:

  • Self Instruction Mix: this table provides a breakdown of instruction types and number of samples attributed to each instruction type within the current selection. Each sub-item contains a hyperlink that can be clicked to jump to that location in the Source View.

  • Input Dependencies: this table provides a list of scoreboard producer instruction categories that the current selection is dependent upon. Each sub-item contains a hyperlink that can be clicked to jump to the location of the scoreboard producer in the Source View.

  • Output Stall Locations: this table provides a list of scoreboard producer instruction categories within the current selection. Each sub-item contains a hyperlink that can be clicked to jump to the consumer of the given scoreboard in the Source View.

Each table has the same columns available:

  • Pipe: hardware pipe used to process the instruction.

  • Family: type of work that the instruction performs.

  • Operation: type of operation of the given work type that the instruction performs.

  • Samples: number of samples attributed to the instruction category or location.

  • Instructions: number of instructions or instruction mix corresponding to the given category or location.

The view provides controls to adjust how data is displayed:

  • Filter by Summary Selection: if checked, results will reflect the current selected item(s) on the Shaders view; otherwise the selection will be ignored and results for all shaders will be shown. This does not impact data shown while in the Source view.

  • Auto-Expand Items: if checked, all top-level cells will automatically expand to reveal sub-items with hyperlinks.

  • Show All Source Locations: if checked, all source locations will be listed as sub-items under each top-level cell; otherwise only the top 3 locations by number of samples will be shown.

../_images/shaderprofiler_summarypage_instruction_mix.png
Source Tab

This section reports a per-line breakdown of the performance of your shaders. It includes several visualization modes, including high-level source reports as well as side-by-side high-level source to lower-level correlation.

There are a few top-level controls that control which shader is viewed and how that shader is viewed.

../_images/uireference_shaderprofiler_source_selection.png

Shader: select the shader that you would like to view.

View: change the source display to show high-level language, a lower-level language, or, alternatively, high-level language alongside a lower-level language.

Source: because shaders can be compiled from a main file and several includes, this selector allows you to select which particular source file you wish to investigate.

Once a shader and view is selected, you can use one of several navigation tools to navigate within the shader source.

../_images/uireference_shaderprofiler_source_navigation.png
  • Find: enter a text string to find. This is useful for finding variables, methods, or register names.

  • image91: navigate to the row with the highest value in the corresponding stall reason column

  • image92: navigate to the row with the next higher value in the corresponding stall reason column

  • image93: navigate to the row with the next lower value in the corresponding stall reason column

  • image94: navigate to the row with the lowest value in the corresponding stall reason column

For many uses cases, you will likely want to start by navigating to the highest value, and from there navigate progressively high values until you have planned your next action.

In addition to using the navigation buttons above, you may use the scrollbar with embedded heatmap to identify areas within the source file that will be of interest given a high sample count.

To copy lines, select the lines in question and use the system shortcut to copy to the clipboard. Multiple successive lines can be selected with shift+click; individual lines can be selected with ctrl+click.

../_images/uireference_shaderprofiler_source_sidebyside.png
Session Info

This section reports information on the profiling session including the events that were profiled, how many passes were required, and what convergence was seen of the profiling session.

Collection Statistics indicate information about the collection proceeded and how the collection converged.

  • Pass Count: the number of passes for which samples were collected.

  • Total Samples: The total number of samples collected in this session.

  • Error (min): Standard error of the mean of % samples for a given stall type. This is used to gain confidence that enough passes have been performed. This value is the minimum error seen.

  • Error (max): Like Error (min), yet this reports the maximum error seen.

  • Error (average): Like Error (min), yet this reports the average error seen.

The Configuration section reports key information about the session so that you can know from which application and configuration the session was collected. This section can be important to reference if you collect many reports from different ranges and application configurations.

The Events section lists all of the API events that were sampled within this range.

Shader Profiling for SM Limited Workloads

The Shader Profiler is a tool for analyzing the performance of SM-limited workloads. It helps you, as a developer, identify the reasons that your shader is stalling and thus lowering performance. With the data that the shader profiler provides, you can investigate, at both a high- and low-level, how to get more performance out of your shaders. The Shader Profiler is currently in beta for D3D12 and Vulkan APIs.

../_images/shaderprofiler_summarypage.png
How do I use it?

The Shader Profiler can be launched from several locations within the tool. Because the Shader Profiler targets the performance of your shaders, we recommend that you launch the Shader Profiler when GPU Trace has identified that a particular range is shader limited. Once identified, you can open the Shader Profiler via Frame Debugger > Shader Profiler to collect and present a Shader Profiler report.

How does it work?

The Shader Profiler works by repeatedly running your shader code in a replay and using dedicated hardware samplers to determine the reasons why your code is stalling. The repeated runs allow for capturing of statistically valid sampling that ensures that you are getting a reliable, actionable analysis. Once the sampling experiment is completed, a report is generated that will allow you to find and action on the key hot spots within your shader pipeline.

The reports contains several sections, including a rollup of all of the shaders that were active within the range that was sampled, a high-level, selection-sensitive Sample Summary of the samples within that range, and a Hot spots view that identifies the key lines that contributed the most samples in the overall range. The report also presents tabs that report on session and application information, as well as the Source tab that allows for mapping, on a line-by-line basis, where samples hit.

Key Concepts

The shader profiler should be used to optimize latency-bound shaders. These types of shaders often have signatures of these forms:

  • SM Activity and SM Occupancy are high. (If not, improve these first.)

  • SM Throughput is low.

  • Cache Throughputs (L1TEX, L1.5, L2) are low or middling.

If SM Throughput is high, the shader is likely computationally-bound, and better solved through a GPU Trace workflow.

Average Warp Latency

The average warp latency is the number of cycles that an average warp was resident on the GPU. The Samples% indicates the % of the average warp latency occupied by a given shader, function, or PC. Sorting by Samples reveals the regions of code with the highest contribution to latency. After identify top latency contributors, determine next steps by inspecting stall reasons.

Interpreting Sample Locations

Stalls are reported at the PC where a warp was unable to make progress. In many cases, this is due to an execution or data dependency on a prior instruction. For example, the following code may report a large number of samples on the line of code that consumes texResult, but the real culprit is the data producer g_MeshTexture.Sample().

float4 texResult = g_MeshTexture.Sample(MeshTextureSampler, In.TextureUV);
Output.RGBColor = texResult * In.Diffuse;

Note that samples can appear in the shadow of a taken branch - that is, on the instruction following a branch, even if that instruction is not executed - because the branch is still resolving at the time of the sampling.

if (constantBuffer.ConditionWeExpectToBeFalse)
{
     texResult = ...; // samples in the shadow of a branch
     output = dot(color, textResult);
}
else
{
     output = dot(color, constant);    // expect all samples to fall here
}
Stall Reasons

Stall reasons explain why a warp was unable to issue an instruction. Each stall reason is provoked by a distinct set of conditions or instructions; by eliminating those conditions or transforming code from one set of instructions to another, you can reduce stalls.

  • Barrier: Compute warps are waiting for sibling warps at a GroupSync.

    • If the thread group size is 512 threads or greater, consider splitting it into smaller groups. This can increase eligible warps without affecting occupancy, unless shared memory becomes a new occupancy limiter.

    • Review whether all GroupSyncs are really necessary.

  • Dispatch Stall: A pipeline interlock prevented instruction dispatch for a selected warp.

    • If dispatch stalls are higher than 5%, please file a bug to NVIDIA with reproducible.

  • Drain : Exited warp is waiting to drain memory writes and pixel export.

  • LG Throttle : Input FIFO to the LSU pipe for local and global memory instructions is full.

    • Avoid using thread-local memory.

      • Are dynamically indexed arrays declared in local scope?

      • Does the shader have excess register pressure causing spills?

    • Eliminate redundant global memory accesses (UAV accesses).

    • Data organization: pack UAV or SRV data to allow 64-bit or 128-bit accesses in place of multiple 32-bit accesses.

  • Long Scoreboard : Waiting on data dependency for local, global, texture, or surface load.

    • Find the instruction or line of code that produces the data being waited upon; that instruction is the culprit.

    • Consider transforming a lookup table into a calculation.

    • Consider transforming global reads in which all threads read the same address into constant buffer reads.

    • If L1 hit rate is low, try to improve spatial locality (coalesced accesses).

    • If VRAM Throughput is high, try to improve spatial locality (coalesced accesses).

  • Math Pipe Throttle : A math pipe input FIFO is full (FMA, ALU, FP16+Tensor).

    • This stall reason implies being computationally bound. Use GPU Trace to best determine how to move computation to a different execution unit.

  • Membar : Waiting for a memory barrier to return.

    • Memory barriers are issued by GroupMemoryBarrier, DeviceMemoryBarrier, AllMemoryBarrier, and their GroupSync variants.

    • Review whether the specified scope of each barrier in the shader is really needed. Group-level barriers resolve much faster than Device-level.

    • Review whether a memory barrier is needed at all. A compute shader where each thread writes to a unique UAV location does not require a memory barrier.

  • MIO Throttle : The input FIFO to MIO is full.

    • May be triggered by local, global, shared, attribute, IPA, indexed constant loads (LDC), and decoupled math.

  • Misc : A stall reason not covered elsewhere.

  • Not Selected : Warp was eligible but not selected, because another warp was.

    • High “not selected” could indicate an opportunity to increase register or shared memory usage (lowering occupancy) without impacting performance. Opening the doors to greater shader complexity or improved quality.

  • Selected : Warp issued an instruction. Technically not a stall.

  • Short Scoreboard : Waiting for short latency MIO or RTCORE data dependency.

    • Includes 3D attribute load/store, pixel attribute interpolation, compute shared memory load/store, indexed constant loads, transcendentals (rcp, rsqrt, …) through the SFU pipe (aka XU pipe), VOTE, and a few other infrequent instructions.

  • TEX Throttle : The TEXIN input FIFO is full.

    • Try issuing fewer texture fetches, surface loads, surface stores, or decoupled math operations.

    • Check whether the shader is using decoupled math (usually to be avoided).

    • Consider converting texture lookups or surface loads into global memory lookups (UAVs). Texture can accept 4 threads’ requests per cycle, whereas global accepts 32 threads.

  • Wait : Waiting for coupled math data dependency (FMA, ALU, FP16+Tensor).

../_images/shaderprofiler_stallreasons_table.png
Hot Spots

Hot spots identify the top locations that have the most hit samples. This listing presents an actionable way of identifying and jumping to high-impact areas of the given report.

../_images/shaderprofiler_hotspots.png
Ray Tracing Live State

Ray Tracing Live State lists ray tracing live state information for ray tracing applications and presents an actionable way of identifying and jumping to high-impact areas of the given report.

Variables initialized before an (HLSL) TraceRay or (GLSL) traceRayEXT call, and used after it, are Live State that need to be maintained across the call while invoking hit and miss shaders. For improved performance, we recommend trying to minimize the amount of live state.

../_images/shaderprofiler_raytracinglivestate.png
Source Correlation

The shader profiler has the ability to correlate the samples that are gathered to source-level lines. This allows you, as the user, to determine, on a line-by-line basis, how your code is running. There are two types of correlation that are supported – high-level shader language correlation and GPU shader assembly (SASS) correlation. High-level shader language correlation prepares a listing of your shaders source code, and along-side it, a chart of the samples that landed on each particular line. High-level correlation is very effective at grounding you to the code you are most familiar with, which is the shader source itself. For users who have access to the Pro builds of Nsight Graphics, and who wish to dive into the lower-level shader assembly, a SASS view is provided for individual instruction association of samples.

../_images/shaderprofiler_sourcecorrelation.png

Linked Programs View

The Linked Programs View lists all of the shaders in your application.

  • If the shader (or its parent program or pipeline object) hasn’t been used by the application yet, it will show up with the image101 symbol in the Status column.

  • If the shader has been used, selected statistics will be presented for that shader.

For programs or pipeline objects, you can view the individual shaders by pressing the ► button to the left of the program/pipeline name. When expanded, you can select the link to open a text view of the shader source (when available).

../_images/linked_programs.03.png

Name

This is the name of the shader. This name is either generated internally, or can be assigned by the user per API.

Type

The type of the shader: Vertex, Pixel, Compute, etc.

UID

Indicates a unique object ID for the associated pipeline or shader.

Context

Indicates to which of the application’s contexts this shader is owned. Shown on multi-context OpenGL applications, only.

Status

This column displays the current status of the shader. The status includes Source or Binary, to denote whether or not source code is available for this shader. Also, if the SASS text is included, this means that we have driver level binary code that is necessary for gathering shader performance metrics.

The image103 symbol means that we are waiting for the shader to be bound by the application.

The image104 symbol means that shader performance metrics are currently being computed.

Debug Info

This column indicates that availability and load status of debug information for the associated shader.

File Name

Lists the file name from which the shader was compiled for those shaders with debug info that provides this information.

App Hash

This column displays a unique hash of application level shader information (e.g. bytecode).

# Reg

This column gives the number of registers used by the program. Register count impacts occupancy/threads in flight. This may be not available for all shaders.

The following columns are only valid for compute shaders

CTA Dim

Indicates the CTA dimensions used by the shader

Smem

Indicates the shared memory allocated for the shader per warp/thread

# Barrier

Indicates the number of barriers used by the shader

Ray Tracing Inspector

The Ray Tracing Inspector shows the geometry that has been specified in build commands when running an application that uses ray tracing APIs. If the application does not use these APIs, the view will not be available.

In Ray tracing APIs, such as DXR and NVIDIA Vulkan Ray Tracing, an acceleration structure is a data structure that describes the full-scene geometry that will be traced when performing the ray tracing operation. This data structure is described in detail in the following links: https://developer.nvidia.com/rtx/raytracing/dxr/DX12-Raytracing-tutorial-Part-1 and https://developer.nvidia.com/rtx/raytracing/vkray.

This data structure is purpose-built to allow for translation to application-specific data structures that perform well on modern GPUs. While constructing this data structure, the developer has the responsibility of constructing the structure correctly and using flags to identify the functional and performance characteristics within it. Needless to say, this can be an error-prone operation.

Nsight Graphics Ray Tracing Inspector allows you to view the structures you are creating, navigate through them, and see the flags that you are using. Additionally, you can filter and colorize the structure to highlight, at a bird’s eye view, different kinds of geometry.

The Ray Tracing Inspector is opened through links from resource thumbnails or entries in many Nsight views, for example from the API Inspector or All Resources View. For example, the Ray Tracing Inspector can be opened from the API Inspector View when scrubbed to a build event trace rays call. When scrubbed to these events, the view will present a list of the active structures with a link to open each.

The view is multi-paned – it shows a hierarchical view of the acceleration structure on the left, a graphical view of the structure in the middle, and controls and options on the right. Additionally, a performance analysis section is present on the lower-left. With the hierarchy of the Acceleration Structure tree view, the top-level acceleration structure (TLAS), bottom-level acceleration structures (BLAS), child instances, child geometries, and memory sizes are presented. When a particular item is selected, the name, flags, and other meta-data for this entry are listed in a section on the bottom left-hand side. Each item within the tree has a check box that allows the rendering of the selected geometry or hierarchy to be disabled. Double-clicking on an item will jump to the item in the rendering view and automatically adjust the camera speed to be relative to the size of the selected object.

../_images/ray_tracing_inspector.png
Table 17. Acceleration Structure Hierarchical Columns

Column

Description

Name

An identifier for each row in the hierarchy. Click on the check box next to the name to show or hide the selected geometry or hierarchy. Double-click on this entry to jump to the item in the rendering view.

# Prims

The number of primitives that make up this geometry.

Surface Area

A calculation of the total surface area for the AABB that bounds the particular entry.

Size

A calculation of the memory usage for this particular level. Hover over this entry to see a tooltip that includes a roll-up calculation of the aggregate memory usage of this hierarchical level and children below it.

Performance analysis tools are accessible in the bottom left corner on the main view. These tools help identify potential performance problems that are outlined in the RTX Ray Tracing Best Practices Guide. These analysis tools aim to give a broad picture of acceleration structures that may exhibit sub-optimal performance. To find the most optimal solution, profiling and experimentation is recommended but these tools may paint a better picture as to why one structure performs poorly compared to another.

Table 18. Acceleration Structure Analysis Tools

Action

Description

Instance Overlaps

Identifies instance AABBs that overlap with other instances. Consider merging BLASes when instance world-space AABBs overlap significantly to potentially increase performance.

Instance Heatmap

Enters a heatmap mode that shows the approximate number of instances AABBs that will be hit by a ray cast from each pixel within the current viewport. This mode offers a convenient way to scan your scene for potentially problematic geometries.

Filtering and Highlight

The acceleration structure tree view supports geometry filtering as well as highlighting of data matching particular characteristics. The checkboxes next to each geometry allow individual toggling between full rendering, wireframe rendering, and no rendering. Combining this capability with search allows for you to identify the geometry of interest (by name when the application has named its resources) and display just that geometry.

Geometry instances can also be selected by clicking on them in the main graphical view. Additionally, right clicking in the main graphical view gives options to hide/show all geometry, hide the selected geometry, or hide all but the selected geometry.

../_images/ray_tracing_inspector_filtering.png

Beyond filtering, the view also supports highlight-based identification of geometry specified with particular flags. Checking each Highlight option will identify those resources matching that flag, colorizing for easy identification. Clicking an entry in this section will dim all geometry that does not meet the filter criteria allowing items that do match the filter to standout. Selecting multiple filters requires the passing geometry to meet all selected filters (e.g., AND logic). Additionally, the heading text will be updated to reflect the number of items that meet this filter criteria.

../_images/ray_tracing_inspector_highlighting.png
Rendering Options

Under the highlight controls, additional rendering options are available. These include methods to control the geometry colors and the ability to toggle the drawing of AABBs.

../_images/ray_tracing_inspector_rendering.png
Performance Analysis

The Ray Tracing Inspector has a number of tools that support performance analysis of your accelerations structures.

../_images/ray_tracing_inspector_performanceanalysis.png

Navigation

CTRL + 0

Disable the current performance analysis mode

CTRL + 1

Enable Instance AABB Overlap Heatmap (screen space)

CTRL + 2

Enable Instance AABB Overlap Heatmap (world space)

CTRL + 3

Enable Traversal Timing Heatmap

CTRL + 4

Enable Ray-Geometry Intersection Heatmap

CTRL + 5

Enable Instance AABB Overlap Table

CTRL + space

Alternate between the current and previous performance analysis mode

Instance AABB Overlap Heatmap (screen space)

This tool helps identify screen-space AABB overlaps with the goal of helping a developer reduce expensive overlap.

../_images/ray_tracing_inspector_performanceanalysis_screenspace.png

This heatmap overlay shows the approximate number of instances AABBs that will be hit by a ray cast from each pixel within the current viewport. The more instances hit, the hotter the color. The heatmap threshold can be controlled by “Heatmap Overlay Options -> AABB Threshold” and depends on the number of instance AABBs overlapping.

When screen-space AABBs of instances overlap, unnecessary BLAS traversal maybe required. If these overlaps are caused by empty space within a BLAS, consider splitting the BLAS such this empty space is minimized. If these overlaps are caused by non-opaque geometries, minimize the area not marked as opaque to increase performance.

Instance AABB Overlap Heatmap (world space)

This tool helps identify world-space AABB overlaps with the goal of helping a developer reduce expensive overlap.

../_images/ray_tracing_inspector_performanceanalysis_worldspace.png

This heatmap overlay shows the maximum number of instance AABBs that overlap in world space for any direction within the current viewport. The more instances hit, the hotter the color. The heatmap threshold can be controlled by “Heatmap Overlay Options -> AABB Threshold” and depends on the number of instance AABBs overlapping.

When world-space AABBs of instances overlap, unnecessary BLAS traversal maybe required. If these overlaps are caused by empty space within a BLAS, consider splitting the BLAS such that this empty space is minimized. If these overlaps are caused by non-opaque geometries, minimize the area not marked as opaque to increase performance.

Traversal Timing Heatmap

This tool provides a heatmap of ray traversal time with the goal of identifying expensive geometry.

../_images/ray_tracing_inspector_performanceanalysis_traversaltiming.png

This heatmap overlay shows the number of GPU clock cycles required to trace a ray from the center of each pixel in the viewport to the closest geometry hit. The heatmap threshold can be controlled by “Heatmap Overlap Options -> Timing Threshold” and depends on the traversal speed of the current acceleration structure.

To determine how certain geometry affects the ray traversal speed, geometry can be hidden in which case it is ignored by this algorithm. Additionally, setting “Heatmap Overlay Options -> Transparency Override” to “Force Opaque” will traverse the acceleration structure as if all geometry is opaque.

Ray-Geometry Intersection Heatmap

This tool provides a heatmap of ray geometry intersection with the goal of identify expensive geometry.

../_images/ray_tracing_inspector_performanceanalysis_rayintersection.png

This heatmap overlay shows the number of surfaces hit by a ray traced into the scene from the center of each pixel. The more surfaces hit, the hotter the color. In this mode, opaque geometry terminates a ray. The heatmap threshold can be controlled by “Heatmap Overlay Options -> Intersection Threshold.” When geometry is not marked as opaque, a ray must be traced through all surfaces of that geometry until an opaque surface is hit. To minimize intersection tests, use the opaque geometry flag wherever possible, or use opacity micromaps the mask off opaque sections of non-opaque geometry.

Instance AABB Overlap Table

This table provides information on overlap to allow you to consider merging BLASes when instance world-space AABBs overlap significantly.

../_images/ray_tracing_inspector_performanceanalysis_aabboverlap.png

When world-space AABBs of instances overlap, the TLAS becomes non-optimal. A ray can then hit more than one instance in a volume in space. Traversing through BLASes of all those instances is then required to resolve the closest hit. Traversing through one merged BLAS would be more efficient. Tracing performance against a BLAS doesn’t depend on the number of geometries in it. Geometries merged into a single BLAS can still have unique materials.

Export

Exporting the view, by clicking on the Save (disk) icon in the upper left of the view toolbar, allows for persisting the data you have collected beyond the immediate analysis session. This capability is particularly valuable for comparing different revisions of your geometry or sharing with others. Bookmarks are persisted as well. An example use case is identifying sub-optimal geometry, bookmarking it, and passing this document to a level designer or artist for correction.

Shader Timing Heatmap View

The Shader Timing Heatmap View shows shader execution time on the GPU. It displays a heatmap image where every pixel represents the time of one shader execution unit. The warmer color (more red) means the shader takes more time to execute, while the colder color (more blue) means the shader takes less time. This display allows you to easily identify hot spots and plan optimizations for these areas.

To access Shader Timing Heatmap View, go to Frame Debugger > Shader Timing Heatmap.

If the current event is a DirectX ray tracing or a Vulkan ray tracing call, it’ll start collecting shader times immediately and display them after everything is loaded. Otherwise it’ll display a note for you to select a proper event.

Note

At this time, Nsight Graphics only supports the collecting shader execution time of raygen shaders in DirectX ray tracing and Vulkan ray tracing applications.

The view contains multiple parts. It has API information, overlay controls and a shader timings table on the left, a heatmap viewer as well as its controls on the right.

../_images/shader_timing_heatmap.png

Heatmap

The heatmap viewer allows you to select a rect using left mouse button, pan using middle mouse button, zoom using mouse wheel. The table will show timings in the selected rect if there’s one.

Overlay

The overlay controls allow you to select an image to be an overlay on the heatmap.

../_images/shader_timing_heatmap_overlay_controls.png

When an overlay image is selected, you are able to adjust the opacity using the slider.

../_images/shader_timing_heatmap_overlay.png

Color Mapping

On the right of the heatmap, there’s a legend that shows the mapping from timing values and colors. If you would like to change the mapping, click the right-most button upon the heatmap and control the sliders.

../_images/shader_timing_heatmap_color_mapping.png

Export

Click the Save button on the top-left side to export the view. This view supports saving the session, the heatmap image, or saving the table contents.

D3D12 Specific Views

D3D12 Descriptor Heaps

The Descriptor Heaps view displays all of the descriptor heaps bound for the current event.

To access this view, go to Frame Debugger > Descriptor Heaps.

../_images/descriptor_heaps_d3d12.01.png

On the left are the descriptor heaps available, and on the right you can view the properties of each descriptor heap. Along the top of the details pane, you can see how populated the descriptor heap is, as well as the maximum contiguous valid and invalid ranges. These properties can help you dive into each descriptor heap, and use it as a diagnostic tool to find any potential bugs in your application.

Note that if you click the hyperlink in the Resources column, it will bring up the Resource Viewer.

D3D12 Heaps View

The Heaps view provides a list of all heaps created by the application, along with detailed information about the resources contained in each heap.

../_images/graphics_d3d12_heaps.003.png

When you select a heap from the left pane, you will see all one of two types of entries: Placed Resources or Tiles. Clicking the hyperlink in the Placed Resources box will take you to the Resources Graphical tab.

Tiles are used to populate sections of a tiled resource.

../_images/graphics_d3d12_heaps_tiles.001.png

The right side of the Heaps view displays the memory data associated with the selected resource, which can also be seen on the Memory tab of the All Resources view.

Heap Map

The Heap Map shows a high-level layout of how the heap is currently being used. You can view the usage either by Type (for example, Buffer, Texture2D, etc.) or by the name of the Resource.

Type:

../_images/graphics_d3d12_heap_map_type.001.png

Resource:

../_images/graphics_d3d12_heap_map_resources.001.png

The Heap Map shows any overlapping regions within the heap.

D3D12 Root Parameters

The Root Parameters view displays all of the root parameters bound for the current event. This allows you to quickly change the state of what you’re sampling from, constants, and other descriptors at a lightweight, faster rate than past APIs.

To access this view, go to Frame Debugger > Root Parameters.

../_images/root_parameters_d3d12.01.png

The root signature displays the structure definition of what’s bound at that moment. Root parameters fill in that structure with the values you’re sampling from and the constants you’re using.

When you select a root parameter on the left, the root arguments for that parameter are displayed on the right. This shows residency information, any invalid descriptors are displayed in red. Using root parameters as a diagnostic tool can help prevent a GPU fault.

Note that if you click the hyperlink in the Resources column, it will bring up the Resource Viewer.

Vulkan Specific Views

Vulkan Descriptor Sets View

The Descriptor Set view displays all of the descriptor sets currently allocated and bound by the application at the current event.

To access this view, go to Frame Debugger > Descriptor Sets.

../_images/graphics_vulkan_descriptor_set_view.001.png

The left pane displays a selectable list of descriptor sets along with their layout, pool, consumption counts, and dynamics offsets.

When a set is selected, the right pane will display the resources currently associated with this descriptor set, as well as information related to the pool from which this descriptor set was allocated. In addition, clicking on a resource within the descriptor set will display more detailed information about that specific resource.

Note that if you click the hyperlink in the Preview column, it will bring up the Resource Viewer associated with this image or buffer.

Vulkan Device Memory View

The Device Memory view provides a list of all device memory allocated by the application, along with detailed information about the resources contained in each memory region.

To access this view, go to Frame Debugger > Device Memory.

../_images/memory_pools_vulkan.01.png

The left-most pane contains information about all device memory objects currently allocated. Once a device memory object is selected, the contained resources will be listed in the middle pane, along with the resource layout map in the bottom left, and contained data on the right.

Vulkan Memory Pools

The Memory Pools view provides a list of all device memory allocated by the application, along with detailed information about the resources contained in each memory region.

../_images/memory_pools_vulkan.01.png

The left-most pane contains information about all device memory objects currently allocated. Once a device memory object is selected, the contained resources will be listed in the middle pane, along with the resource layout map in the bottom left, and contained data on the right.

Vulkan Texture and Sampler Pools

The Texture and Sampler Pools View provides a visualization of these different pool types. This can be useful for determining if a particular set of resources are in the resource pools they are expected to be in. The left-hand side allows you to select the pool you’re interested in, based on type. Included in the list are appropriate parameters about how the pool was created. On the right side is a list of the resource descriptors, some information about the resource itself, and a thumbnail preview. There is a link below the thumbnail that allows you to open that resource in the Resource Viewer for deeper inspection.

To access this view, go to Frame Debugger > Texture and Sampler Pools.

../_images/texture_sampler_pools_vulkan.01.png

Generate C++ Capture UI

Compiling and launching C++ captures

../_images/generated_capture_numbered.01.png

The additional features of an ngfx-cppcap file include:

  1. Screenshot of the capture taken from the original application

  2. Information about the captured application and its original system

  3. Statistics about the captured API stream

  4. Utilities to build the C++ capture or open the associated CMake project

  5. Utilities to launch the compiled application:

    1. The Execute button will launch the compiled executable.

    2. The Connect… button will populate a new connection dialog that allows you to run a specific activity on the generated capture.

  6. User comments that are persisted within this file.

GPU Trace UI

GPU Trace profiles live applications. Once a capture is complete, the data is saved in a capture file and can be analyzed offline on any computer where NVIDIA Nsight Graphics is installed, without the need to have the specific GPU installed or the profiled application running.

The GPU Trace window is comprised of 5 sections:

  1. Capture Toolbar

  2. Scrubber: Frames Data and Per-Queue Events

  3. Scrubber: Metric Graphs

  4. Information Tabs

  5. Ranges Table

../_images/gpu_trace_numbered.003.png

Trace Toolbar

At the top left of the scrubber view, there are 5 buttons that extend the scrubber’s capabilities:

../_images/gpu_trace_toolbar.png
  • Ruler Relative: Controls the zero point of the ruler. This can be:

    • Capture: Zero is when the capture begins.

    • Viewport: In this mode, if you select a range and expand it, the beginning of the selected range will be the zero point of the ruler.

    • Cursor: Zero is where the mouse is.

  • Trace Compare: See Trace Compare.

  • Overlay Barriers: See Resource Barriers.

  • Overlay Subshannel Switch: See Subshannel Switch.

  • Flat Queue Rows: In modern graphics API, actions, commands and markers can be executed on different queues. GPU Trace captures these events according to the queue they were executed on, and shows it by default according to this hierarchy. For better granularity, it is possible to toggle this view from hierarchy to flat mode. The flat mode can be used when the user will want to chose to remove some of the rows and / or rearrange the rows order. One specific case would be when the user would like to rearrange the synchronization rows in a way that the signal and wait rows will be one above the other:

    Hierarchy Rows Mode:

    ../_images/gpu_trace_hierarchy_rows_mode.png

    Flat Rows Mode:

    ../_images/gpu_trace_flat_rows_mode.png
  • Aggregate Frames: This option is enabled only when the capture contains multiple frames, and turning it on activates aggregate mode. In this mode:

    • The scrubber shows only the first frame.

    • The ranges table shows only marker data.

    • The metric values shown in the GPU Trace (in the ranges table, metrics tab and scrubber tooltip) are values averaged across all the frames (hovering over the values shows a tooltip displaying the original values used to compute the average).

    • Values which have significant variation between the frames are shown in gray. The threshold for determining this is available in the settings.

    • image119

    This mode is useful when analyzing a capture with multiple frames, to see averaged data and minimize the effect of frame variation.

At the top right of the scrubber view, there are buttons controlling the zoom level. These buttons may assist in navigating the scrubber to the desired view.

../_images/gpu_trace_toolbar_zoom.png
  • Start / End: Marks down the exact time for your start and end selection.

  • Reset Zoom: Will reset the scrubber zoom for the entire capture.

  • Zoom to Selection: Will zoom to the selected range. For multiple markers selection: Select multiple markers using mouse left-click + CTRL.

Scrubber: Frames Data and Per-Queue Events

Frames Row

GPU Trace allows you to capture up to 15 consecutive frames in a single capture. The Frames row shows the frame execution boundary. Double-clicking on a frame will automatically zoom in the Scrubber to the frame boundaries.

../_images/gpu_trace_scrubber_frames.png
Per-Queue Events

NVIDIA GPUs contain multiple independent engines that provide specialized functionality. These engines (e.g. graphics, compute and copy) can execute work in parallel, and work can be submitted to them in separate queues.

In the GPU Trace scrubber, you can observe actions and events that occurred throughout the frame execution, according to the queue they were submitted on. The per-queue part of the scrubber presents events, user markers, and actions.

Queue Synchronization Objects

Since work can be submitted in separate queues, graphics APIs support synchronization of work between queues. GPU Trace capture unveils when Wait and Signal commands are being executed with relevance to the queue. Once such a synchronization object bar is selected, a line connecting to the relevant event will be drawn. This makes it easy to understand when a wait event was triggered, when a signal event released it, and how much time a queue was in a ‘waiting’ state.

../_images/gpu_trace_scrubber_synch.png
Resource Barriers

GPU Trace can capture resource barrier calls. These calls appear as additional events in the synchronization row, relevant to the queue they were triggered on.

../_images/gpu_trace_resource_barriers.png

Use the “Overlay Barriers” toggle button to see how the resource barrier event impacts the metrics graph data:

../_images/gpu_trace_resource_barriers_overlay.png
Subchannel switch overlay

Subchannel Switches can be overlayed on the GPU Trace timeline via a new checkbox. This helps you to better identify the potential performance impact due to a change in the graphics pipeline workload type (which can cause a delay that will decrease SM occupancy and HW unit throughput). This feature is available on NVIDIA Ampere Architecture GPUs and newer..

../_images/gpu_trace_capture_subchannelswitch.png

Use the “Overlay Barriers” toggle button to see how the subchannel switches events impact the metrics graph data:

../_images/gpu_trace_capture_subchannelswitch_overlay.png
User Markers

GPU Trace also captures any User Markers that exist in the application, and display them on the relevant queue it was executed on. This may help understand the frame workflow.

../_images/gpu_trace_scrubber_user_markers.001.png
Actions Row

The Actions row shows work submission actions, in correlation to the time it was executed and the queue it was executed on.

Command lists (or command buffers) submission calls are also shown in the actions row. In this case, the text also shows the number of command lists submitted in the call.

../_images/gpu_trace_actions_row_ecl.png

Scrubber: Metrics Graphs

The Metrics Data Rows can track NVIDIA GPU hardware units’ activity using performance monitors. GPU Trace enables capturing this data and observing in detail the hardware utilization during frame execution.

Note

In order to understand more what action items you can conclude from this data, the following blog is recommended:

The Peak-Performance-Percentage Analysis Method for Optimizing Any GPU Workload

https://devblogs.nvidia.com/the-peak-performance-analysis-method-for-optimizing-any-gpu-workload/

When hovering your mouse over the scrubber, a tooltip appears that displays the average of the metrics data per the selected time. The data is sorted from high to low:

../_images/gpu_trace_scrubber_tooltip.png

GPU Unit’s Metrics Data Rows

GPU Trace presents hardware units’ metric data captured throughout the frame execution. This data is presented in the scrubber. Each counter data is presented in a specific row, while some counters are grouped for convenience. Hovering over the metric’s name, a tooltip will be presented with the counter description. Group rows can be expanded to view individual counters.

The tooltip shows the counter data for the specific time where your mouse is pointed, or the average counter value for the selected range.

Handling Row in the Scrubber

GPU Trace captures a lot of data. It is possible to arrange the Scrubber in a way that will better meet your current needs and will allow you to focus on the area of your interest.

Removing Rows

Focus your performance triage operation by removing rows that are not the main concern by clicking the red - square.

../_images/gpu_trace_scrubber_remove_rows.png

Clicking on the red - square will remove the row from the Scrubber view, but will not delete the data from the database. You can add the row back to the Scrubber by clicking the green + square at the bottom of the Scrubber.

Change Rows Location

You can change the Metrics Data Rows’ location by pressing Alt + Left Click and dragging the rows to the desired location.

Pinned Rows Option

GPU Trace Scrubber allows you to pin rows. When hovering over the row, a pinned button will pop up. If you click this row, it will automatically move to the top of the Scrubber and will remain anchored when scrolling down the other rows. You can choose more then one row to pin. This information will be saved so when reopening the file, the settings will remain. In the below example, the markers row is pinned, and this will allows you to keep the markers row visible:

../_images/gpu_trace_scrubber_pinned_rows.png

User Ranges

User Ranges are ranges that can be added and edited on the captured file. This can be used as personal notes and enhance performance triage capabilities.

To add a user range:

  1. Select a range in the User Ranges row with “SHIFT + Mouse Right-Click.”

  2. In the dialog that pops add you label and description.

  3. Press OK.

  4. Next to the capture file name, there is an asterisk (*), which indicates that this capture has been edited.

  5. You can edit or remove the marker by using the right-click menu.

User ranges acts like any other marker and its data is reflected accordingly in the Summary and Metrics tab. The user ranges information can also be viewed in the ranges table.

../_images/gpu_trace_scrubber_user_ranges_rows.png

Information Tabs

The Information Tabs section provides general information on the capture, and also provides an additional view on the metrics data that were captured.

It contains 3 tabs:

  1. Summary Tab

  2. Metrics Tab

  3. Capture Information

Summary Tab

The upper section on the Summary tab provides details for the selected range. If no selection has been made, the information will be relevant to the entire visible range:

  • Start: The start time of the selected range or the visible range.

  • End: The end time of the selected range or the visible range.

  • Duration: The duration on the selected range or the visible range.

  • Range: An indicator whether the relevant data is applicable to a selected range on the visible range.

../_images/gpu_trace_tab_summary.png

Unit Throughput Summary Table

In this table, you can easily see the average value of the throughput units for the selected range. You can sort values from high to low.

Warp Occupancy Table

In this table, you can easily see the average value of the warp occupancy counters.

Metrics Tab

The Metrics tab encapsulates all metrics data and shows the average value for the selected range. You can easily filter and search for the desired counter using the text search bar. To do so, simply type the counter name (or part of the name), and the table will be filtered automatically.

The metrics are divided according to the GPU Units / roll they represent.

Values in the metrics tab changes according to the corresponding selected range in the metrics graph area.

../_images/gpu_trace_tab_metrics.png
Capture Information Tab

The Capture Information tab provides general information of the capture, such the GPU model, CPU, and operating system that were used for the executable and comma line arguments run. This might be useful when trying to analyze workload behavior or reproduce issues.

../_images/gpu_trace_tab_info.png

Note that if there were any warnings or errors occurred while making this capture, they will appear in this tab.

Ranges Table

The Ranges Table shows the various ranges in the trace (primarly user markers) and their associated metric data. It is correlated with the scrubber, so selecting a marker in the table automatically selects the marker in the scrubber (and vice-versa).

../_images/gpu_trace_regime_table.png

Type

You can select which type of ranges the table displays (with the default being user markers).

Top Metrics Display

With this option enabled, the table shows just the highest metric values for each range. You can change the number of top metrics and their appearance in the settings.

Only Visible

By checking this option, the ranges in the table will be automatically limited to the visible ranges in the scrubber.

Copy Data

For convenience, you can select data from the table and copy to clipboard or save it as a CSV file.

../_images/gpu_trace_regime_table_copy.png

Search Area

You can search for a specific range name by simply typing the range name or part of it. This is a powerful filter that can help detect areas of interest. Examples of such filters:

  • $('SM') > 50 && $('VRAM') < 20 : This filters for all the markers where its SM values is higher than 50 and VRAM is lower then 20.

  • $('Top Unit #1') > 50: This filters for all markers where the top unit value is larger than 50.

Trace Compare

The Trace Compare tool enables the GPU Trace user to easily analyze the effect of his code changes on a specific frame. It displays a simplified version of the GPU Trace time line for 2 frames. The frames are placed one on top of the other, with their start time aligned. Trace compare enables to compare either 2 frames from 2 different capture files or 2 frames within the same file.

Launch the Trace Compare Tool

Option 1: Project Explorer:

Select two capture files in the explorer tree, right click and choose trace compare:

../_images/GT_TraceCompare_LaunchOpetion1.001.png

Option 2: Click on the toolbar button.

Trace Compare Dialog

The Trace Compare dialog shows the selected files to compare. It also enables the user to choose the frame to compare from each capture in cases of multiple frames captures.

../_images/GT_TraceCompare_TCDialog.001.png
Using the Trace Compare Tool

Trace Compare displays the selected frames in a simplified version of the GPU Trace timeline, one on top of the other, aligning the frames’ start time.

../_images/gpu_trace_trace_compare.png

Markers are correlated as well, so when you click on a certain marker on one frame, the matching marker on the other frame will be chosen, if found.

Align to Marker

Sometimes it is easier to spot differences when the selected markers’ start times are aligned. Choose a specific marker and click the Align matching markers check box to activate automatic alignment of matching markers.

Align Selections

It is also possible to manually align any pair of selections. Select any marker, action, or other range in the top and bottom views. Then press the Align Selections button to align the views on the selections.

Metrics Table in Trace Compare Mode

The detailed Metrics Table appears in this mode and shows the metrics data for each frame, side by side, and the delta between the values.

Metrics information

The trace compare tool shows the metrics data for each trace and the ratio between those values.

Profiling Frameless Applications

GPU Trace is able to profile frameless workloads if different criteria are set for when the trace begins and ends.

Configuring GPU Trace:

To profile such applications, change the Start After and Limited To settings within the GPU Trace connection dialog.

  1. Change the Start After condition to one of the following options:

    • Manual Trigger: Specifies that the trace will be manually triggered by the user through the host application or the Target application capture hotkey on the running application.

    • Submit Count: The trace will automatically start after a select number of submits have been performed. Specifying 0 will capture all submits.

    • Elapsed Time: The trace will automatically start after a select amount of time has elapsed.

  2. Change the Limited To condition to one of the following options:

    • Max Submits: The trace will be limited to a set number of submits in addition to the max duration. The trace will start on the the next submit once the Start After condition has been met.

    • None: The trace will only be limited by the max duration. The trace will start on the the next submit once the Start After condition has been met.

Depending on the choice of the Limited To option, it may also be necessary to update the Max Duration setting.

../_images/gpu_trace_oneshot.PNG

Set the rest of the settings as you normally would, providing the executable file and path, working directory, command line arguments and environment variables.

Generating a Trace:

To generate a trace, all you need to do is press the “Launch GPU Trace” button. A trace will be automatically generated when GPU Trace will detect the supported API is in flight.

Open a trace file:

Once the trace has been created, simply open the generated file and analyze it as you normally would.

Things to keep in mind:

Allocated Timestamps

GPU Trace is a detailed profiler and it collects a lot of metrics data, hence, it is limited in the profiling session duration. The Allocated Timestamps setting influences the size of the buffer that the GPU Trace will allocate to keep track of the GPU events. If you get an error message in the Output Messages window saying you run out of resources you might want to try and increase the number of Allocated Timestamps.

../_images/gpu_trace_oneshot1.PNG

Application disconnected:

GPU Trace host launches the target application and profiles it. In this mode the target application may exit automatically, upon application end, you may get warning message saying the communication to the target was lost even though the trace was generated and all worked fine.

../_images/gpu_trace_oneshot2.PNG

Additional Capture Options

Nsight Graphics framework enables launching an application with a specific set of command line arguments and / or environment variables. This is done via ‘Connect to Process’ dialog.

Below are special pre-defined environment variables:

Automatic capture after X number of frames

Set WARPVIZ_CAPTURE_ON_FRAME to trigger a capture automatically after X number of frames elapsed.

For example:

WARPVIZ_CAPTURE_ON_FRAME=100 will trigger capture automatically, once, after 100 frames.

Repeat automatic capture for every X number of elapsed frames

Set WARPVIZ_CAPTURE_FRAME_INTERVAL to automatically trigger a capture for every X frames elapsed

For Example:

WARPVIZ_CAPTURE_FRAME_INTERVAL=100 will trigger a capture every 100 frames.

Lock Clocks to Base

For better consistency between different captures, GPU Trace runs the target applications with ‘Lock Clocks to Base’. This means that the application will not run at maxnimum speed, but will be more consistent between runs. Turn it off if profiling at miximum speed is required.

../_images/gpu_trace_additional_capture_options.png

Trace with Multi-Pass Metrics

GPU Trace trace hardware throughput data on a single frame. This data is collected according to the metrics set defined when launching the application. It is now possible to configure the application to collect ‘Multi-Pass Metrics’. In this mode, the GPU Trace will automatically capture many more counters on consecutive frames. At the end of the capture, you will be able to view all this data presented as a single profiling session.

It is required that when using this mode, the traced application will have user markers since GPU Trace matched frames according to the markers. It is also preferred that the markers execution order will be consistent.

This mode provides additional counters that may explain “Why” there is low throughput.

Enable Multi-Pass Metrics:

Check the “Multi-Pass Metrics” checkbox in the project setting dialog:

../_images/gpu_trace_options_advanced_mode_config.png

Trace with Multi-Pass Metrics:

You can capture using the Target application capture hotkey or the “Generate Capture File” as in the regular mode. However, you might notice that the capture takes a longer time. This is because in this mode much more data is being collected.

This mode relies on markers consistency across frames. If GPU Trace detects such inconsistency, the inconsistent markers will be removed with a warning message:

../_images/gpu_trace_capture_advanced_mode_warning.png

Markers matching algorithm:

The current marker matching heuristics have the following goals:

  • Generate valid timeline mapping for perfectly matching markers

  • Deal with parameterized marker names (to some degree)

  • Put an emphasis on leaf marker matching (as those as typically used for performance analysis)

  • Mark markers which do not match as such, so they can be shown to the user as mismatched

The current implementation handles the marker hierarchy bottom-up (compared to the previous approach, which was top-down), and is done in two phases:

  • Leaf markers matching

  • Parent markers matching

In addition, if multiple frames are captured per pass, the user optionally (on by default) uses “best-frame matching” (see below) to select the best matching frames from each pass (see below).

Markers name comperison:

When comparing marker names, the current heuristics trim any trailing numbers / spaces from the end of the marker names, and then perform the string comparison. This successfully deals with the Unreal Engine frame marker (“Frame N”) but fails for more complex cases.

Leaf Markers Matching:

For each queue, the algorithm collects a list of all the leaf markers (markers without child markers) of that queue, and for each one,k.

The algorithm considers leaf markers as matched if:

  • They have the same name

  • They have the same number of parents

  • All parent names are the same

Parent Markers Matching:

After the leaf marker matching, the algorithm tries to match parent markers from the bottom up.

Parent markers are considered as matched if:

  • They have the same number of child markers

  • All the child markers are matched

Best Frame Matching:

To handle target applications which have workloads alternating in multi-frame cycles (e.g. executing a specific workload once every 2 frames) and improve marker matching chances in general, the best-frame matching logic was introduced.

This works when capturing multiple frames per pass with Multi-Pass Metrics, and results in the user having a single frame to view, which represents frames that match the most.

The algorithm operates as follows:

  • GPU Trace collects N>1 frames per pass with Multi-Pass Metrics

  • When processing a pass, GPU Trace tries to match each frame from the first pass to each frame from the current pass and assigns a score to each frame-to-frame match, based on the total duration of matched markers (ending with NxN scores per pass)

  • The frame from the first pass which received the highest total score is selected, and used to build the timeline mapping by matching it with the frames from each pass which matched best against it

To Activate “Best Frame Matching” go to Tools->Options->GPU Trace->Multi-Pass Metrics

Notes to be considered:

Note

Advance mode automatically capture on concequitive frame. It is recommended to freeze the game if possible, or not move the scene throught the capture process.

View Multi-Pass Metric data:

Note

To work with Multi-Pass Metrics, the target application should use user markers.

The additional counters which are being collected with Multi-Pass Metrics are presented in the summary and metrics tabs and the markers table.

Summary Tab:

../_images/gpu_trace_advanced_mode_summary.png

Metrics Tab:

../_images/gpu_trace_advanced_mode_metrics.png

Markers Table

View markers table information

Profiling Applications with Multiple Windows

It is possible to profile applications with multiple windows. When GPU Trace detects that there are multiple windows in the attached application, it will automatically show a drop-down menu where you can choose the windows you would like to profile. This enables profiling application from within editors:

../_images/gpu_trace_capture_app_with_multiple_windows.png

Detect Interfering Processes While Profiling

GPU Trace collects a performance trace of the GPU during a period that corresponds to the target application activity (whole frames for typical graphics applications). The assumption is that during that time, the GPU performance data represents work done solely on behalf of the target application. However, as the GPU is a shared device, the trace can contain workloads done on behalf of other processes during the capture time, and this can affect the trace data and subsequent performance triage.

Some steps can be taken to minimize the chance of other processes interfering with the trace:

  • Run the target application on a dedicated test machine and do a remote trace

  • Close all other applications which might interfere (e.g. Outlook)

  • Run the application in full-screen mode

However, even with those steps, sometimes other processes can execute GPU workloads unexpectedly, such as the Windows Desktop Window Manager (DWM), and there’s value in detecting these workloads and indicating them to the user in the trace.

GPU Trace will try to detect if there was another process that used the GPU while profiling. If such a process was detected, this region will be shown in the timeline.

You will be able to see the process name to make sure to close it, or else, you cannot rely on this range of metrics information for performance analysis.

../_images/gpu_trace_detect_other_processes.PNG

Notes to be considered:

Note

This feature is currently limited to DirectX 12 on Windows and you must ensure that ‘Hardware-Accelerated GPU Scheduling’ is enabled for it to work properly.

GPU Crash Dumps

GPU Crash Dump Monitor

GPU Crash Dump Monitor Settings

To configure the NVIDIA Nsight Aftermath Monitor settings, left-click the NVIDIA Nsight Aftermath Monitor icon in the Microsoft Windows system notification area (system tray) or right-click the icon and select the Settings option from the pop-up menu.

General Settings

The General Settings page allows to configure the directory where GPU crash dumps will be stored, the directory where NVIDIA shader debug information files are stored, and whether the NVIDIA Nsight Aftermath Monitor should prompt to open newly crash dumps in Nsight Graphics.

../_images/gcd_monitor_settings_general.png
Aftermath Settings

The Aftermath Settings page allows you to configure various options that control Nsight Aftermath graphics driver features and allows you to select for which applications GPU crash dumps will be captured.

Note

On Windows, modifying Aftermath graphics driver settings requires Windows Administrator privileges. Therefore, when any of these settings are modified and applied, a User Account Control confirmation window may pop-up asking for permission to modify system settings.

../_images/gcd_monitor_settings_aftermath.png

Supported Aftermath Modes are the following:

  • Disabled disables all GPU crash dump creation.

  • Global enables crash dump creation for all applications using the D3D11/D3D12 or Vulkan APIs.

  • Whitelist allows you to limit the GPU crash dump creation to a specific set of applications on the whitelist.

Generate Shader Debug Information enables the generation of shader debug information (line tables for mapping from the shader IL passed to the NVIDIA graphics driver to the shader microcode executed by the GPU) for all shaders loaded by the applications for which Aftermath crash dump creation is enabled.

The GPU Crash Dump Monitor will store the debug information into files with the .nvdbg extension in the Debug Info Dump Directory configured in the General Settings Tab.

The shader debug information is required for mapping shader microcode instructions of active or faulted shader warps to shader IL or shader source lines.

See also the section about Source Shader Debug Information for details on how to compile shader source with source-level debug information.

Note

Enabling this setting will cause additional compilation overhead for generating the debug information and general driver overhead for handling the debug information during shader compilation.

Enable Resource Tracking enables driver side tracking of live and recently destroyed resources (textures, buffers, etc.) that is used to augment the GPU fault information in crash dumps.

This allows Aftermath to identify resources related to GPU virtual addresses seen in the case of a crash due to a GPU page fault. The resource information being tracked includes details about the size of the resource, its format, and the current deletion status of the resource object. D3D12 developers may also consider instrumenting their application using the GFSDK_Aftermath_DX12_RegisterResource function to register the D3D12 resources the application creates. That will allow Aftermath to track additional information, such as the resources’ debug names set by the application. For Vulkan applications, the resources’ debug names set via vkSetDebugUtilsObjectNameEXT will also be captured.

Note

Enabling this feature will cause additional driver overhead for tracking resource information.

Enable Call Stack Capturing enables the automatic generation of Aftermath event markers for tracking the origin of all draw calls, compute and ray tracing dispatches, ray tracing acceleration structure build operations, or resource copies initiated by the application. This data can augment the data collected via Aftermath user markers.

The automatic event markers are added into the command stream right after the corresponding commands with the CPU call stacks of the functions recording the commands as the marker data payloads.

Note

Enabling this feature will cause considerable driver overhead for gathering the necessary information.

Note

When this feature is enabled the GPU crash dump file may contain the file path for the crashing application’s executable as well as the file paths for all DLLs or DSOs it has loaded.

Enable Additional Shader Error Reporting puts the GPU in a special mode that allows the GPU to report additional runtime shader errors. This may provide additional information when debugging GPU hangs, GPU crashes, or unexpected behavior related to shader execution.

Enabling this feature may result in additional crash dumps reporting issues in shaders that exhibit undefined behavior or have hidden bugs, which so far went unnoticed because by default the HW silently ignores them. The additional error checks that are enabled when using this option will cause GPU exceptions for the following situations:

  • Accessing memory using misaligned addresses, such as reading or writing a byte address that is not a multiple of the access size.

  • Accessing memory out-of-bounds, such as reading or writing beyond the declared bounds of (group) shared or thread local memory or reading from an out-of-bounds constant buffer address.

  • Accessing a texture with incompatible format or memory layout.

  • Hitting call stack limits.

Note

This feature is only supported with NVIDIA graphics driver R515 or later.

System Settings

The System Settings page contains the systemwide Enable SM Register Data Collection setting. On Linux, the collection of SM registers is always enabled so the System Settings page won’t be present in the monitor.

Note

On Windows, modifying Aftermath system settings requires Windows Administrator privileges. Therefore, when any of these settings are modified and applied, a User Account Control confirmation window may pop up asking for permission to modify system settings.

../_images/gcd_monitor_settings_system.png

Enable SM Register Data Collection to collect SM register values when faults happen inside SMs. This can provide additional information when debugging GPU crashes related to shader execution.

Since this is a system setting, modifying it might also affect other tools such as Nsight VSE CUDA debugger and may result in unexpected behavior! On Linux, this feature is always enabled without incompability with other tools.

Note

This feature is only supported for the D3D12 and Vulkan APIs with NVIDIA graphics driver R535 or later and requires Nsight Graphics Pro to visualize the data.

Command Line Settings

All Aftermath GPU crash dump monitor settings can also be configured through command line parameters.

The available command line flags are:

  • –help Print a help message with a list of the available options.

  • –version Print the release version of the executable.

  • –crashdump-dir arg Set the crash dump directory.

  • –debuginfo-dir arg Set the debug information dump directory.

  • –prompt-on-crash Prompt to open Nsight Graphics after a crash is generated.

  • –hostname arg The host name of the machine on which to look for already running Nsight Graphics instances.

All Aftermath settings and System settings can also be configured through a separate command line tool installed next to the crash dump monitor executable. On Windows, that command line tool is nv-aftermath-control.exe. On Linux, the tool is called nv-aftermath-control.bin.

The command line flags supported by the configuration tool are:

  • –mode arg Set Nsight Aftermath mode. Supported options for arg are: Disabled, Whitelist, or Global.

  • –whitelist arg Add application to the Nsight Aftermath whitelist. arg must be of the following form:

    ApplicationName MyApp ExecutableName myApp.exe
    

    This option can be repeated to add multiple applications to the whitelist. This option also clears a previously set up whitelist.

  • –debuginfo Generate NVIDIA shader debug information.

  • –resource-tracking Enable resource tracking.

  • –callstacks Enable automatic marker generation with call stack capturing.

  • –shader-error-reporting Enable additional shader error reporting.

  • –register-data-collection Enable SM register data collection. Windows only.

Note

On Windows, modifying Aftermath graphics driver settings requires Windows Administrator privileges. Therefore, when nv-aftermath-control.exe is executed, a User Account Control confirmation window may pop-up asking for permission to modify system settings.

New Crash Dump Notification Dialog

If the NVIDIA Nsight Aftermath Monitor is configured to prompt on new crash dumps, every time a new GPU crash dump file is stored to the crash dump directory, a notification dialog will pop up indicating that a new GPU crash dump is available. This dialog shows the name generated for the new crash dump and also allows you to directly open it in a newly launched instance of Nsight Graphics or in an already running instance of Nsight Graphics.

../_images/gcd_monitor_new_crash_dialog.png
New Crash Dump Notification Dialog

If the NVIDIA Nsight Aftermath Monitor is configured to prompt on new crash dumps, every time a new GPU crash dump file is stored to the crash dump directory, a notification dialog will pop up indicating that a new GPU crash dump is available. This dialog shows the name generated for the new crash dump and also allows you to directly open it in a newly launched instance of Nsight Graphics or in an already running instance of Nsight Graphics.

../_images/gcd_monitor_new_crash_dialog.png

GPU Crash Dump Inspector

The GPU Crash Dump Inspector window is comprised of two major views:

  • In the left part of the window, there is a set of tabs that provide summary information for the open GPU crash dump file, as well as information about the captured crash.

  • In the right part of the window, there is a multi-purpose area that shows detailed information based on selections made in some of the sections of the left-side tabs.

../_images/gcd_inspector_crash_info_whole_window.png
Dump Info

The Dump Info tab provides summary information for the open GPU crash dump file and the data contained in the dump. It is comprised of the following sections:

  • The Dump Details section summarizes information about the GPU crash dump file, such as the file name, the date and time the dump was created, and the size of the file.

  • The Application section summarizes information about the application for which the GPU crash dump file was captured, like the name of the executable, the process identifier of the corresponding process, and which graphics API was used.

  • The Exception Summary section summarizes information about the reason for the GPU crash or GPU hang captured in the GPU crash dump file. The first section contains an analysis of any page faults or shader faults detected in the dump. This provides potential causes for these issues and includes links to any available resource information, shader locations, and relevant markers. Analysis for other crash reasons will be added in future updates. The second section shows what state the graphics adapter and D3D or Vulkan device were in when the device recovery was triggered (TDR).

  • The System Info section summarizes the information about the system on which the GPU crash dump file was captured. This include information about the operating system, the graphics driver, and the GPU on which the has crash happened.

    ../_images/gcd_inspector_dump_info.png
Crash Info

The Crash Info tab provides detailed information for data captured in the open GPU crash dump file. The available sections will vary based on the type of the crash and what information was captured into the crash dump.

  • The Active Warps section, if available, shows all active shader executions at the time of the crash or hang. Each row shows the summary for all the warps executing at a specific shader address, including the number of warps, the type of the shader, the shader hash, and the corresponding location within the shader (if shader debug information is available). Clicking a row in the table will open the corresponding Shader View.

    ../_images/gcd_inspector_crash_info_active_warps.png
  • The Faulted Warps section, if available, shows all shader executions that have hit errors. Each row shows a summary of the fault hit on a specific shader address, including the error type, the type of the shader, the shader hash, and the corresponding location within the source shader (if shader debug information is available). Clicking a row in the table will open the corresponding Shader View.

    ../_images/gcd_inspector_crash_info_faulted_warps.png
  • The Active/Faulted Warps section, if available, shows expandable rows each representing the group of all the warps that were executing at a specific shader address. Rows are marked as faulted if any of the warps hit a fault. The Active Warps column shows the number of warps executing at the shader address in the GPU PC Address column. The rows can be expanded to show details about the individual warps. Each entry shows the fault status of the warp, a unique identifier for the warp, the type of the shader, the shader hash, and the source or IL location within the shader (if shader debug information is available). Clicking a row in the table or the address link in the Faulted column will open the corresponding Shader View. The warp information is populated according to the selection and if the warp faulted, additional information such as the name of the fault, its description and the shader address at which the warp hit that fault is also shown. Clicking on a GPU PC Address or Shader Location link will open the Shader View.

    ../_images/gcd_inspector_crash_info_active_faulted_warps.png
  • The Page Fault section, if available, shows information about the GPU page fault that caused the crash. Besides the address of the page that could not be accessed, the type of the fault, the type of the access and the GPU unit from which the page was accessed, it may also show information about the resource that is mapped or was mapped at that address if the Aftermath resource tracking feature was enabled.

    ../_images/gcd_inspector_crash_info_page_fault.png
  • The Fault Info section, if available, shows extra information about the error that caused the crash. This section is shown instead of the Page Fault section for certain other error types.

    ../_images/gcd_inspector_crash_info_fault_info.png
  • The GPU State section shows a high-level summary of the state of various parts of the GPU. This can be helpful to track down which parts of the graphics pipeline were active or have faulted in the case of a crash.

    ../_images/gcd_inspector_crash_info_gpu_state.png
  • The Aftermath Markers section, if available, shows a summary of the Aftermath event markers last processed by the GPU for each of the registered Aftermath contexts. For user event markers, clicking the Payload link in the table will open the corresponding Aftermath Markers View that allows to inspect the user provided marker payload. For automatic event markers, clicking the CallStack link will open a Aftermath Call Stack View showing the call stack of the function that recorded the corresponding graphics command into the D3Dcommand list or Vulkan command buffer. See also the event marker documentation in GFSDK_Aftermath.h for more detail.

../_images/gcd_inspector_crash_info_markers.png
Shader View

The Shader Source view shows the shader code related to the selection made in the Active Warps, Faulted Warps or the Active/Faulted Warps view. This requires that the appropriate information is made available by Configuring the GPU Crash Dump Inspector.

Depending on what information is available for the shader the Language selection box provides the following options:

  • If Source is selected, the view shows the high-level shader source of the shader corresponding to the row selected in the Active Warps, Faulted Warps or the Active/Faulted Warps view. If the shader was compiled from several source files the File selection box allows to switch between the source files. If a row in the Active Warps or the Active/Faulted Warps view is selected, the shader source line that was executing by the selected warp when the crash dump data was captured is marked with a yellow arrow. If that warp has faulted, a red circle will mark the location of the fault. If a row in the Faulted Warps view is selected, the shader source line that corresponds to the faulted instruction is marked with the red circle. The yellow arrow and red circle buttons will jump directly to the corresponding marked instructions.

    ../_images/gcd_inspector_shader_viewer_source.png
  • If IL is selected, the view shows the intermediate assembly of the shader (DXIL or SPIR-V) corresponding to the row selected in the Active Warps, Faulted Warps or Active/Faulted Warps view. If a row in the Active Warps or the Active/Faulted Warps view is selected, the intermediate language statement that was executing by the selected warp when the crash dump data was captured is marked with a yellow arrow. If that warp has faulted, a red circle will mark the location of the fault. If a row in the Faulted Warps view is selected, the intermediate language statement that corresponds to the faulted instruction is marked with the red circle. The yellow arrow and red circle buttons will jump directly to the corresponding marked instructions.

    ../_images/gcd_inspector_shader_viewer_il.png
Aftermath Marker Data View

The Aftermath Marker Data view allows inspection of the Aftermath event marker data provided by the application. Since Aftermath event marker data is typeless the marker data view supports different Data view modes for interpretation of the raw data:

  • As string interprets the event marker data as zero-terminated UTF-8 character string.

    ../_images/gcd_inspector_marker_data_view_string.png
  • As wide string interprets the event marker data as zero-terminated wide character string.

    ../_images/gcd_inspector_marker_data_view_wstring.png
  • Custom allows to inspect the raw event marker byte data or to provide a custom interpretation of the data using a Structured Memory Configuration.

    ../_images/gcd_inspector_marker_data_view_custom.png
Aftermath Marker Call Stack View

The Aftermath Marker Call Stack view shows the call stack for the last draw, dispatch, or copy call processed by the GPU. Resolving the call stack to source location requires a properly set up symbol search path in the Search Paths Settings. Alternatively, clicking the Unknown Symbol link allows to provide a symbol file for a specific call stack element.

../_images/gcd_inspector_callstack_view.png

Project Explorer

The Project Explorer offers a view of all data associated with the current project. It will contain data files, sorted by the time of generation. Note that you may also include arbitrary links to other files as a useful aid in correlating data.

Items in the Project Explorer can be adjusted by right-clicking and selecting an option to interact with it. They may also be renamed by selecting the item and pressing F2.

In addition to navigating via the Project Explorer, you may wish to see the files that were recently generated. Load these through File > Recent Files, or File > Open File.

../_images/gpu_trace_recent_files.02.png

Options

The Options dialog, accessed via the Tools > Options… menu, allows you to configure Nsight Graphics in a number of different ways. Each section is detailed below. The options selected are persisted in user settings for the next time you run the tool.

Environment

../_images/options_environment.03.png

On the Environment tab, select whether to use the light or dark theme, font selection, the default document folder for Nsight Graphics to use, and your preferred startup behavior.

GPU Trace

../_images/options_gpu_trace.03.png

On the GPU Trace tab, you can change the time units and the time precision that are displayed in a GPU Trace. You can change the grid density and the GPU bound threshold (which affects the GPU Bound calculation in the summary tab).

Search Paths

On the Search Paths tab, you can configure search path settings for shader and application debug files used by the NVIDIA Nsight Aftermath GPU Crash Dump Inspector, NVIDIA Nsight Graphics FrameDebugger, and other tools.

../_images/options_search_paths.png
  • Shader Source specifies a list of directories where shader source files can be found. This option is used to associate the high-level shader (HLSL, GLSL, etc.) source files that are used in your application to the file names that are embedded in shader objects by your shader compiler. In many cases source will already be embedded into the shader binaries, but in some cases, especially if the shader compilation tool-chain that is used has its own proprocessing steps, only source file information may be available.

  • Shader Binaries specifies a list of directories where pre-compiled binary shader objects (DirectX shader binary files generated by the HLSL compiler, SPIR-V shader binaries, etc.) can be found.

  • Separate Shader Debug Information specifies a list of directories where shader debug information files separate from the shader object can be found. These are the shader debug information files that may have been produced by your compiler toolchain when compiling the shader objects (.lld or .pdb files generated by dxc.exe for instance).

  • NVIDIA Shader Debug Information specifies a list of directories where NVIDIA shader debug information files can be found. These are the shader debug information files generated by the Nsight Aftermath GPU Crash Dump Monitor (.nvdbg files) or the files created based on the data provided by the shader debug information callback for applications that are instrumented with the GPU crash dump collection feature of the Nsight Aftermath SDK. For more details see the Nsight Aftermath SDK documentation.

  • Application Debug Information specifies a list of directories where debug information (e.g. PDB files or shared object files with debug information) for the application that is analyzed and the dynamic libraries it has loaded can be found. This is necessary to resolve application call stacks in several views.

  • For all of the above paths, it is possible to recursively search the configured directories by enabling the Search sub-directories option that is associated with each.

Injection

../_images/options_injection.01.png

On the Injection tab, select whether to enable or disable debugging Steam overlay.

Frame Debugger (Host)

../_images/options_frame_debugger.03.png

On the Frame Debugger tab, you can configure the time unit and precision settings for the host display, settings for C++ Capture, and set the timeout for a Pixel History.

Feedback

../_images/options_feedback.03.png

On the Feedback tab, choose whether or not you wish to allow Nsight Graphics to collect usage and platform data.

Common View Capabilities

Nsight Graphics supports docking multiple windows within the main window. Any window may be moved, adjusted, tabbed, or pulled out from the docking system that it provides. Most default layouts have multiple documents already specified, but if you wish to adjust these documents you can do so at any time.

Beyond positioning, when frame debugging or profiling, there are buttons that are common across several frame debugger views.

../_images/api_inspector_clone_lock.01.png
  • The Clone button makes a copy of the current view, so that you can compare different parts of the API Inspector (or other cloned views) for the current action.

  • The Lock button freezes the current view so that changing the current event does not update this view. This is helpful when trying to compare the same state on two different actions.

Window Layouts

Nsight Graphics has an intuitive window management and docking system. Windows may be dragged and docked in your preferred locations.

Floating Windows

Windows may be dragged out of the main Nsight Graphics window so that they can be independently managed. Other windows can then dock onto that floating window. On operating systems that support it, such as Microsoft Windows, windows may also be managed from the taskbar.

Pinned Windows

../_images/window_layout_pinning.png

Select the pin icon to collapse the associated window to the side of the main window.

../_images/window_layout_pinned.png

Once pinned, the item can be clicked to open the window for quick reference; clicks away will collapse the pinned window to view other concerns.

User Named Layouts

Nsight Graphics allows users to customize the size and position of the views to create a layout that is targeted to the task at hand. For example, if you are focused on debugging a problem with API usage, you can put the Events View and API Inspector next to each other as you work your way through the frame, inspecting the API state at different points in the frame. The view locations are automatically saved when you exit the frame replay and restored when capture a new frame.

However, different problem types may require unique layouts. To facilitate a smooth transition from one layout to another, you can save and restore activity-centric view arrangements via user named layouts.

You can access this save/load capability from the Window pull down in the main menu. The section pertaining to layouts includes entries to “Save Window Layout…”, “Restore Window Layout”, “Manage Window Layouts…”, and restore the “Default Window Layout”.

../_images/user_named_layouts_menu.png

“Save Window Layout…” will bring up a dialog that allows you to specify a name for the current layout. The layouts are saved to a Layouts folder in the documents directory as named “.nvlayout” files and can be shared with colleagues.

../_images/user_named_layouts_save_dialog.png

Once you have saved a layout (or two), you can restore them by using the “Restore Window Layout” menu entry. When you mouse to it, a sub-menu will pop out with all of your saved layouts. Simply select the entry you want and the views will be restored to their original locations.

../_images/user_named_layouts_restore_menu.png

There may come a time when you want to clean up some unused layouts. When you select the “Manage Window Layouts” entry, a dialog will come up that allows you to delete or rename old layouts, etc.

../_images/user_named_layouts_manage_dialog.png

Finally, the “Reset Window Layout” entry in this section allows you to restore the layout to the default one for the current activity.

Window Chooser

Nsight Graphics has a window chooser for fast enumeration and selection of opened documents and windows . To open this dialog, select Windows > Windows…

../_images/tools_window_chooser.png

Once activated, a window chooser is brought up that contains all of the opened documents and windows.

../_images/window_chooser.png

Navigate with the mouse or keyboard to select an entry and press enter to activate the selected window or document. Alternatively you may double click an entry to activate it.

You may also enter filter expressions to filter to the window of interest.

../_images/window_chooser_filter.png

Troubleshooting

Due to the complex nature of the underlying mechanisms that make arbitrary application analysis possible, there is the possibility of errors. Nsight Graphics offers a significant number of ways where you can discover opportunities to correct issues that you may encounter.

See the sections below for general tools as well as listings of common problems and possible solutions for them. Also, you may want to review known issues to determine if you are encountering an issue already known.

General Tools

This section provides troubleshooting tips for Nsight Graphics.

Caution

Important: some below are considered INTERNAL-ONLY. Some of these options are particularly sensitive. Do not publicize their availability or suggest their use to customers without first verifying that the feature is publicly available by either

  • Looking at a public build for an explicit option or

  • Asking NVIDIA Devtools for confirmation.

Output Messages

Throughout the operation of the tool, Nsight Graphics provides messages that inform on the status of operations as well as if any issues are encountered. This could provide some assistance when trying to determine why your application may not run, connect, or capture correctly. Error messages are indicated by a red flag in the bottom right of the application window. This flag may be double-clicked to open the Output Messages window. Alternatively this window may be accessed via Tools > Output Messages.

../_images/gpu_trace_troubleshooting_red_flag.001.png

Crash Reporting

When an application crashes, or hangs, a crash report can be one of the most valuable pieces of information in helping to fix the issue. Accordingly, if you have the ability to send a crash report, it would be greatly appreciated.

Automatic Crash Reports

Nsight Graphics’s (host and target) are configured to automatically send crash reports when they encounter a crash. Submitting via the dialog is a good approach, but saving the minidump for explicit communication can be useful too.

Note

If you encounter a crash and do not have the option of sending a crash report, you may need to instead generate a crash report manually, as described below. One typical cause reason that crash reports might not be generated is if the application is configured with its own automated crash reporting that overrides the Nsight Graphics crash reporting mechanism.

Manual Crash Reports

Manual crash reports are an effective approach to collecting information in case you are finding that automatic crash reports are not triggering. A process dump be collected by attaching to the crashing process with a debugger and manually creating a dump in the case of a crash.

Windows

A crash dump can be created by Microsoft Visual Studio. To accomplish this:

  1. Start Visual Studio.

  2. Follow the instructions for Debugging Your Application with a Debugger.

  3. Start the application with Nsight Graphics.

  4. Attach the Visual Studio debugger to it with “Debug > Attach To Process”

  5. When you encounter a crash, use the Visual Studio “Debug > Save Dump As” menu option.

Linux

A crash dump can be created by GDB, the GNU Debugger. To accomplish this:

  1. Start gdb.

  2. Follow the instructions for Debugging Your Application with a Debugger.

  3. Start the application with Nsight Graphics.

  4. Attach gdb to it.

  5. When you encounter a crash, use the “generate-core-file” command.

  6. Next, while the process is still alive, use the core2md utility to translate the core file into a dump that can be consued by running: core2md <core dump> /proc/<crash process ID>/ <minidump>

    1. The core2md utility can be found in the Nsight Graphics installation directory under host/linux-desktop-nomad-x64.

Manual Hang Reports

If the application encounters a hang, a process dump can be one of the most effective ways to identify the source of the hang. A process dump be collected by attaching to the crashing process with a debugger and manually creating a dump by following the instructions below:

Windows

A crash dump can be created by Microsoft Visual Studio. To accomplish this:

  1. Start Visual Studio.

  2. Attach the Visual Studio debugger to the hanging process with “Debug > Attach to Process”

  3. Stop program execution by using the Visual Studio “Debug > Break All” command.

  4. Generate a process dump using the Visual Studio “Debug > Save Dump As” command.

Linux

A crash dump can be created by GDB, the GNU Debugger. To accomplish this:

  1. Start gdb.

  2. Attach gdb to your process.

  3. The process should been stopped after being attached by GDB, otherwise press Ctrl + C to send a SIGINT to stop the process.

  4. Generate a process dump using the “generate-core-file” command.

  5. Next, while the process is still alive, use the core2md utility to translate the core file into a dump that can be consued by running: core2md <core dump> /proc/<hang process ID>/ <minidump>

    1. The core2md utility can be found in the Nsight Graphics installation directory under host/linux-desktop-nomad-x64.

Debugging Your Application with a Debugger

Although launching your application with Nsight Graphics might appear to be an alternative to CPU debugging, the application that is launched is still very much a debuggable application. This can be useful to determine if a problem you are encountering is in your own code by tracing the paths taken by your application.

To do this, set an environment variable of NVIDIA_PROCESS_INJECTION_ATTACH_DIALOG=1 and attach a debugger when you see a message box. Click OK to resume your application once you have set breakpoints that will allow you to inspect if your application is following the expected paths.

Collecting DirectX Debug Logging

Sometimes a device lost or other issue can be narrowed by observing what the DirectX debug layer has to say.

If you need to install the layer it should be part of the OS in Windows 10:

Apps & Features > Manage Optional Features > Graphics Tools

Open dxcpl, which should look like the below. Make sure your installed application is in the Scope List, and set the Direct3D/DXGI Debug Layer to Force On.

../_images/directx_control_panel.png

There are two ways to see the spew:

  1. You can see logging without attaching Visual Studio by just running DbgView.exe. https://docs.microsoft.com/en-us/sysinternals/downloads/debugview.

  2. Alternately, attach using Visual Studio. Logging will be in the Visual Studio Output window.

Setting an environment variable

There are occasionally times where you might be asked to set an undocumented variable to help disambiguate problems.

Apply the environment variable in the connection dialog Environment setting when starting an application.

Common Problems

Problem: Application Fails to Launch

You’ve tried to launch your application, but it is failing to launch.

Possible Causes

  1. Incorrect command line arguments.

  2. Incorrect working directory.

  3. You’re trying to launch on a remote machine that does not have the Nsight Monitor running.

Possible Solutions

Make sure that your command line arguments and working directory are as expected.

If you are trying to run on a remote machine, please ensure that the remote monitor is running and that the name of the machine is correct. See Remote Launching.

Disambiguate if the application is launching at all. Follow the instructions in Debugging Your Application with a Debugger. Check to see if your application is launched at all and if so, whether it is following its expected path. If the application doesn’t launch at all, please send an email to devtools-support@nvidia.com .

Problem: Application Crashes at Runtime

You’ve found that your application appears to launch, but it crashes during runtime.

Possible Causes

  1. Lack of API support by Nsight

  2. Application not checking return codes from device/object creation, assuming it has succeeded

  3. Interception-library crash

  4. Internal-driver crash

  5. D3D-debug runtime interaction

Possible Solutions

Try disabling the following features:

For D3D apps, try running without the D3D debug runtime enabled, as the debug runtime occasionally differs in behavior when compared with the release runtime.

If none of the above works, please try to collect a crash dump if possible and send it to devtools-support@nvidia.com .

Problem: Application Hangs at Runtime

You’ve found that your application appears to launch, but it hangs during runtime.

Possible Causes

  1. Multi-threading issue

  2. HUD Issue

Possible Solutions

Try disabling the following features:

If none of the above works, please try to collect a process dump if possible and send it to devtools-support@nvidia.com .

Problem: Application Crashes during Capture

You’ve found that you’re able to run the application successfully, but upon trying to perform a live analysis, the application crashes.

Possible Causes

  1. Multi-threading issue

  2. Out of memory

  3. The application is tearing itself down due to a watchdog timeout

Possible Solutions

Try disabling the following features:

If you suspect a multi-threading issue (D3D’s runtime sometimes indicates this), try disabling multi-threaded capture.

If Nsight Graphics reports out of memory, trying reducing the requirements of the application or try running with a more capable GPU.

If the application exits without any clear sign of a crash, the application could be tearing itself down. Please contact devtools-support@nvidia.com with your concern and we will investigate if there is any opportunity for deactivating the thread.

Problem: Application Hangs during Capture

You’ve found that you’re able to run the application successfully, but upon trying to perform a live analysis, the application hangs. This hang sometimes appears as a white screen on the target application.

Possible Causes

  1. The application is lazily presenting frames, preventing progress

  2. Multi-threading issue

  3. App is running in fullscreen mode

Possible Solutions

If the application is lazily presenting frames, it may prevent capture progress given that Nsight performs work on frame boundaries. If this is the case, try turning on the Force Repaint feature so as to force the application to present frames.

If you suspect a multi-threading issue, try changing the following feature to RenderOnly:

If none of the above works, please try to collect a process dump if possible and send it to devtools-support@nvidia.com .

Problem: Application Encounters an Incompatibility

This problem arises when the application you are running is using API methods or patterns that are not supported by Nsight

Possible Causes

  1. An unsupported API method was used

  2. An unsupported API pattern was used

Possible Solutions

When encountering this issue, Nsight will present a list of API methods or reasons that it has encountered as incompatible. This listing is listed alongside an explanation of the reasons why Nsight has prevented capture, which include application crashes, incorrect data, etc. Because Nsight is a replay based debugger, the absence of methods may lead to critical issues as replay is attempted. In some cases, however, the missing methods are innocuous and replay may proceed correctly without them. When capturing through the host, Nsight will offer the user an opportunity to capture despite these incompatibilities. From this point, it is up to the user to determine if the data is meaningful.

When encountering an incompatibility, we recommend that you communicate this incompatibility to devtools-support@nvidia.com so that the Nsight development team may track this issue and determine if it is something that will be supported in the future.

Note that if you wish to ignore all incompatibilities on every run, and wish to accept the possible errors that come with it, you may set the option of ‘Troubleshooting > Ignore Incompatibilities’ to accomplish that.

Problem: Application Captures Successfully, but Exits after a Time in Capture

This problem indicates that you have had some level of success, but even if the application generally inactive, the application crashes.

Possible Causes

  1. Serving a host query leads to a crash

  2. Memory leak

  3. Watchdog timer

Possible Solutions

When encountering this issue, take note of what you are doing when you encounter if. The first thing to try is doing nothing – does the application still crash when doing so? If there is nothing going on, this is either a memory leak or a watchdog timer.

  1. Look at the memory usage of the process – is it growing? It’s a memory leak, either from the application or the tool.

  2. Set a stopwatch to count how long it takes to crash – is it a “round” number like 30 or 60 seconds? It’s probably a watchdog.

  3. If this is a memory leak (uncommon but possible) please contact support to help identify the issue.

If this is a watchdog issue, disable the watchdog in your application.

Problem: Application Runs Extremely Slowly

You’ve observed that the application runs at a significantly lower rate than normal operation.

Possible Causes

  1. Too much work is being done

  2. The application may be exercising uncommon paths

Possible Solutions

Try disabling optional features, such as collecting shader sourcecollecting native shaders, or collecting hardware performance metrics.

Problem: D3D12 Replayer Shows More CPU Overhead than Expected

If you encounter more overhead in your generated C++ capture, conservative synchronization may be the problem.

Possible Causes

  1. Nsight’s default fence syncing policy may be too conservative for this application

Possible Solutions

Try experimenting with replay fence behavior.

Problem: I Can’t Capture a Vulkan Application

If you find that the button to Capture for Live Analysis is disabled, or you do not see a message that your application has Nsight analysis enabled, the Nsight Vulkan layer may not be enabled. This symptom is often accompanied by an error in the Output Messages window, so look for errors in that window for an indication of the failure.

Possible Causes

  1. The Nsight Vulkan layer configuration has been removed from your system configuration

Possible Solutions

One workaround is to re-enable the Nsight Vulkan layer explicitly. To do this, follow the steps for your system:

Windows

  1. run <install directory>/host/windows-desktop-nomad-x64/VK_LAYER_NV_nomad.bat

Linux

  1. Check the existence of vulkan manifest file (.json) under <install directory>/target/linux-desktop-nomad-x64/NomadVulkanLayer/vulkan/implicit_layer.d/

  2. Delete any dangling manifest of uninstalled Nsight Graphics in ~/.local/share/vulkan/implicit_layer.d

  3. If you still see the issue after above steps, then it’s could be a Nsight Graphics bug and please report it to us. Meanwhile, one possible workaround is to add environment variable at the “Connect to process” dialog: XDG_DATA_DIRS=<install directory>/target/linux-desktop-nomad-x64/NomadVulkanLayer

If, after repeating the steps, you find that your system still cannot capture, gather a log of the output of the vulkaninfo application from the Vulkan SDK and send it to devtools-support@nvidia.com.

Problem: I Can’t Attach to the Application

The application launches, but you are unable to attach to it with the Nsight Graphics host.

Possible Causes

  1. You launched a piece of the process hierarchy without Nsight Graphics.

  2. You set the connection to automatically attach when the root application launches child processes that are the actual processes of interest.

  3. The application is interfering with the interception of Nsight Graphics, preventing it from intercepting.

  4. The application is using a software renderer.

Possible Solutions

Nsight Graphics is essentially an in-process debugger, so it cannot attach to an application that wasn’t originally launched through Nsight Graphics. The attach feature is meant to be used to attach to applications that have been launched through other means (e.g., a command line launcher), as well as to allow for some recoverability in the case of a host issue, as it allows you attach at a later time.

Make sure to kill any processes related to the process hierarchy of an application and try to launch it again.

Problem: The Host UI Crashes

The host UI crashes while you are analyzing an application.

Possible Causes

  1. UI Bug

Possible Solutions

Try reducing the number of views that you have open when running to pinpoint which view causes the issue.

If at all possible, try to collect a crash dump of the UI application and send it to devtools-support@nvidia.com.

Try deleting the UI persistence data with Help > Reset Application Data .

Problem: The Target Window Blocks the Host Window

While running a live analysis, you find that the target window is blocking the host window and interfering with the analysis you wish to perform on the host. This is most often reported on machines that do not have access to multiple monitors.

Possible Causes

  1. The application has fullscreen settings

  2. The application has a topmost flag set to keep the application on top

Possible Solutions

We suggest running without fullscreen or topmost settings. If fullscreen-like behavior is desired, many applications support a borderless window mode.

If you must analyze an application with these characteristics, and you do not have access to a second monitor, the virtual desktop or workspaces support on most modern operating system shells presents an effective path forward. Creating one desktop for the target application and one for the host often avoids the target from interfering. For more information on using these features, see one of the articles below.

NOTE: If you wish to suppress the dialog that reports replay window interference, set an environment variable of NSIGHT_REPORT_REPLAY_WINDOW_INTERFERENCE=0 .

Windows: https://blogs.windows.com/windowsexperience/2015/04/16/virtual-desktops-in-windows-10-the-power-of-windowsmultiplied/

Linux/Gnome: https://help.gnome.org/users/gnome-help/stable/shell-workspaces.html.en

Problem: Force-failed QueryInterface is Reported

It is possible that applications will attempt to QueryInterface for types that Nsight does not know about or understand. To avoid crashes, incorrect rendering, or bad data with these unknown types, Nsight will report a force-failed QueryInterface warning. After reporting this warning, Nsight will nullify the result of this QueryInterface call and return E_NOINTERFACE to report that this interface is unsupported.

Possible Causes

  1. Using an older version of Nsight against an application that uses newer runtime capabilities

  2. Using multiple tools that intercept the application at one time

  3. Lack of API support by Nsight

Possible Solutions

When this issue is encountered, it is recommended that you first attempt to understand what the source of the incompatibility is. Nsight will attempt to print out the source and target types in the QueryInterface call. When the target is unknown, however, this type will be printed out as a GUID.

In some cases, the failure may be apparent, and you might be able to do a text search to determine where your application is making this problematic QueryInterface call. If that is too difficult to find, you may also try Debugging Your Application with a Debugger and setting a function breakpoint on MessageBoxA before running the application, which will report the call stack in which Nsight performs the report.

If you are unable to workaround this type support, you may attempt to set an environment variable to suppress this force-failed query. Note that this is not guaranteed to fix all concerns, and may result in future unspecified failures, but it is available as a possibility for working around problems. The environment variable of NSIGHT_PASSTHROUGH_UNKNOWN_GUIDS is a comma-delimited list of GUIDs to allow to passthrough without a force-failure. GUIDs must be fully specified with a brace syntax, as in NSIGHT_PASSTHROUGH_UNKNOWN_GUIDS={5b746c30-24e2-4385-81f6-39f7a068945b}.

If you suspect that the type being reported should be supported by Nsight, please send a report to devtools-support@nvidia.com to ask for assistance.

Appendix

Feature Support Matrix

Table 20. Nsight Graphics feature matrix

Feature

D3D11

D3D12

OpenGL

Vulkan

Frame Capture and Live Analysis

Yes

Yes

Yes

Yes

C++ Capture

Yes

Yes

Yes

Yes

Shader Profiling

Yes

Yes

Pixel History

Yes

Yes

Yes

Yes

Dynamic Shader Editing

Yes

Yes

Yes

Yes

GPU Trace

Yes

Yes

Yes

Ray Tracing Debugging

Yes

Yes

Nsight Aftermath GPU Crash Dumps

Yes

Yes

Supported OpenGL Functions

Nsight Graphics’s Frame Debugger supports the set of OpenGL operations, which are defined by the OpenGL 4.5 core profile. Note that it is not necessary to create a core profile context to make use of the frame debugger. An application that uses a compatibility profile context, but restricts itself to using the OpenGL 4.5 core subset, will also work. A few OpenGL 4.5 compatibility profile features, such as support for alpha testing and a default vertex array object, are also supported.

The Frame Debugger supports three classes of OpenGL extensions, described below.

1. OpenGL Core Context Support

The OpenGL extensions listed below are supported in as much as the extension has been adopted by the OpenGL 4.5 core profile. For example, EXT_subtexture is included as part of OpenGL 1.1. Calls to glTexSubImage2DEXT are supported and behave the same as calls to glTexSubImage2D. On the other hand, while EXT_vertex_array is also included as part of OpenGL 1.1, glColorPointerEXT is not supported by the Frame Debugger. The operation of glColorPointerEXT was modified when it was included as part of OpenGL 1.1. Additionally, glColorPointer is part of the compatibility subset, but not the core subset.

// GL 1.1
EXT_vertex_array
EXT_polygon_offset
EXT_blend_logic_op
EXT_texture
EXT_copy_texture
EXT_subtexture
EXT_texture_object
// GL 1.2
EXT_texture3D
EXT_bgra
EXT_packed_pixels
EXT_rescale_normal
EXT_separate_specular_color
SGIS_texture_edge_clamp
SGIS_texture_lod
EXT_draw_range_elements
EXT_color_table
EXT_color_subtable
EXT_convolutionHP_convolution_border_modes
SGI_color_matrix
EXT_histogram
EXT_blend_color
EXT_blend_minmax
EXT_blend_subtract
// GL 1.2.1
EXT_SGIS_multitexture
// GL 1.3
ARB_texture_compression
ARB_texture_cube_map
ARB_multisample
ARB_multitexture
ARB_texture_env_add
ARB_texture_env_combine
ARB_texture_env_dot3
ARB_texture_border_clamp
ARB_transpose_matrix
// GL 1.4
SGIS_generate_mipmap
NV_blend_square
ARB_depth_texture
ARB_shadow
EXT_fog_coord
EXT_multi_draw_arrays
ARB_point_arameters
EXT_secondary_color
EXT_blend_func_separate
EXT_stencil_wrap
EXT_texture_env_crossbar
EXT_texture_lod_bias
ARB_texture_mirrored_repeat
ARB_window_pos
// GL 1.5
ARB_vertex_buffer_object
ARB_occlusion_query
EXT_shadow_funcs
// GL 2.0
ARB_shader_objects
ARB_vertex_shader
ARB_fragment_shader
ARB_draw_buffers
ARB_texture_non_power_of_two
ARB_point_sprite
EXT_blend_equation_separate
ATI_separate_stencil
EXT_stencil_two_side
// GL 2.1
ARB_pixel_buffer_object
EXT_direct_state_access
EXT_texture_sRGB
// GL 3.0
EXT_gpu_shader4
NV_conditional_render
APPLE_flush_buffer_range
ARB_color_buffer_float
NV_depth_buffer_float
ARB_texture_float
EXT_packed_float
EXT_texture_shared_exponent
EXT_framebuffer_object
NV_half_float
ARB_half_float_pixel
EXT_framebuffer_multisample
EXT_framebuffer_blit
EXT_texture_integer
EXT_texture_array
EXT_packed_depth_stencil
EXT_draw_buffers2
EXT_texture_compression_rgtc
EXT_transform_feedback
APPLE_vertex_array_object
EXT_framebuffer_sRGB
// GL 3.1
EXT_draw_instanced
ARB_draw_instanced
ARB_copy_buffer
NV_primitive_restart
ARB_texture_buffer_object
ARB_texture_rectangle
ARB_uniform_buffer_object
// GL 3.2
ARB_vertex_array_bgra
ARB_draw_elements_base_vertex
ARB_fragment_coord_conventions
ARB_provoking_vertex
ARB_seamless_cube_map
ARB_texture_multisample
ARB_depth_clamp
ARB_geometry_shader_4
ARB_sync
// GL 3.3
ARB_shader_bit_encoding
ARB_blend_func_extended
ARB_explicit_attrib_location
ARB_occlusion_query2
ARB_sampler_objects
ARB_texture_rgb10_a2ui
ARB_texture_swizzle
ARB_timer_query
ARB_instanced_arrays
ARB_vertex_type_2_10_10_10_rev
// GL 4.0
ARB_texture_query_lod
ARB_draw_buffers_blend
ARB_draw_indirect
ARB_gpu_shader5
ARB_gpu_shader_fp64
ARB_sample_shading
ARB_shader_subroutine
ARB_tessellation_shader
ARB_texture_buffer_object_rgb32
ARB_texture_cube_map_array
ARB_texture_gather
ARB_transform_feedback2
ARB_transform_feedback3
// GL 4.1
ARB_ES2_compatibility
ARB_get_program_binary
ARB_separate_shader_objects
ARB_shader_precision
ARB_vertex_attrib_64bit
ARB_viewport_array
// GL 4.2
ARB_texture_compression_bptc
ARB_compressed_texture_pixel_storage
ARB_shader_atomic_counters
ARB_texture_storage
ARB_transform_feedback_instanced
ARB_base_instance
ARB_shader_image_load_store
ARB_conservative_depth
ARB_shading_language_420pack
ARB_internalformat_query
ARB_map_buffer_alignment
// GL 4.3
ARB_multi_draw_indirect
ARB_program_interface_query
ARB_shader_storage_buffer_object
ARB_copy_image
ARB_vertex_attrib_binding
ARB_texture_view
ARB_invalidate_subdata
ARB_framebuffer_no_attachments
ARB_stencil_texturing
ARB_explicit_uniform_location
ARB_texture_storage_multisample
ARB_program_interface_query
ARB_robust_buffer_access_behavior
ARB_ES3_compatibility
ARB_clear_buffer_object
ARB_internal_format_query2
ARB_texture_buffer_range
ARB_compute_shader
ARB_debug_group
ARB_debug_label
ARB_debug_output
// GL 4.4
ARB_query_buffer_object
ARB_enhanced_layouts
ARB_multi_bind
ARB_vertex_type_10f_11f_11f_rev
ARB_texture_mirror_clamp_to_edge
ARB_clear_texture
// GL 4.5
ARB_clip_control
ARB_cull_distance
ARB_conditional_render_inverted
GL_KHR_context_flush_control
ARB_get_texture_sub_image
GL_KHR_robustness
ARB_texture_barrier
ARB_ES3_1_compatibility
ARB_direct_state_access
ARB_shader_texture_image_samples
ARB_derivative_control

2. Other Supported OpenGL Extensions

The second class of OpenGL extensions is listed below. These extensions are not part of OpenGL 4.5 core or compatibility, but are fully supported by the frame debugger target. Context and object state, which is added by these extensions, may not be displayed by the host UI.

ARB_framebuffer_object
EXT_texture_filter_anisotropic
NV_buffer_store
ARB_vertex_attrib_binding
ARB_multi_draw_indirect
NV_gpu_multicast
ARB_parallel_shader_compile
ARB_seamless_cubemap_per_texture
NV_shader_buffer_load
NV_vertex_buffer_unified_memory

3. Partially Supported OpenGL Extensions

The third class of OpenGL extensions are ones for which there is partial support. These extensions are listed below.

ARB_bindless_texture
WGL_ARB_extensions_string
WGL_ARB_pixel_format
WGL_EXT_extensions_string
WGL_EXT_swap_control
WGL_EXT_swap_control_tear
WGL_ARB_create_context

4. OpenGL Immediate Mode

Beyond the core functionality and extensions, a selection of immediate-mode functions is supported.

glBegin
glEnd
glVertex*
glColor*
glIndex*
glNormal*
glTexCoord*
glDrawElement
glEnableClientState
glDisableClientState
glVertexPointer
glColorPointer
glSecondaryColorPointer
glIndexPointer
glNormalPointer

Supported Vulkan Functions

Nsight Graphics™ 2023.4 frame debugging supports all of Vulkan 1.3.267.

Additionally, the follow extensions to Vulkan 1.3.267 are supported:

VK_EXT_4444_formats
VK_EXT_acquire_xlib_display
VK_EXT_astc_decode_mode
VK_EXT_attachment_feedback_loop_dynamic_state
VK_EXT_attachment_feedback_loop_layout
VK_EXT_blend_operation_advanced
VK_EXT_border_color_swizzle
VK_EXT_buffer_device_address
VK_EXT_calibrated_timestamps
VK_EXT_color_write_enable
VK_EXT_conditional_rendering
VK_EXT_conservative_rasterization
VK_EXT_custom_border_color
VK_EXT_debug_marker
VK_EXT_debug_report
VK_EXT_debug_utils
VK_EXT_depth_clamp_zero_one
VK_EXT_depth_clip_control
VK_EXT_depth_clip_enable
VK_EXT_depth_range_unrestricted
VK_EXT_descriptor_buffer
VK_EXT_descriptor_indexing
VK_EXT_device_address_binding_report
VK_EXT_device_fault
VK_EXT_device_memory_report
VK_EXT_direct_mode_display
VK_EXT_discard_rectangles
VK_EXT_display_surface_counter
VK_EXT_extended_dynamic_state
VK_EXT_extended_dynamic_state2
VK_EXT_extended_dynamic_state3
VK_EXT_external_memory_host
VK_EXT_filter_cubic
VK_EXT_fragment_density_map
VK_EXT_fragment_shader_interlock
VK_EXT_full_screen_exclusive
VK_EXT_global_priority
VK_EXT_graphics_pipeline_library
VK_EXT_hdr_metadata
VK_EXT_headless_surface
VK_EXT_host_query_reset
VK_EXT_image_2d_view_of_3d
VK_EXT_image_compression_control
VK_EXT_image_compression_control_swapchain
VK_EXT_image_robustness
VK_EXT_image_sliced_view_of_3d
VK_EXT_image_view_min_lod
VK_EXT_index_type_uint8
VK_EXT_inline_uniform_block
VK_EXT_legacy_dithering
VK_EXT_line_rasterization
VK_EXT_load_store_op_none
VK_EXT_memory_budget
VK_EXT_memory_priority
VK_EXT_mesh_shader
VK_EXT_multi_draw
VK_EXT_multisampled_render_to_single_sampled
VK_EXT_mutable_descriptor_type
VK_EXT_nested_command_buffer
VK_EXT_non_seamless_cube_map
VK_EXT_opacity_micromap
VK_EXT_pageable_device_local_memory
VK_EXT_pci_bus_info
VK_EXT_physical_device_drm
VK_EXT_pipeline_creation_cache_control
VK_EXT_pipeline_creation_feedback
VK_EXT_pipeline_library_group_handles
VK_EXT_pipeline_protected_access
VK_EXT_post_depth_coverage
VK_EXT_primitive_topology_list_restart
VK_EXT_primitives_generated_query
VK_EXT_private_data
VK_EXT_provoking_vertex
VK_EXT_queue_family_foreign
VK_EXT_rasterization_order_attachment_access
VK_EXT_rgba10x6_formats
VK_EXT_robustness2
VK_EXT_sample_locations
VK_EXT_sampler_filter_minmax
VK_EXT_scalar_block_layout
VK_EXT_separate_stencil_usage
VK_EXT_shader_atomic_float
VK_EXT_shader_atomic_float2
VK_EXT_shader_demote_to_helper_invocation
VK_EXT_shader_image_atomic_int64
VK_EXT_shader_module_identifier
VK_EXT_shader_object
VK_EXT_shader_stencil_export
VK_EXT_shader_subgroup_ballot
VK_EXT_shader_subgroup_vote
VK_EXT_shader_tile_image
VK_EXT_shader_viewport_index_layer
VK_EXT_subgroup_size_control
VK_EXT_subpass_merge_feedback
VK_EXT_surface_maintenance1
VK_EXT_swapchain_colorspace
VK_EXT_swapchain_maintenance1
VK_EXT_texel_buffer_alignment
VK_EXT_texture_compression_astc_hdr
VK_EXT_tooling_info
VK_EXT_transform_feedback
VK_EXT_validation_cache
VK_EXT_validation_features
VK_EXT_validation_flags
VK_EXT_vertex_attribute_divisor
VK_EXT_vertex_input_dynamic_state
VK_EXT_video_encode_h264
VK_EXT_video_encode_h265
VK_EXT_ycbcr_2plane_444_formats
VK_EXT_ycbcr_image_arrays
VK_KHR_16bit_storage
VK_KHR_8bit_storage
VK_KHR_acceleration_structure
VK_KHR_android_surface
VK_KHR_bind_memory2
VK_KHR_buffer_device_address
VK_KHR_cooperative_matrix
VK_KHR_copy_commands2
VK_KHR_create_renderpass2
VK_KHR_dedicated_allocation
VK_KHR_deferred_host_operations
VK_KHR_depth_stencil_resolve
VK_KHR_descriptor_update_template
VK_KHR_device_group
VK_KHR_device_group_creation
VK_KHR_display
VK_KHR_display_swapchain
VK_KHR_draw_indirect_count
VK_KHR_driver_properties
VK_KHR_dynamic_rendering
VK_KHR_external_fence
VK_KHR_external_fence_capabilities
VK_KHR_external_fence_fd
VK_KHR_external_fence_win32
VK_KHR_external_memory
VK_KHR_external_memory_capabilities
VK_KHR_external_memory_fd
VK_KHR_external_memory_win32
VK_KHR_external_semaphore
VK_KHR_external_semaphore_capabilities
VK_KHR_external_semaphore_fd
VK_KHR_external_semaphore_win32
VK_KHR_format_feature_flags2
VK_KHR_fragment_shader_barycentric
VK_KHR_fragment_shading_rate
VK_KHR_get_display_properties2
VK_KHR_get_memory_requirements2
VK_KHR_get_physical_device_properties2
VK_KHR_get_surface_capabilities2
VK_KHR_global_priority
VK_KHR_image_format_list
VK_KHR_imageless_framebuffer
VK_KHR_incremental_present
VK_KHR_maintenance1
VK_KHR_maintenance2
VK_KHR_maintenance3
VK_KHR_maintenance4
VK_KHR_map_memory2
VK_KHR_multiview
VK_KHR_pipeline_executable_properties
VK_KHR_pipeline_library
VK_KHR_portability_enumeration
VK_KHR_present_id
VK_KHR_present_wait
VK_KHR_push_descriptor
VK_KHR_ray_query
VK_KHR_ray_tracing_maintenance1
VK_KHR_ray_tracing_pipeline
VK_KHR_ray_tracing_position_fetch
VK_KHR_relaxed_block_layout
VK_KHR_sampler_mirror_clamp_to_edge
VK_KHR_sampler_ycbcr_conversion
VK_KHR_separate_depth_stencil_layouts
VK_KHR_shader_atomic_int64
VK_KHR_shader_clock
VK_KHR_shader_draw_parameters
VK_KHR_shader_float16_int8
VK_KHR_shader_float_controls
VK_KHR_shader_integer_dot_product
VK_KHR_shader_non_semantic_info
VK_KHR_shader_subgroup_extended_types
VK_KHR_shader_subgroup_uniform_control_flow
VK_KHR_shader_terminate_invocation
VK_KHR_shared_presentable_image
VK_KHR_spirv_1_4
VK_KHR_storage_buffer_storage_class
VK_KHR_surface
VK_KHR_surface_protected_capabilities
VK_KHR_swapchain
VK_KHR_swapchain_mutable_format
VK_KHR_synchronization2
VK_KHR_timeline_semaphore
VK_KHR_uniform_buffer_standard_layout
VK_KHR_variable_pointers
VK_KHR_video_decode_h264
VK_KHR_video_decode_h265
VK_KHR_video_decode_queue
VK_KHR_video_encode_queue
VK_KHR_video_queue
VK_KHR_vulkan_memory_model
VK_KHR_wayland_surface
VK_KHR_win32_keyed_mutex
VK_KHR_win32_surface
VK_KHR_workgroup_memory_explicit_layout
VK_KHR_xcb_surface
VK_KHR_xlib_surface
VK_KHR_zero_initialize_workgroup_memory
VK_NVX_binary_import
VK_NVX_image_view_handle
VK_NVX_multiview_per_view_attributes
VK_NV_clip_space_w_scaling
VK_NV_compute_shader_derivatives
VK_NV_cooperative_matrix
VK_NV_copy_memory_indirect
VK_NV_corner_sampled_image
VK_NV_coverage_reduction_mode
VK_NV_dedicated_allocation
VK_NV_dedicated_allocation_image_aliasing
VK_NV_device_diagnostic_checkpoints
VK_NV_device_diagnostics_config
VK_NV_device_generated_commands
VK_NV_external_memory
VK_NV_external_memory_capabilities
VK_NV_external_memory_win32
VK_NV_fill_rectangle
VK_NV_fragment_coverage_to_color
VK_NV_fragment_shader_barycentric
VK_NV_fragment_shading_rate_enums
VK_NV_framebuffer_mixed_samples
VK_NV_geometry_shader_passthrough
VK_NV_glsl_shader
VK_NV_linear_color_attachment
VK_NV_low_latency
VK_NV_memory_decompression
VK_NV_mesh_shader
VK_NV_present_barrier
VK_NV_ray_tracing
VK_NV_ray_tracing_invocation_reorder
VK_NV_ray_tracing_motion_blur
VK_NV_representative_fragment_test
VK_NV_sample_mask_override_coverage
VK_NV_scissor_exclusive
VK_NV_shader_image_footprint
VK_NV_shader_sm_builtins
VK_NV_shader_subgroup_partitioned
VK_NV_shading_rate_image
VK_NV_viewport_array2
VK_NV_viewport_swizzle

The follow extensions to Vulkan 1.3.267 are not currently supported. If your application uses these extensions, please send a feedback feature request to let the Nsight team know about your interest and needs.

VK_EXT_acquire_drm_display
VK_EXT_application_parameters
VK_EXT_depth_bias_control
VK_EXT_directfb_surface
VK_EXT_display_control
VK_EXT_dynamic_rendering_unused_attachments
VK_EXT_external_memory_acquire_unmodified
VK_EXT_external_memory_dma_buf
VK_EXT_fragment_density_map2
VK_EXT_frame_boundary
VK_EXT_global_priority_query
VK_EXT_host_image_copy
VK_EXT_image_drm_format_modifier
VK_EXT_metal_objects
VK_EXT_metal_surface
VK_EXT_pipeline_properties
VK_EXT_pipeline_robustness
VK_KHR_maintenance5
VK_KHR_object_refresh
VK_KHR_performance_query
VK_KHR_portability_subset
VK_NV_acquire_winrt_display
VK_NV_descriptor_pool_overallocation
VK_NV_device_generated_commands_compute
VK_NV_displacement_micromap
VK_NV_extended_sparse_address_space
VK_NV_external_memory_rdma
VK_NV_external_memory_sci_buf
VK_NV_external_sci_sync
VK_NV_external_sci_sync2
VK_NV_inherited_viewport_scissor
VK_NV_low_latency2
VK_NV_optical_flow
VK_NV_private_vendor_info
VK_NV_win32_keyed_mutex

Supported NVAPI Functions

Nsight Graphics’s Frame Debugger supports a large set of NVAPI functions. The list of functions are the following:

NvAPI_GetErrorMessage
NvAPI_GetInterfaceVersionString
NvAPI_D3D11_AliasMSAATexture2DAsNonMSAA
NvAPI_D3D11_BeginUAVOverlap
NvAPI_D3D11_BeginUAVOverlapEx
NvAPI_D3D11_CreateCubinComputeShader
NvAPI_D3D11_CreateCubinComputeShaderWithName
NvAPI_D3D11_CreateDevice
NvAPI_D3D11_CreateDeviceAndSwapChain
NvAPI_D3D11_CreateDomainShaderEx
NvAPI_D3D11_CreateFastGeometryShader
NvAPI_D3D11_CreateFastGeometryShaderExplicit
NvAPI_D3D11_CreateGeometryShaderEx_2
NvAPI_D3D11_CreateHullShaderEx
NvAPI_D3D11_CreateMetaCommand
NvAPI_D3D11_CreatePixelShaderEx_2
NvAPI_D3D11_CreateRasterizerState
NvAPI_D3D11_CreateSamplerState
NvAPI_D3D11_CreateShadingRateResourceView
NvAPI_D3D11_CreateVertexShaderEx
NvAPI_D3D11_DecompressView
NvAPI_D3D11_EndUAVOverlap
NvAPI_D3D11_EnumerateMetaCommands
NvAPI_D3D11_ExecuteMetaCommand
NvAPI_D3D11_GetResourceHandle
NvAPI_D3D11_InitializeMetaCommand
NvAPI_D3D11_IsFatbinPTXSupported
NvAPI_D3D11_IsNvShaderExtnOpCodeSupported
NvAPI_D3D11_MultiDrawIndexedInstancedIndirect
NvAPI_D3D11_MultiDrawInstancedIndirect
NvAPI_D3D11_MultiGPU_GetCaps
NvAPI_D3D11_MultiGPU_Init
NvAPI_D3D11_RSGetPixelShadingRateSampleOrder
NvAPI_D3D11_RSSetExclusiveScissorRects
NvAPI_D3D11_RSSetPixelShadingRateSampleOrder
NvAPI_D3D11_RSSetShadingRateResourceView
NvAPI_D3D11_RSSetViewportsPixelShadingRates
NvAPI_D3D11_SetDepthBoundsTest
NvAPI_D3D11_SetNvShaderExtnSlot
NvAPI_D3D11_SetNvShaderExtnSlotLocalThread
NvAPI_D3D12_BuildRaytracingAccelerationStructureEx
NvAPI_D3D12_BuildRaytracingOpacityMicromapArray
NvAPI_D3D12_CopyTileMappings
NvAPI_D3D12_CreateCommittedResource
NvAPI_D3D12_CreateComputePipelineState
NvAPI_D3D12_CreateCubinComputeShader
NvAPI_D3D12_CreateCubinComputeShaderWithName
NvAPI_D3D12_CreateDDisplayPresentBarrierClient
NvAPI_D3D12_CreateGraphicsPipelineState
NvAPI_D3D12_CreateHeap
NvAPI_D3D12_CreateHeap2
NvAPI_D3D12_CreateMetaCommand
NvAPI_D3D12_CreatePresentBarrierClient
NvAPI_D3D12_CreateReservedResource
NvAPI_D3D12_EmitRaytracingOpacityMicromapArrayPostbuildInfo
NvAPI_D3D12_EnumerateMetaCommands
NvAPI_D3D12_ExecuteMetaCommand
NvAPI_D3D12_GetGraphicsCapabilities
NvAPI_D3D12_GetNeedsAppFPBlendClamping
NvAPI_D3D12_GetRaytracingCaps
NvAPI_D3D12_InitializeMetaCommand
NvAPI_D3D12_IsFatbinPTXSupported
NvAPI_D3D12_IsNvShaderExtnOpCodeSupported
NvAPI_D3D12_NotifyOutOfBandCommandQueue
NvAPI_D3D12_QueryCpuVisibleVidmem
NvAPI_D3D12_QueryModifiedWSupport
NvAPI_D3D12_QueryPresentBarrierSupport
NvAPI_D3D12_QuerySinglePassStereoSupport
NvAPI_D3D12_RegisterPresentBarrierResources
NvAPI_D3D12_ReservedResourceGetDesc
NvAPI_D3D12_ResourceAliasingBarrier
NvAPI_D3D12_SetAsyncFrameMarker
NvAPI_D3D12_SetCreatePipelineStateOptions
NvAPI_D3D12_SetDepthBoundsTestValues
NvAPI_D3D12_SetModifiedWMode
NvAPI_D3D12_SetNvShaderExtnSlotSpace
NvAPI_D3D12_SetNvShaderExtnSlotSpaceLocalThread
NvAPI_D3D12_SetSinglePassStereoMode
NvAPI_D3D12_UpdateTileMappings
NvAPI_D3D1x_BindSwapBarrier
NvAPI_D3D1x_DisableShaderDiskCache
NvAPI_D3D1x_GetGraphicsCapabilities
NvAPI_D3D1x_JoinSwapGroup
NvAPI_D3D1x_Present
NvAPI_D3D1x_QueryFrameCount
NvAPI_D3D1x_QueryMaxSwapGroup
NvAPI_D3D1x_QuerySwapGroup
NvAPI_D3D1x_ResetFrameCount
NvAPI_D3D_BeginResourceRendering
NvAPI_D3D_ConfigureAnsel
NvAPI_D3D_EndResourceRendering
NvAPI_D3D_GetCurrentSLIState
NvAPI_D3D_GetLatency
NvAPI_D3D_GetObjectHandleForResource
NvAPI_D3D_GetSleepStatus
NvAPI_D3D_ImplicitSLIControl
NvAPI_D3D_InitializeSMPAssist
NvAPI_D3D_IsGSyncActive
NvAPI_D3D_IsGSyncCapable
NvAPI_D3D_QueryModifiedWSupport
NvAPI_D3D_QueryMultiViewSupport
NvAPI_D3D_QuerySMPAssistSupport
NvAPI_D3D_QuerySinglePassStereoSupport
NvAPI_D3D_RegisterDevice
NvAPI_D3D_SetFPSIndicatorState
NvAPI_D3D_SetLatencyMarker
NvAPI_D3D_SetModifiedWMode
NvAPI_D3D_SetMultiViewMode
NvAPI_D3D_SetReflexSync
NvAPI_D3D_SetResourceHint
NvAPI_D3D_SetSinglePassStereoMode
NvAPI_D3D_SetSleepMode
NvAPI_D3D_SetVerticalSyncMode
NvAPI_D3D_Sleep
NvAPI_DestroyPresentBarrierClient
NvAPI_JoinPresentBarrier
NvAPI_LeavePresentBarrier
NvAPI_QueryPresentBarrierFrameStatistics
NvAPI_DISP_AcquireDedicatedDisplay
NvAPI_DISP_GetNvManagedDedicatedDisplays
NvAPI_DISP_ReleaseDedicatedDisplay
NvAPI_OGL_ExpertModeDefaultsGet
NvAPI_OGL_ExpertModeDefaultsSet
NvAPI_OGL_ExpertModeGet
NvAPI_OGL_ExpertModeSet

Unsupported Captures

Nsight Graphics maintains a list of the unsupported functions or operations that are used by the application. If an unsupported operation is encountered, an unsupported capture will be reported. This unsupported capture guards against crashes or incorrect results coming from known limitations.

In some cases, however, these unsupported operations might not impact any analysis that follows. Accordingly, after warning about the risks of an unsupported capture, Nsight Graphics will offer the opportunity to proceed despite this warning. If the user proceeds, Nsight Graphics will continue into capture on a best-effort basis.

If you determine that this unsupported operation is innocuous, and you wish to turn it off completely, you may suppress this warning via the Ignore Incompatibilities option. Note that this will prevent you from being notified of future incompatibilities, however, so please use with caution.

Update Notification

Nsight Graphics can check for a new version and notify the user of any updates. There are 2 options available for controlling this feature, found in the Environment tab of the Tools > Options view.

../_images/VersionUpdateOptions.png

By default, Nsight Graphics checks for updates every time the app is started. This can be changed by selecting “No” for the “Check for updates at startup” option. With this option disabled, Nsight Graphics will still check for updates every 3 days.

Update notifications can be completely disabled by setting the “Show version update notifications” value to “No”.

If the automatic checking feature is disabled, the user can still check for updates by selecting the Help > Check for updates… menu option.

Microsoft Visual Studio Integration

NVIDIA Nsight Integration is a Visual Studio extension that allows you to access the power of Nsight Graphics from within Visual Studio.

When Nsight Graphics is installed along with NVIDIA Nsight Integration, Nsight Graphics activities will appear under the Nsight menu in the Visual Studio menu bar. These activities launch Nsight Graphics with the current project settings and executable, allowing you to reuse all of the settings without manually copying any setting over. You can even set keybindings to launch sessions with a specified Nsight Graphics activity. When you use multiple Nsight tools, such as Nsight Systems or Nsight Compute, you will see independent commands for each of them, greatly simplifying your workflow.

For more information about using Nsight Graphics from within Visual Studio, please visit: