Linux

Remote debugging on Linux is supported through SSH connections. Enter your SSH information when establishing the connection to connect to the target machine.

Perfmarkers are integral to nearly all workflows. We recommend that your application always run with perfmarkers when running under tools analysis.

Performance markers are most commonly used to delineate sections of events and note where in your application they begin and end. They can also be nested to show sub-sections of events. Perf markers are generally used to measure the amount of time that an inner portion of algorithm will take.

There are three different types of perf markers that are supported in Nsight Graphics:

1. D3D9 perrmarkers are supported.
2. Direct3D11.1 and higher applications support ID3DUserDefinedAnnotation. (For more information, see ID3DUserDefinedAnnotation interface on MSDN.)
3. OpenGL applications use the KHR_debug group, glPushDebugGroup and glPopDebugGroup .
4. NVTX tools extension.

The Frame Debugger activity offers a comprehensive set of tools for discovering problems with your application's rendering or general operation. This activity enables the inspection of events, API state, resource values, and dependencies to understand where your application might have issues. Use this activity when:

The Frame Debugger activity supports all APIs that are generally supported by Nsight Graphics.

To start this activity, select Frame Debugger from the connection dialog.

The basic workflow for the Frame Debugger activity is to capture an application and then navigate the events, data, and resources that your application is submitting/using to identify your issue.

Whether you are debugging on the CPU or GPU, the first step of any debugging process is to narrow in on the set of data that you need to analyze to understand your problem. Generally, this means that you will want to scrub to a particular event of interest in either the Scrubber or the Event Viewer. Because Nsight Graphics™ will show you the rendering contribution of every draw call, looking at either the HUD or the Current Target View will give you an indication of where your rendering might be going wrong. Another alternative is to use the Pixel History experiment to automatically identify the draw calls that relate to a particular texture update.

From there, you will want to use your knowledge of the graphics pipeline to try to understand what might be causing a problem. Some questions to ask yourself:

In some cases, there may be a combination of problems that exacerbate a given problem. Isolating the symptoms can be challenging, but an effective use of the tools can offer increased confidence that you are heading in the right direction.

The profiling activity provides detailed performance information for all units of the GPU. Use this activity when:

The profiling activity currently supports profiling D3D11, D3D12, and OpenGL applications.

While C++ captures can be collected in while Frame Debugging, the C++ capture activity provides a focused activity to streamline the creation of captures. Non-necessary analysis subsystems are turned off in order to allow for the quickest and more robust application capture. This activity is an excellent way to save a snapshot of your application, frozen in time. Use this activity when:

The Generate C++ Capture activity supports all APIs that are generally supported by Nsight Graphics.

To start this activity, select Generate C++ Capture from the connection dialog.

Once the application is running, the Generate C++ Capture button will be available on the main toolbar.

Once a capture is started, the target application will temporarily pause, and a progress dialog will be shown detailing the steps of the export to C++ process. When complete, the C++ project is written to the disk and the application will resume.

By default, the save directory is co-located beside the current project. If no project is currently loaded the default save directory is used (see Options > Environment > Default Documents Folder).

In addition to the C++ project, the code generation process also produces an nsight-gfxcppcap file with additional information and utilities. These nsight-gfxcppcap files are automatically associated with the current project and can be reopened later.

The additional features of an nsight-gfxcppcap file include:

1. Screenshot of the capture taken from the original application
2. Information about the captured application and its original system
3. Statistics about the captured API stream
4. Utilities to build the C++ capture without opening the generated Visual Studio project
5. Utilities to launch the compiled application:
   1. The Execute button will launch the compiled executable.
   2. The Connect... button will populate a new connection dialog that allows you to run a specific activity on the generated capture.
6. User comments that are persisted within this file.

Using a Saved Capture

1. To use the saved capture, open the saved project from the Captures directory in Visual Studio. Saving a capture will generate the source code, as well as project and solution files for all supported Visual Studio versions.

   Versioned Visual Studio projects may be opened with Visual Studio versions that do not match the project version, but note that you may be asked to upgrade your project when opening it for the first time.

2. These solution files contain a number of generated source files.
   1. Main.cpp — This is where all of the initialization code is called, resources are created, and each frame portion is called in a message loop.
   2. ResourcesNN.cpp — Depending on the number of resources to be created, there will be multiple ResourcesNN.cpp files, each with a CreateResourcesNN call in them, that will construct all of the resources (device(s), textures, shaders, etc.) that are used in the scene. These are called in Main.cpp before replaying the frame in the message loop.
   3. FrameSetup.cpp — This file contains all of the state setting calls to set the API state to the proper values for the beginning of the frame, including what buffers are bound, which shaders are enabled, etc.
   4. FrameNPartMM.cpp — In Direct3D and single-threaded OpenGL captures, these files contain the API functions, each named RunFrameNPartMM(), to replay the frame. It is split into multiple files so generated code is easier to work with. These functions are called sequentially in the message loop in Main.cpp.

      In this scenario, both N and MM are placeholders for numbers in the multiple files generated. FrameN will typically be Frame0 since only a single frame is captured, and PartMM will typically be in the 00-05 range, depending on how many API calls are in the frame.

   5. ThreadLLFrameNPartMM.cpp — In multi-threaded OpenGL captures, these files contain the API functions, each named ThreadLLRunFrameNPartMM(), to replay the frame. The functions correspond to the work done by each thread during the frame. These functions are called by their respective threads and synchronized to replay the saved events in the same order as captured.
   6. ReadOnlyDatabase.cpp — This is a helper class to access resource data that is stored in the data.bin file. It is accessed throughout the code via the GetResource() call.
   7. Helpers.cpp — These functions are used throughout the replayer for various conversions and access to the ReadOnlyDatabase.
   8. Threading.cpp — This file contains helper functions and classes to manage threads used in the project.
3. Build and run the project.

Changing a Resource

If you want to change a resource (for example, to swap in a different texture), you can change the parameters for the construction by looking within the ResourcesNN.cpp files for the texture in question. Textures can be matched by size and/or format. Once you find the variable for the texture, look for that name in the FrameSetup.cpp file. This will contain source lines to lock the texture, call GetResource() to retrieve the data from the ReadOnlyDatabase, and then call memcpy(…) to link the data to the texture. You can substitute the call to the ReadOnlyDatabase with a call to read from a file of choice to load the alternate texture.

Changing a Draw Call

If you want to change the state for a given draw call, you can locate the draw call by replaying the capture within Nsight Graphics and scrubbing to find the call you want to examine. Search in the FrameNPartMM.cpp files for Draw NN, where NN is the 0-based draw call index that Nsight Graphics displayed on the scrubber. Doing this will bring you to the source line for that draw call, and from here, you can add any state changes before that call. Alternatively, you can also disable that specific call by commenting out the source call containing the draw call.

Parameters

• -repeat N — This setting enables Nsight Graphics to use serialized captures in the normal arch workflow. The N setting indicates the number of times to repeat the entire capture; the default setting is -1, which keeps the capture running on an infinite loop.
• -noreset — This setting controls whether context state and all resources are reset to their beginning of frame value. When this setting is specified, all frame restoration operations will be skipped, avoiding the performance cost associated with them. Note that this may introduce rendering errors if the rendered frame has a data dependency on the results of a previous frame.

The GPU Trace activity provides detailed performance information for SM units of the GPU as well as identifying asynchronous compute usage and opportunities. Use this activity when:

The GPU Trace activity currently supports profiling D3D12 applications.

To start this activity, select GPU Trace from the connection dialog.

1. Set up your application for connection (see How to Launch and Connect to Your Application for more information.).

   

2. Set a Frame Count. This parameter defines how many frames will be captured. The maximum value is 5.

   Note: GPU Trace consumes a lot of memory, especially in complex frames. You need to make sure that by capturing large number of frames, there is enough memory to consume it all.

3. Launch or attach to your application. (See How to Launch and Connect to Your Application for more information.) 
4. If the application successfully connected, the process name will appear in the lower right corner on the window.

   

   Note: Currently, it is only possible to run GPU Trace when the host and target are running on the same machine.

5. Once launched and connected, click Generate GPU Trace Capture (or select it from the GPU Trace menu) to create a capture report file.

   

   Note: It is recommended that you close the application after capturing, in order to free up your system's memory while exploring the captured file.

The Launch tab enables launching applications for analysis. This is where you will add the basic process information to launch and subsequently connect to the application you wish to analyze.

This tab has the following controls:

Application Executable - Specifies the root application to launch. Note that this may not be the final application that you wish to analyze; see this section on how to launch different types of applications.

Working Directory - The directory in which the application will be launched

Command Line Arguments - specify the arguments to pass the application executable.

Environment - the environment variables to set in the launched application

Automatically Connect - specifies whether the launched application should be automatically connected to. If the launched application is a launcher that creates the process that you ultimately wish to analyze, set this to 'No'.

To attach to an application, it must have previously been launched through the launch tab. This page will list the launched application as well as any children that the application has launched.

Nsight Graphics allows for adjusting the activity with a large set of options.

Nsight Graphics offers several tools for real-time performance analysis and debugging.

The API inspector is a common view to all supported APIs that offers an exhaustive look at all of the state that is relevant to a particular event to which the capture analysis is scrubbed.

To access this view, go to Frame Debugger > API Inspector.

While the view is common, the state within it is particular to each API. See the section below that relates to your API of interest.

The API Statistics View is a high-level view of important API calls, and includes information to help you see where GPU and CPU time is spent.

To access this view, go to Frame Debugger > API Statistics.

The Current Target view is used to show the currently bound output targets. This can be useful because it focuses in on the bound output resources, rather than having to search for them in the Resources view.

To access this view, go to Frame Debugger > Current Target.

Current Target will display thumbnails along the left pane for all currently bound color, depth, and stencil targets. This view will change as you scrub from event to event. All of the thumbnails on the left can be selected to show a larger image on the right. You can also click the link below each to open the target in the Resources View.

The Geometry view takes the state of the Direct3D, OpenGL, or Vulkan machine, along with the parameters for the current draw call, and shows pre-transformed geometry.

To access this View, go to Frame Debugger > Geometry.

There are two views into this data: a graphical view and a memory view.

Graphical Tab

Attribute Options

• Position — Specifies the vertex attribute to use for positional geometry data.
• Color — Specifies how to color the geometry. If Diffuse Color is selected, the selected diffuse color swatch will be used for coloring. If a vertex attribute is selected, the selected attribute will be used for per-vertex coloring.

  

• Normal — Specifies the per-vertex normal. This selection applies when using a shade mode that specifies Normal Attribute or when rendering normal vectors.

Rendering Options

Clicking Configure in the bottom right corner of the Geometry View will open up the rendering options menu.

• Reset Camera — Resets the camera to its default orientation. By default, the viewer bounds all geometry with a bounding sphere for optimal orientation.
• Render Mode — Determines how to render and raster geometry.
  • Solid: renders filled geometry.
  • Points: renders a vertex point cloud.

    

  • Wireframe: renders a wireframe of the geometry.
  • Wireframe + Solid: renders filled geometry with a wireframe on top of it.
• Shade Mode — Specifies the lighting mode of the rendered image.
  • Selected Color Attribute: Shades with the specified color attribute
  • Flat Shading Using Generated Normals: Renders the geometry using flat shading with calculated normals
  • Flat Sharing Using Normal Attribute: Renders the geometry using flat shading with the specified Normal Attribute.
  • Smooth Shading Using Normal Attribute: Renders the geometry using smooth shading with the specified Normal Attribute.
• Render Normal Vectors — Renders the specified normal attribute as a vector pointing from each vertex. The vector may be colored by the Normal Color selection and may be scaled by the Normal Scale selection.

  

Memory Tab

The Memory tab of the Geometry View shows the contents of the vertex buffer, as interpreted by the current vertex or input attribute specification. This view is useful for seeing the raw data of your draw call. An additional capability of this view is that it highlights invalid or corrupt vertices to streamline finding problematic data.

There are two modes of display for the geometry data:

1. Index Buffer Order shows the vertices as indexed by the current index buffer and current draw call.

   

2. Vertex Buffer Order shows the vertices as linearly laid out from the start of the vertex buffer and draw call specification.

   

The Range Profiler is a powerful tool that can help you determine how sections of your frame utilize the GPU, and give you direction to optimize the rendering of your application. Once you have captured a frame, the Range Profiler displays your frame broken down into a collection of ranges, or groups of contiguous actions. For each action, you can see the GPU execution times for each, as well as detailed GPU hardware statistics across all of the units in the GPU. The Range Profiler also includes unmatched data mining capabilities that allow you to group calls in the frame into ranges based on various criteria that you choose.

To access this view, go to Frame Debugger > Range Profiler.

The Nsight Graphics Frame Debugger has two parts. One part appears as the Frame Debugger window on the host. The other part appears as a Heads-Up Display (HUD) on the target application.

To access this view, go to Frame Debugger > Scrubber.

The part of the Frame Debugger that appears as a HUD on the target machine is comprised of the following:

• HUD Toolbar — controls the frame capture, along with a number of other options (help, etc.).
• Frame Scrubber — indicates the current draw event. There is a scrubber view in the Frame Debugger on the host, as well as a frame scrubber on the HUD. The frame scrubber controls stay in synch with each other, meaning that when you move the controls on one, it affects the other. For example, if you move the frame scrubber on the HUD to highlight a new draw event, the scrubber on the Frame Debugger moves in synch to do likewise.

Understanding the Frame Scrubber

For the sake of discussion when it comes to graphics debugging, it helps to note some common terminology.

• An event is a single call to the API. It could be a triangle draw call, or backbuffer clear, or a less obvious call, like configuring buffers. A snapshot is a sequence of events.
• An action is a subset of the event types. It can be one of the following: (1) Draw Call, (2) Clear, or (3) Dispatch. Actions are interesting since they explicitly change data which may result in visual changes.

When you debug your graphics project, the Scrubber window shows the perf markers you implemented. When working with user-defined markers, the Scrubber window will use the color and label that you defined for the perf marker.

On the Scrubber, you can select one performance marker and it will automatically create a range of all of the draw calls that occurred within that time frame. Clicking on it again will cause the scrubber to automatically zoom to that range of events. You can zoom in on a nested/child marker the same way.

To zoom out, click the parent performance marker, or use CTRL + mouse wheel.

Performance markers are also displayed on the HUD, color-coded the same way that they are on the Scrubber. However, on the HUD, the information is condensed, and you must hover your mouse over the selected performance marker to get its details.

The default view will show the events in your application, in addition to any performance markers you have defined. Clicking the Add... button will open a dialog that allows you to select what type of range you want to add.

• Program Ranges — Actions that use the same shader program.
• Viewport — Actions that render to the same viewport rectangle.
• Alpha Blending Enabled — Actions that have alpha blending enabled.
• Alpha Test Enabled — Actions that have alpha test enabled.
• Back Face Cull Enabled — Actions that have back face cull enabled.
• User — A range defined by you on the fly. Use SHIFT + left-click and drag the scrubber on the created "User" row to create a new range.

Right-clicking on a specific action in the Scrubber will allow you to open the API Inspector for that action, change your view settings, or initiate a profile session with the Range Profiler.

Scrubber View Options

From the Mode drop-down menu, choose one of the following:

• Event ID -- Unit Scale is the default view, which simply shows the actions and events on the timeline.
• Sequence ID -- Unit Scale shows the sequence of events on the timeline.
• Event ID -- GPU Time Scale displays the GPU activity and how much each event or action cost the GPU.
• Event ID -- CPU Time Scale displays the CPU activity and how much each event or action cost the CPU.
• Event ID -- X by CPU, Y by GPU displays the CPU time scale on a horizontal X-axis, and the GPU time scale on a vertical Y-axis.

Depending on which mode you select, you can also select whether you want to view the ruler relative to the capture, viewport, or cursor.

From the Hierarchy drop-down, Queue Centric sorts the events by queue, while Thread Centric sorts the events by the thread.

Using Hotkeys to Scrub Through a Frame

When the scrubber has focus, you can use the following hotkeys to move the scrubber cursor from one event to another.

For the purpose of moving the scrubber cursor, the following are considered action events:

• Draw methods
• Clear methods
• Dispatch methods
• Present methods

For example, if you are looking for the next draw method that was called, you can press the CTRL + RIGHT ARROW on the keyboard to skip over events that are not typically of interest, and only stop on events that are considered action events.

The Resources View allows you to see all of the available resources in the scene.

To access this view, go to Frame Debugger > Resources.

To open the Resources page, go to Frame Debugger > Resources. There are two tabs available here:

1. Graphical
2. Memory

At the top of the Resources view, you'll find a toolbar:

• Clone — makes a copy of the current view, so that you can open another instance.
• Lock — freezes the current view so that changing the current event does not update this view. This is helpful when trying to compare the state or a resource at two different actions.
• Save — saves the captured resources to disk.
• Red, Green, and Blue — toggles on and off specific colors.
• Alpha — enables alpha visualization. In the neighboring drop-down, you can select one of the following two options:
  • Blend — blends the alpha with a checkerboard background.
  • Grayscale — alpha values are displayed as grayscale.
• Flip Image — inverts the image of the resource displayed.

Below the toolbar is a set of buttons, described below, for high-level filtering of the resources based on type. Next to that, there is a drop-down menu that allows you to select how you wish to view the resources: thumbnails, small thumbnails, tiles, or details.

If you select the Details view, you can sort the resources by the available column headings (type, name, size, etc.).

The Graphical tab allows you to inspect the resource, pan using the left mouse button to click and drag, zoom using the mouse wheel, and inspect pixel values. Also, this is where you can save the resource to disk. If supported on your GPU and API, this is also where you can initiate a Pixel History session to get all of the contributing fragments for a given pixel.

When you have selected a buffer from the left pane, the Show Histogram button will be available on the right side of the Graphical tab, which allows for remapping the color channels for the resource being viewed.

To modify the histogram view, the following options are available:

• You can set the minimum and maximum cutoff values via the sliders under the histograms, or by typing in values in the Minimum and Maximum boxes.
• You can change the scale by using the Log button.
• The Luminance button allows you to visualize luminance instead of color values.
• The Normalize button can preset the minimum and maximum values to the extents of the data in the resource.

The Memory tab shows a dump of the resource data.

You can use multiple options to configure how this memory is displayed:

• The Axis drop-down changes between address (memory offset) and index (array element) views.
• The Offset entry limits the view to an offset within the given resource.
• The Extent entry limits the view to a maximum extent within the given resource.
• The Precision spin box controls the number of decimal places to show for floating point entries.
• The Hex Display toggles between decimal (base-10) and hex (base-8) display formats.
• Hash shows a hash value representative of the given memory resource within the current offset and extent. This is useful for comparing memory objects or sub-regions.
• The Transpose button swaps the rows and columns of the data representation.
• The Configure button opens the Structured Memory Configuration dialog.

            int;      // creates a column with an anonymous int
            int x;    // creates a second column with an int named x
            float y;  // creates a third column with a float named y

            struct MyType{ int x; float y;};
            struct MyOtherType{ MyType z; double u; };

The Linked Programs View lists all of the shaders in your application.

To access this view, go to Frame Debugger > Linked Programs.

• If the shader (or its parent program or pipeline object) hasn’t been used by the application yet, it will show up with the symbol in the Status column.
• If the shader has been used, but the statistics are being calculated, the symbol will be displayed in the Status column.

For programs or pipeline objects, you can view the individual shaders by pressing the ► button to the left of the program/pipeline name. The list also contains a number of statistics:

Name

This is the name of the shader. This name is either generated internally, or can be assigned by the user per API.

Status

This column displays the current status of the shader. The status includes Source or Binary, to denote whether or not source code is available for this shader. Also, if the µCode text is included, this means that we have driver level binary code that is necessary for gathering shader performance metrics.

The symbol means that we are waiting for the shader to be bound by the application.

The symbol means that shader performance metrics are currently being computed.

Cycles

This value is the absolute cost for a single primitive (vertex, tessellation control point, fragment, etc.) to execute through the shader. The value takes into account latency for memory accesses, but it does not take branching or loops into account. The values are summed up at the program level to show the absolute cycle count for a single fragment to be rendered on the screen.

Avg Cycles

Since primitives are submitted in large groups, this gives an average cycle cost for a single primitive, assuming it is submitted as a larger block of work (for instance, many fragments from the same object with the shame shader). The values are summed up at the program level.

ALU/TEX Inst Ratio

This gives the ratio of ALU to texture instructions for the shader. The values are averaged at the program level.

ALU/TEX Cycle Ratio

Since not all ALU calls are the same cost in terms of cycles, this value gives the ratio of cycles spent in ALU versus texture instructions. The values are averaged at the program level.

Regs

This column gives the number of registers used by the program. Register count impacts occupancy/threads in flight so if the value gets too high you will get closer to the Cycles value than the Avg Cycles value.

LMem (Bytes)

This is the number of bytes of local memory used by the shader. Similar to registers, this can impact occupancy and contribute to a lower overall throughput of primitives running this shader.

Selected Program(s) & Shader(s)

This pane will show extended shader performance information about the currently selected shader. If multiple shaders are selected, then you can view and compare the statistics between the selected shaders.

The Acceleration Structure View shows the geometry that has been specified in build commands when running an application that uses ray tracing APIs. If the application does not use these APIs, the view will not be available.

To access this View, go to Frame Debugger > Acceleration Structure.

The view is dual-paned -- it shows a hierarchical view of the acceleration structure on the left, with a graphical view of the structure on the right.

With the hierarchy of the acceleration structure view, the top-level acceleration structure (TLAS) and bottom-level acceleration structure (BLAS) are presented. When a particular TLAS or BLAS is selected, the name, flags, and other meta-data for this structure are listed in a section on the bottom left-hand side. Each item within the tree has a checkbox that allows for cycling through one of full rendering, wireframe rendering, or no rendering of that level of the hierarchy.

Below the rendering pane, information on the camera position and direction are presented. Each of these controls is editable to navigate the scene. The view uses WASD or up, down, left, right keys to change the position. Holding Shift while navigating increases the navigation speed. Clicking with the mouse and dragging allows for additional navigation. Additionally, items in the hierarchy can be double-clicked to place that item in view. To vertically flip the camera, select the double-arrow button. To reset the camera at any time, click Reset Camera.

The VR Inspector view allows you to inspect how your application is using VR APIs. It will be available when an application is captured with a supported API. Supported APIs include Oculus (LibOVR) and OpenVR.

To access this view, go to Frame Debugger > VR Inspector.

Once opened, this view is context-specific to the VR API in use. See the sections below for a discussion on each API.

The additional features of an nsight-gfxcppcap file include:

1. Screenshot of the capture taken from the original application
2. Information about the captured application and its original system
3. Statistics about the captured API stream
4. Utilities to build the C++ capture without opening the generated Visual Studio project
5. Utilities to launch the compiled application:
   1. The Execute button will launch the compiled executable.
   2. The Connect... button will populate a new connection dialog that allows you to run a specific activity on the generated capture.
6. User comments that are persisted within this file.

The GPU Trace window is comprised of 3 sections:

1. Scrubber
2. Information Tabs
3. Events table

GPU Trace Scrubber

The GPU Trace scrubber is the main component in which you can observe the captured frame data.

GPU Occupancy

The GPU Occupancy row shows the occupancy of the hardware stages, in terms of warps. This shows the total warps' execution on the GPU. The warps may be grouped and colored according to stages, marker, actions, or Command Lists. By default, the warps' color is determined by stage (e.g., Vertex, Geometry, Hull, Domain, Fragment, Compute, Pixel, Compute, and Async Compute shaders).

When hovering your mouse over the scrubber, a tooltip will appear that displays the percentage of the warps' occupancy by stage, per the specific time.

GPCxTPC

The GPCxTPC row is a semantic grouping of the graphics processing units. Each GPU has multiple GPCs (Graphics Processing Clusters). Each GPC has numerous logical Texture Processing Clusters, or TPCs. The TPCs are numbered.

TPC#N blocks from each GPC in the GPU serve as a logical group for warp executions.

For example: if you have 1 GPU with 2 GPCs, and 3 TPCs per GPC, you’ll get the following logical groupings:

• GPCxTPC0 holds GPC0TPC0, GPC1TPC0
• GPCxTPC1 holds GPC0TPC1, GPC1TPC1
• GPCxTPC2 holds GPC0TPC2, GPC1TPC2

SM Active

The SM Active row shows a "flattened" view of the warps, in terms of total occupancy for each SM. If an individual SM has at least one warp in it, then it is counted as active. If all SMs have at least one active warp, then this graph would show a value 100% the height of the row. All graphics operations are shown in green, while all compute operations are shown in orange or pink.

Graphics/Compute Idle

The Graphics/Compute Idle metric represents the percentage of cycles spent idle in the Graphics Engine of the GPU, meaning the GPU is not performing any warp's execution for graphics and compute. The Graphics Engine services 3D, 2D, I2M, and Compute processing hardware. The GPU is composed of several engines, including Copy, Video, Display, and Security. As such, this metric does not account for any GPU work included in that list. Idle Graphics/Compute time may indicate:

1. The workload is CPU-bound. This could occur when the CPU is not feeding commands fast enough to the GPU Front End (FE), so the FE has no work to process.
2. DX12 Wait calls on fences in the Graphics/Compute engine (see Synchronization and Multi-Engine).

Frames

The Frames row helps detect warps per frame. When selecting a frame, only the warps related to the selected frame will be highlighted. When hovering your mouse over the frame, a tooltip summarizing the warps' activity will appear.

Command Lists

This row shows the captured Command List per queue. The Command List ID is displayed on the Command List bar. By hovering with the mouse, a tooltip summarizing the warp's activity will appear. Clicking on a Command List will highlight all of the warps which are part of that Command List.

Incremental Actions

Incremental Actions represent actions by how much extra time is required to execute them on a GPU. If an action can execute inside the time of another action (i.e., perfect parallelism), then it would have no width on this line.

Actions

An action is a subset of the event types. It can be one of the following:

1. Draw call
2. Clear
3. Dispatch

Actions are of interest because they explicitly change data, which may result in visual changes on the scrubber.

The Actions row shows the captured actions per queue. By hovering with the mouse, a tooltip summarizing the warp's activity will appear, as well as the action's ID. Clicking on the Action will highlight all of the warps which are part of that Action.

User Markers

GPU Trace also captures any user markers that exist in the application. This may help understand the frame workflow.

Scrubber Toolbar

At the top of the Scrubber view, there are 4 buttons that extend the Scrubber's capabilities.

View Options

The View button on the Scrubber toolbar allows you to change the way the data is presented, from Grouped by GPCxTPC to per SM and vice versa. Viewing the warp's occupancy by SM may give more information as to how the GPU is being utilized. This is how the view appears like when sorted per SM:

Color Options

Color by Markers

The Direct3D 12 API added the ability to add User markers. This helps us to understand the frame execution and debug it. GPU Trace captures user markers, and also allows you to color the warps according to the marker execution.

Color by Action

Choosing this option will color the warps according to which Action they belong to.

Color by Command List

Choosing this option will color the warps according to which Command List they belong to.

Export

GPU Trace allows you to export the warps data into a file in CSV format. This provides some flexibility in further calculation, if desired.

Zoom to Fit

Clicking this button will restore the original window zoom.

Annotations

While analyzing the captured data, you may want to add comments in certain locations. This can later serve as a reminder of where to look, or it may help if you wish to send the file to another user for further analysis.

To add an annotation, do the following:

1. Select a range within the Annotations row.
2. In the information section, the annotation tab will be opened.
3. The annotation has 2 sections which you can edit:
   1. Label
   2. Description

The annotation section behaves like any other component in the scrubber; you can select, hover to review the corresponding tooltip, or zoom.

Once you create an annotation, an asterisk will appear by the name of the file in the window tab:

You can use the File menu to Save (or Save As...) the file for future reference.

Using the Scrubber

For more information on using the Scrubber, see the Scrubber view.

Information Tabs

Summary Tab

The Summary tab shows important information on the captured data and the area that was selected in the Scrubber.

The main table shows information per shader type. You can uncheck the box next to a certain shader type to filter out those warps in the Scrubber view, and thus make it easier to understand the warp's occupancy.

• Range — Shows whether the data shown reflects the entire captured data or a user selection.
• Duration — The duration of the range selected.
• Start and End — Start and end times of the range selected.
• Num Warps — The number of warps in the range.
• Num GPCxTPC — The number of GPCxTPCs participating in the range's execution.
• Num SM — The number of SMs participating in the range's execution.
• Warp Active Time — The table shows time, percentage, and number of warps per stage. In this table, you can view the color scheme for each stage. The tables also allow you to select and deselect warps, according to which stage they belong to.

• SM Active time — This table shows how many SMs were active, in percentage, according to Graphics, Compute, and Async Compute for the selected range. This view can quickly show whether the GPU was idle unexpectedly longer then assumed for this range, which could indicate the application is CPU-bound.

Capture Information Tab

The Capture Information tab gives general information about the captured file. This might be useful when trying to analyze workload behavior or reproduce issues.

• Session Info — The Session Info section includes the process file name and location, as well as any command line arguments that were used when the application was run.
• System Info — This section lists the computer name, operating system, operating system build, and processor that was used.
• GPU Device — The GPU that was used when running the session.
• User Comments — This field can be changed or edited by the user. It is saved in the captured file for future reference. This can be very useful when collaborating with others on an application.

Annotations Tab

The Annotations tab is used for getting the data for a newly created annotation. For specifications, see the Annotations section.

Events Table

The Events Table summarizes the captured events according to types:

• Performance markers
• Draw Calls
• Command Lists
• Dispatches

The Events Table allows you to browse the various events, and sort them according to name, queue, duration, Incremental Cost, or frame.

The Events table correlates with the Scrubber. Selecting one of the events will automatically select the correlating event in the Scrubber, making it easy to find.

Throughout the operation of the tool, Nsight Graphics provides messages that inform on the status of operations as well as if any issues are encountered. This could provide some assistance when trying to determine why your application may not run, connect, or capture correctly. Error messages are indicated by a red flag in the bottom right of the application window. This flag may be double-clicked to open the Output Messages window. Alternatively this window may be accessed via Tools > Output Messages.

When an application crashes, a crash report can be one of the most valuable pieces of information in helping to fix the issue. Accordingly, if you have the ability to send a crash report, it would be greatly appreciated.

Although launching your application with Nsight Graphics might appear to be an alternative to CPU debugging, the application that is launched is still very much a debuggable application. This can be useful to determine if a problem you are encountering is in your own code by tracing the paths taken by your application.

To do this, set an environment variable of NVIDIA_PROCESS_INJECTION_ATTACH_DIALOG=1 and attach a debugger when you see a message box. Click ok to resume your application once you have set breakpoints that will allow you to inspect if your application is following the expected paths.

Sometimes a device lost or other issue can be narrowed by observing what the DirectX debug layer has to say.

If you need to install the layer it should be part of the OS in Windows 10:

Apps&features -> Manage optional features -> Graphics Tools

Then open dxcpl which should look like this. Make sure your installed application is in the Scope List and force on debugging.

2 ways to see the spew:

1. You can see logging without attaching VisualStudio by just running DbgView.exe. https://docs.microsoft.com/en-us/sysinternals/downloads/debugview
2. Alternately attach using Visual Studio. Logging will be in the VS Output window. See section below to enable attaching to the launched process.

There are occasionally times where you might be asked to set an undocumented variable to help disambiguate problems.

You've tried to launch your application, but it is failing to launch.

You've found that your application appears to launch, but it crashes during runtime.

You've found that your application appears to launch, but it hangs during runtime.

You've found that you're able to run the application successfully, but upon trying to performa a live analysis, the application crashes.

This problem indicates that you have had some level of success, but even if the application generally inactive, the application crashes.

You've observed that the application runs at a significantly lower rate than normal operation.

If you encounter more overhead in your range profiling session or generated C++ capture, conservative synchronization may be the problem.

The application launches, but you are unable to attach to it with the Nsight Graphics host.

The host UI crashes while you are analyzing an application.

Nsight Graphics's Frame Debugger supports the set of OpenGL operations, which are defined by the OpenGL 4.5 core profile. Note that it is not necessary to create a core profile context to make use of the frame debugger. An application that uses a compatibility profile context, but restricts itself to using the OpenGL 4.5 core subset, will also work. A few OpenGL 4.5 compatibility profile features, such as support for alpha testing and a default vertex array object, are also supported.

The Frame Debugger supports three classes of OpenGL extensions, described below.

1. OpenGL Core Context Support

The OpenGL extensions listed below are supported in as much as the extension has been adopted by the OpenGL 4.5 core profile. For example, EXT_subtexture is included as part of OpenGL 1.1. Calls to glTexSubImage2DEXT are supported and behave the same as calls to glTexSubImage2D. On the other hand, while EXT_vertex_array is also included as part of OpenGL 1.1, glColorPointerEXT is not supported by the Frame Debugger. The operation of glColorPointerEXT was modified when it was included as part of OpenGL 1.1. Additionally, glColorPointer is part of the compatibility subset, but not the core subset.

// GL 1.1
EXT_vertex_array
EXT_polygon_offset
EXT_blend_logic_op
EXT_texture
EXT_copy_texture
EXT_subtexture
EXT_texture_object

// GL 1.2
EXT_texture3D
EXT_bgra
EXT_packed_pixels
EXT_rescale_normal
EXT_separate_specular_color
SGIS_texture_edge_clamp
SGIS_texture_lod
EXT_draw_range_elements
EXT_color_table
EXT_color_subtable
EXT_convolutionHP_convolution_border_modes
SGI_color_matrix
EXT_histogram
EXT_blend_color
EXT_blend_minmax
EXT_blend_subtract

// GL 1.2.1
EXT_SGIS_multitexture

// GL 1.3
ARB_texture_compression
ARB_texture_cube_map
ARB_multisample
ARB_multitexture
ARB_texture_env_add
ARB_texture_env_combine
ARB_texture_env_dot3
ARB_texture_border_clamp
ARB_transpose_matrix

// GL 1.4
SGIS_generate_mipmap
NV_blend_square
ARB_depth_texture
ARB_shadow
EXT_fog_coord
EXT_multi_draw_arrays
ARB_point_arameters
EXT_secondary_color
EXT_blend_func_separate
EXT_stencil_wrap
EXT_texture_env_crossbar
EXT_texture_lod_bias
ARB_texture_mirrored_repeat
ARB_window_pos

// GL 1.5
ARB_vertex_buffer_object
ARB_occlusion_query
EXT_shadow_funcs

// GL 2.0
ARB_shader_objects
ARB_vertex_shader
ARB_fragment_shader
ARB_draw_buffers
ARB_texture_non_power_of_two
ARB_point_sprite
EXT_blend_equation_separate
ATI_separate_stencil
EXT_stencil_two_side

// GL 2.1
ARB_pixel_buffer_object
EXT_direct_state_access
EXT_texture_sRGB

// GL 3.0
EXT_gpu_shader4
NV_conditional_render
APPLE_flush_buffer_range
ARB_color_buffer_float
NV_depth_buffer_float
ARB_texture_float
EXT_packed_float
EXT_texture_shared_exponent
EXT_framebuffer_object
NV_half_float
ARB_half_float_pixel
EXT_framebuffer_multisample
EXT_framebuffer_blit
EXT_texture_integer
EXT_texture_array
EXT_packed_depth_stencil
EXT_draw_buffers2
EXT_texture_compression_rgtc
EXT_transform_feedback
APPLE_vertex_array_object
EXT_framebuffer_sRGB

// GL 3.1
EXT_draw_instanced
ARB_draw_instanced
ARB_copy_buffer
NV_primitive_restart
ARB_texture_buffer_object
ARB_texture_rectangle
ARB_uniform_buffer_object

// GL 3.2
ARB_vertex_array_bgra
ARB_draw_elements_base_vertex
ARB_fragment_coord_conventions
ARB_provoking_vertex
ARB_seamless_cube_map
ARB_texture_multisample
ARB_depth_clamp
ARB_geometry_shader_4
ARB_sync

// GL 3.3
ARB_shader_bit_encoding
ARB_blend_func_extended
ARB_explicit_attrib_location
ARB_occlusion_query2
ARB_sampler_objects
ARB_texture_rgb10_a2ui
ARB_texture_swizzle
ARB_timer_query
ARB_instanced_arrays
ARB_vertex_type_2_10_10_10_rev

// GL 4.0
ARB_texture_query_lod
ARB_draw_buffers_blend
ARB_draw_indirect
ARB_gpu_shader5
ARB_gpu_shader_fp64
ARB_sample_shading
ARB_shader_subroutine
ARB_tessellation_shader
ARB_texture_buffer_object_rgb32
ARB_texture_cube_map_array
ARB_texture_gather
ARB_transform_feedback2
ARB_transform_feedback3

// GL 4.1
ARB_ES2_compatibility
ARB_get_program_binary
ARB_separate_shader_objects
ARB_shader_precision
ARB_vertex_attrib_64bit
ARB_viewport_array

// GL 4.2
ARB_texture_compression_bptc
ARB_compressed_texture_pixel_storage
ARB_shader_atomic_counters
ARB_texture_storage
ARB_transform_feedback_instanced
ARB_base_instance
ARB_shader_image_load_store
ARB_conservative_depth
ARB_shading_language_420pack
ARB_internalformat_query
ARB_map_buffer_alignment

// GL 4.3
ARB_multi_draw_indirect
ARB_program_interface_query
ARB_shader_storage_buffer_object
ARB_copy_image
ARB_vertex_attrib_binding
ARB_texture_view
ARB_invalidate_subdata
ARB_framebuffer_no_attachments
ARB_stencil_texturing
ARB_explicit_uniform_location
ARB_texture_storage_multisample
ARB_program_interface_query
ARB_robust_buffer_access_behavior
ARB_ES3_compatibility
ARB_clear_buffer_object
ARB_internal_format_query2
ARB_texture_buffer_range
ARB_compute_shader
ARB_debug_group
ARB_debug_label
ARB_debug_output

// GL 4.4
ARB_query_buffer_object
ARB_enhanced_layouts
ARB_multi_bind
ARB_vertex_type_10f_11f_11f_rev
ARB_texture_mirror_clamp_to_edge
ARB_clear_texture

// GL 4.5
ARB_clip_control
ARB_cull_distance
ARB_conditional_render_inverted
GL_KHR_context_flush_control
ARB_get_texture_sub_image
GL_KHR_robustness
ARB_texture_barrier
ARB_ES3_1_compatibility
ARB_direct_state_access
ARB_shader_texture_image_samples
ARB_derivative_control

2. Other Supported OpenGL Extensions

The second class of OpenGL extensions is listed below. These extensions are not part of OpenGL 4.5 core or compatibility, but are fully supported by the frame debugger target. Context and object state, which is added by these extensions, may not be displayed by the host UI.

ARB_framebuffer_object
EXT_texture_filter_anisotropic
NV_buffer_store
ARB_vertex_attrib_binding
ARB_multi_draw_indirect
NV_gpu_multicast
ARB_parallel_shader_compile
ARB_seamless_cubemap_per_texture
NV_shader_buffer_load
NV_vertex_buffer_unified_memory

3. Partially Supported OpenGL Extensions

The third class of OpenGL extensions are ones for which there is partial support. These extensions are listed below.

ARB_bindless_texture
WGL_ARB_extensions_string
WGL_ARB_pixel_format
WGL_EXT_extensions_string
WGL_EXT_swap_control
WGL_EXT_swap_control_tear
WGL_ARB_create_context

Nsight Graphics™ 2018.7 frame debugging supports all of Vulkan 1.1.86, with the exception of functions and resources associated with sparse textures.

Additionally, the follow extensions are supported:

VK_AMD_gcn_shader
VK_AMD_gpu_shader_half_float
VK_AMD_negative_viewport_height
VK_AMD_rasterization_order
VK_AMD_shader_ballot
VK_AMD_shader_explicit_vertex_parameter
VK_AMD_shader_trinary_minmax
VK_EXT_blend_operation_advanced
VK_EXT_conditional_rendering
VK_EXT_conservative_rasterization
VK_EXT_debug_marker
VK_EXT_debug_report
VK_EXT_debug_utils
VK_EXT_depth_range_unrestricted
VK_EXT_descriptor_indexing
VK_EXT_discard_rectangles
VK_EXT_global_priority
VK_EXT_hdr_metadata
VK_EXT_post_depth_coverage
VK_EXT_queue_family_foreign
VK_EXT_sample_locations
VK_EXT_sampler_filter_minmax
VK_EXT_shader_stencil_export
VK_EXT_shader_subgroup_ballot
VK_EXT_shader_subgroup_vote
VK_EXT_shader_viewport_index_layer
VK_EXT_swapchain_colorspace
VK_EXT_validation_cache
VK_EXT_validation_flags
VK_EXT_vertex_attribute_divisor
VK_IMG_filter_cubic
VK_IMG_format_pvrtc
VK_KHR_16bit_storage
VK_KHR_8bit_storage
VK_KHR_android_surface
VK_KHR_bind_memory2
VK_KHR_create_renderpass2
VK_KHR_dedicated_allocation
VK_KHR_descriptor_update_template
VK_KHR_device_group
VK_KHR_device_group_creation
VK_KHR_display
VK_KHR_display_swapchain
VK_KHR_draw_indirect_count
VK_KHR_external_fence
VK_KHR_external_fence_capabilities
VK_KHR_external_memory
VK_KHR_external_memory_capabilities
VK_KHR_external_memory_fd
VK_KHR_external_semaphore
VK_KHR_external_semaphore_capabilities
VK_KHR_external_semaphore_fd
VK_KHR_external_memory_win32
VK_KHR_external_semaphore_win32
VK_KHR_get_memory_requirements2
VK_KHR_get_physical_device_properties2
VK_KHR_get_surface_capabilities2
VK_KHR_image_format_list
VK_KHR_incremental_present
VK_KHR_maintenance1
VK_KHR_maintenance2
VK_KHR_maintenance3
VK_KHR_multiview
VK_KHR_push_descriptor
VK_KHR_relaxed_block_layout
VK_KHR_sampler_mirror_clamp_to_edge
VK_KHR_sampler_ycbcr_conversion
VK_KHR_shader_draw_parameters
VK_KHR_shared_presentable_image
VK_KHR_storage_buffer_storage_class
VK_KHR_surface
VK_KHR_swapchain
VK_KHR_variable_pointers
VK_KHR_vulkan_memory_model
VK_KHR_wayland_surface
VK_KHR_win32_keyed_mutex
VK_KHR_win32_surface
VK_KHR_xcb_surface
VK_KHR_xlib_surface
VK_NV_clip_space_w_scaling
VK_NV_dedicated_allocation
VK_NV_device_diagnostic_checkpoints
VK_NV_external_memory_capabilitie
VK_NV_fill_rectangle
VK_NV_fragment_coverage_to_color
VK_NV_framebuffer_mixed_samples
VK_NV_geometry_shader_passthrough
VK_NV_geometry_shader_passthrough
VK_NV_glsl_shader
VK_NV_sample_mask_override_coverage
VK_NV_shader_subgroup_partitioned
VK_NV_viewport_array2
VK_NV_viewport_swizzle
VK_NVX_raytracing
VK_NV_compute_shader_derivatives
VK_NV_corner_sampled_image
VK_NV_fill_rectangle
VK_NV_fragment_shader_barycentric
VK_NV_representative_fragment_test
VK_NV_scissor_exclusive
VK_NV_shader_image_footprint
VK_NV_shading_rate_image