The first step for improving performance in a GPU bound application is to determine where you are spending GPU time in the rendering of the scene. This can be accomplished a number of ways using the Frame Debugger. First, adjust the scaling of the Scrubber to be based on GPU time.

This will allow you to see at a glance where the time is being spent on the frame. These ranges will show up in the Scrubber, also scaled by the amount of time the work executed within them takes. Finally, if you haven’t added debug ranges, you can use various criteria to create them on the fly in your debugging session, including render target sets, shader programs in use, etc.

Look for ranges that seem larger than expected, given what you are trying to accomplish in that section of the frame. Also, larger ranges/draw calls likely have more headroom for improvement, so they can be good places to start deeper investigation.

You can also see how much GPU time is spent on various actions and ranges in the Event Viewer. By sorting by GPU Time, you can quickly find the most expensive parts of the frame and begin your analysis from there.

Once you find an area you are interested in profiling, use the right mouse button context menu to initiate the Range Profiler. This will open up the profiler focused on the range or call you determined to be interesting. (Alternatively, you can open the Range Profiler through Frame Debugger > Range Profiler.)

Is my program CPU or GPU bound?

Try hotkey experiments such as minimum geometry and null scissor, to determine if you are GPU bound.

What are the most expensive draw calls in my application?

Capture a frame, and then run the Range Profiler. Once the Range Profiler is done running experiments, the entire scene will be selected by default. This will allow you to see details about all of the draw calls and dispatches in the scene. If you select Action Details in the Range Info section, you will see details on each draw call, including the execution time. Sort the table to time to see the most expensive draw call.

How can I optimize a range of draw calls?

In the Pipeline section, select Range Details and you will see an image with a virtual GPU pipeline. The red bars indicate units in the GPU that are not being used as efficiently as they could, so look for the higher bars to indicate where you need to spend time optimizing. (See below for specific tips on optimizing your API inputs for a particular unit).

How do I see collections of draw calls which share common state (like pixel shaders and vertex shaders)?

The Range Profiler contains a powerful grouping capability that allows you make new ranges based on common state. These include ranges based on program/shaders being used, viewport, render targets, and even user ranges that can be declared on the fly.

How do I profile draw calls which are in a specific performance marker?

The scrubber at the top of the Range Profiler View shows all of the performance marker ranges defined by the application, along with the amount of time spent for each one. A good strategy would be to look for ranges with a large amount of time, then drill down to where you see a large amount of time being spent. Once you click on that range, you can look at the Pipeline section for details on how that selected range is utilizing the GPU.

Why does my application run at a different frame rate under NVIDIA Nsight Graphics?

The NVIDIA Nsight Graphics Frame Debugger disables VSYNC, so applications that have VSYNC enabled under normal circumstances may see a higher frame rate when the same application is run under the Frame Debugger. NVIDIA Nsight Graphics also has a small performance overhead, which may reduce the frame rate slightly.

To understand PC Sampling data, you must first understand how the GPU executes shader instructions. Start by consulting the GPU Programming Guide:

The following explains how the PC sampler works in conjunction with the hardware warp schedulers, to produce the final output.

Each SM has four schedulers, one per subpartition. On Turing, each scheduler governs 8 warp slots, with up to 8 launched & resident warps, and can issue an instruction from one selected warp per cycle. A warp can only issue an instruction if it is eligible : the instruction must be fetched, all of its operands available (with no data dependencies), and the destination pipeline must be available. When a warp is not eligible, it is said to be stalled; the stall reason describes why the warp is not eligible.

Each warp has a program counter register (PC) that points at the next instruction fetch location for the active threads in that warp. Inactive threads also have PC registers to support Independent Thread Scheduling, but inactive threads are irrelevant; the PC Sampler only inspects active threads’ PCs.

Nsight configures the PC Sampler hardware to sample a { PC, Stall Reason ID} at a regular interval across the entire SM (all four subpartitions); that data is streamed to memory. The tool then converts raw data into a count (Samples) per PC, and a count-per-reason per PC.

A compute-centric treatment of this topic can be found here.

The average warp latency is the number of cycles that an average warp was resident on the GPU. The Samples% indicates the % of the average warp latency occupied by a given shader, function, or PC. Sorting by Samples reveals the regions of code with the highest contribution to latency. After identify top latency contributors, determine next steps by inspecting stall reasons.

Stalls are reported at the PC where a warp was unable to make progress. In many cases, this is due to an execution or data dependency on a prior instruction. For example, the following code may report a large number of samples on the line of code that consumes texResult, but the real culprit is the data producer g_MeshTexture.Sample().

float4 texResult = g_MeshTexture.Sample(MeshTextureSampler, In.TextureUV);
Output.RGBColor = texResult * In.Diffuse;

Note that samples can appear in the shadow of a taken branch - that is, on the instruction following a branch, even if that instruction is not executed - because the branch is still resolving at the time of the sampling.

if (constantBuffer.ConditionWeExpectToBeFalse)
{
     texResult = ...; // samples in the shadow of a branch
     output = dot(color, textResult);
}
else
{
     output = dot(color, constant);    // expect all samples to fall here
}

Stall reasons explain why a warp was unable to issue an instruction. Each stall reason is provoked by a distinct set of conditions or instructions; by eliminating those conditions or transforming code from one set of instructions to another, you can reduce stalls.

• Barrier: Compute warps are waiting for sibling warps at a GroupSync.

  • If the thread group size is 512 threads or greater, consider splitting it into smaller groups. This can increase eligible warps without affecting occupancy, unless shared memory becomes a new occupancy limiter.
  • Review whether all GroupSyncs are really necessary.

• Dispatch Stall: A pipeline interlock prevented instruction dispatch for a selected warp.

  • If dispatch stalls are higher than 5%, please file a bug to NVIDIA with reproducible.

• Drain : Exited warp is waiting to drain memory writes and pixel export.

• LG Throttle : Input FIFO to the LSU pipe for local and global memory instructions is full.

  • Avoid using thread-local memory.

    • Are dynamically indexed arrays declared in local scope?

    • Does the shader have excess register pressure causing spills?

  • Eliminate redundant global memory accesses (UAV accesses).

  • Data organization: pack UAV or SRV data to allow 64-bit or 128-bit accesses in place of multiple 32-bit accesses.

• Long Scoreboard : Waiting on data dependency for local, global, texture, or surface load.

  • Find the instruction or line of code that produces the data being waited upon; that instruction is the culprit.

  • Consider transforming a lookup table into a calculation.

  • Consider transforming global reads in which all threads read the same address into constant buffer reads.

  • If L1 hit rate is low, try to improve spatial locality (coalesced accesses).

  • If VRAM Throughput is high, try to improve spatial locality (coalesced accesses).

• Math Pipe Throttle : A math pipe input FIFO is full (FMA, ALU, FP16+Tensor).

  • This stall reason implies being computationally bound. Use the Range Profiler to best determine how to move computation to a different execution unit.

• Membar : Waiting for a memory barrier to return.

  • Memory barriers are issued by GroupMemoryBarrier, DeviceMemoryBarrier, AllMemoryBarrier, and their GroupSync variants.

  • Review whether the specified scope of each barrier in the shader is really needed. Group-level barriers resolve much faster than Device-level.

  • Review whether a memory barrier is needed at all. A compute shader where each thread writes to a unique UAV location does not require a memory barrier.

• MIO Throttle : The input FIFO to MIO is full.

  • May be triggered by local, global, shared, attribute, IPA, indexed constant loads (LDC), and decoupled math.

• Misc : A stall reason not covered elsewhere.

• Not Selected : Warp was eligible but not selected, because another warp was.

  • High “not selected” could indicate an opportunity to increase register or shared memory usage (lowering occupancy) without impacting performance. Opening the doors to greater shader complexity or improved quality.

• Selected : Warp issued an instruction. Technically not a stall.

• Short Scoreboard : Waiting for short latency MIO or RTCORE data dependency.

  • Includes 3D attribute load/store, pixel attribute interpolation, compute shared memory load/store, indexed constant loads, transcendentals (rcp, rsqrt, …) through the SFU pipe (aka XU pipe), VOTE, and a few other infrequent instructions.

• TEX Throttle : The TEXIN input FIFO is full.

  • Try issuing fewer texture fetches, surface loads, surface stores, or decoupled math operations.

  • Check whether the shader is using decoupled math (usually to be avoided).

  • Consider converting texture lookups or surface loads into global memory lookups (UAVs). Texture can accept 4 threads’ requests per cycle, whereas global accepts 32 threads.

• Wait : Waiting for coupled math data dependency (FMA, ALU, FP16+Tensor).

See also in the GPU Programming Guide:

Hot spots identify the top locations that have the most hit samples. This listing presents an actionable way of identifying and jumping to high-impact areas of the given report.

The shader profiler has the ability to correlate the samples that are gathered to source-level lines. This allows you, as the user, to determine, on a line-by-line basis, how your code is running. There are two types of correlation that are supported -- high-level shader language correlation and GPU shader assembly (SASS) correlation. High-level shader language correlation prepares a listing of your shaders source code, and along-side it, a chart of the samples that landed on each particular line. High-level correlation is very effective at grounding you to the code you are most familiar with, which is the shader source itself. For users who have access to the Pro builds of Nsight Graphics, and who wish to dive into the lower-level shader assembly, a SASS view is provided for individual instruction association of samples.