Nsight Systems provides a simple interface to profile on localhost or manage multiple connections to Linux or Windows based devices via SSH. The network connections manager can be launched through the device selection dropdown:

On x86_64:

On Tegra:

The dialog has simple controls that allow adding, removing, and modifying connections:

Security notice: SSH is only used to establish the initial connection to a target device, perform checks, and upload necessary files. The actual profiling commands and data are transferred through a raw, unencrypted socket. Nsight Systems should not be used in a network setup where MITM (man-in-the-middle) attack is possible, or where untrusted parties may have network access to the target device.

While connecting to the target device, you will be prompted to input the user's password. Please note that if you choose to remember the password, it will be stored in plain text in the configuration file on the host. Stored passwords are bound to the public key fingerprint of the remote device.

The No authentication option is useful for devices configured for passwordless login using root username. To enable such a configuration, edit the file /etc/ssh/sshd_config on the target and specify the following option:

PermitRootLogin yes

Then set empty password using passwd and restart the SSH service with service ssh restart.

Open ports: The Nsight Systems daemon requires port 22 and port 45555 to be open for listening. You can confirm that these ports are open with the following command:

sudo firewall-cmd --list-ports --permanent 
sudo firewall-cmd --reload

To open a port use the following command, skip --permanent option to open only for this session:

sudo firewall-cmd --permanent --add-port 45555/tcp 
sudo firewall-cmd --reload

Likewise, if you are running on a cloud system, you must open port 22 and port 45555 for ingress.

Kernel Version Number - To check for the version number of the kernel support of Nsight Systems on a target device, run the following command on the remote device:

cat /proc/quadd/version

Minimal supported version is 1.82.

Additionally, presence of Netcat command (nc) is required on the target device. For example, on Ubuntu this package can be installed using the following command:

sudo apt-get install netcat-openbsd

Target sampling behavior is somewhat different for Nsight Systems Workstation Edition and Nsight Systems Embedded Platforms Edition.

Nsight Systems Workstation Edition can use hotkeys to control profiling. Press the hotkey to start and/or stop a trace session from within the target application’s graphic window. This is useful when tracing games and graphic applications that use fullscreen display. In these scenarios switching to Nsight Systems' UI would unnecessarily introduce the window manager's footprint into the trace. To enable the use of Hotkey check the Hotkey checkbox in the project settings page:

The default hotkey is F12.

Nsight Systems Embedded Platforms Edition can work with Linux-based devices in three modes:

1. Attaching to a process by name

2. Attaching to a process by name, or launching a new process

3. Attaching to a process by its PID

The purpose of the configuration here is to define which process the profiler will attach to for sampling and tracing. Additionally, the profiler can launch a process prior to attaching to it, ensuring that all environment variables are set correctly to successfully collect trace information.

In Attach only mode, the process is selected by its name and command line arguments, as visible using the ps tool.

In Attach or launch mode, the process is to first search as if in the Attach only mode, but if it is not found, the process is launched using the same path and command line arguments. If NVTX, CUDA, or other trace settings are selected, the process will be automatically launched with appropriate environment variables.

Note that in some cases, the capabilities of Nsight Systems are not sufficient to correctly launch the application; for example, if certain environment variables have to be corrected. In this case, the application has to be started manually and Nsight Systems should be used in Attach only mode.

The Edit arguments... link will open an editor window, where every command line argument is edited on a separate line. This is convenient when arguments contain spaces or quotes.

To properly populate the Search criteria field based on a currently running process on the target system, use the Select a process button on the right, which has ellipsis as the caption. The list of processes is automatically refreshed upon opening.

Attach by PID mode should be used to connect to a specific process.

To choose one of the currently running processes on the target system, use the Select a process button on the right.

To perform remote profiling to a target Windows based machines, install and configure an OpenSSH Server on the target machine.

Nsight Systems Workstation Edition can use hotkeys to control profiling. Press the hotkey to start and/or stop a trace session from within the target application’s graphic window. This is useful when tracing games and graphic applications that use fullscreen display. In these scenarios switching to Nsight Systems' UI would unnecessarily introduce the window manager's footprint into the trace. To enable the use of Hotkey check the Hotkey checkbox in the project settings page:

The default hotkey is F12.

Changing the Default Hotkey Binding - A different hotkey binding can be configured by setting the HotKeyIntValue configuration field in the config.ini file.

Set the decimal numeric identifier of the hotkey you would like to use for triggering start/stop from the target app graphics window. The default value is 123 which corresponds to 0x7B, or the F12 key.

Virtual key identifiers are detailed in MSDN's Virtual-Key Codes.

Note that you must convert the hexadecimal values detailed in this page to their decimal counterpart before using them in the file. For example, to use the F1 key as a start/stop trace hotkey, use the following settings in the config.ini file:

HotKeyIntValue=112

Nsight Systems can sample one process tree. Sampling here means interrupting each processor periodically. The sampling rate is defined in the project settings and is either 100Hz, 1KHz (default value), 2Khz, 4KHz, or 8KHz.

On Windows, Nsight Systems can collect thread activity of one process tree. Collecting thread activity means that each thread context switch event is logged and (optionally) a backtrace is collected at the point that the thread is scheduled back for execution. Thread states are displayed on the timeline.

If it was collected, the thread backtrace is displayed when hovering over a region where the thread execution is blocked.

Symbol resolution happens on host, and therefore does not affect performance of profiling on the target.

Press the Symbol locations... button to open the Configure debug symbols location dialog.

Use this dialog to specify:

• Paths of PDB files

• Symbols servers

• The location of the local symbol cache

To use a symbol server:

1. Install Debugging Tools for Windows, a part of the Windows 10 SDK.

2. Add the symbol server URL using the Add Server button.

   Information about Microsoft's public symbol server, which enables getting Windows operating system related debug symbols can be found here.

To work with Nsight Systems, the target Android device should be configured for USB debugging in the Developer options settings menu. Please refer to Android development documentation to learn how to configure the device for USB debugging.

On the host, a compatible USB driver should be installed. Please refer to device manufacturer's documentation to learn how to obtain and install the driver.

Connect your target device via a USB cable and power it on (or wake it up). Make sure that you have the adb command available (it is part of Android SDK Platform Tools package). Nsight Systems can only connect to devices that are marked as device in the output of the adb devices command. Make sure you can enter the ADB shell of the target device by running adb shell on the host.

Launch the Nsight Systems application. On the first launch, a new project called Project 1 is created automatically.

When connecting to the target device, Nsight Systems will validate it and install its daemon into the following location on the device:

/data/local/tmp/com.nvidia.nsightsystems.tools/

Once the daemon and all required files are installed correctly, a green check mark will appear and Device is ready text will be displayed:

This section allows you to choose which application to profile. All information will be collected about the main process of the selected application, except when the Trace all processes checkbox is enabled.

For non-rooted Android devices, the list of applications only shows information about debuggable applications. By default, applications that are being developed using the Android SDK already contain the debuggable option in their manifests.

On rooted Android devices, profiling of all applications is allowed.

For convenience, the application list also shows the process identifiers (PID) of processes correlated to the listed packages. To refresh this information, use the button in the upper right corner of the list.

The two checkboxes below the application list are important to ensure that the correct launch or attach behavior is configured.

Allow sending intent to launch the default activity, when unselected, forces the profiler to attach to a running process. If no processes are found to correlate to the specified application name, the profiling session fails to start with an error message. When selected, Nsight Systems may launch the default intent of the selected application to make sure it is running and appears on top of the screen on the target device.

In some applications, especially in early stages of development, common bugs related to handling the lifecycle of activities can be found. In such cases, sending the default intent may lead to undesired behavior or even crashes of the profilee. Leaving the checkbox unselected ensures that the profiler does not affect the application.

Restart application if running is a convenient option in two cases:

1. When profiling from the very beginning of the application is desired.

2. When using some of the trace features described below. They require that a special library is injected into the application in runtime, which happens when the application is paused by the Android runtime's virtual machine just after starting. In this case, enabling this option helps ensure that the application is always restarted and the injection always happens, as opposed to potentially attaching to the application's process without injection.

   Collect NVTX trace. See NVTX Trace for more information.

   Collect OpenGL trace. See OpenGL Trace for more information.

The Nsight Systems CLI provides a simple interface to collect on a target without using the GUI. The collected data can then be copied to any system and analyzed later.

The CLI is distributed in the Target directory of the standard Nsight Systems download package. Users who want to install the CLI as a standalone tool can do so by copying the files within the Target directory. If you want the CLI output file (.qdstrm) to be auto-converted (to .qdrep) after the analysis is complete, you will need to copy the host directory as well.

If you wish to run the CLI without root (recommended mode), you will want to install in a directory where you have full access.

The Nsight Systems command lines can have one of two forms:

nsys [global_option]

or

nsys [command_switch][optional command_switch_options][application] [optional application_options]

All command line options are case sensitive. For command switch options, when short options are used, the parameters should follow the switch after a space; e.g. -s cpu. When long options are used, the switch should be followed by an equal sign and then the parameter(s); e.g. --sample=cpu.

For this version of Nsight Systems, you must launch a process from the command line to begin analysis. If an instance of the requested process is already running when the CLI command is issued, the collection will fail. The launched process will be terminated when collection is complete unless the user specifies the --kill none option (details below).

The Nsight Systems CLI supports concurrent analysis by using sessions. Each Nsight Systems session is defined by a sequence of CLI commands that define one or more collections (e.g. when and what data is collected). A session begins with either a start, launch, or profile command. A session ends with a shutdown command, when a profile command terminates, or, if requested, when all the process tree(s) launched in the session exit. Multiple sessions can run concurrently on the same system.

A couple of notes about the use of paths in your command line.

The Nsight Systems command line interface can be used in two modes. You may launch your application and begin analysis with options specified to the nsys profile command. Alternatively, you can control the launch of an application and data collection using interactive CLI commands.

Version Information

nsys -v

Effect: Prints tool version information to the screen.

Default analysis run

nsys profile <application> [application-arguments]

Effect: Launch the application using the given arguments. Start collecting immediately and end collection when the application stops. Trace CUDA, OpenGL, NVTX, and OS runtime libraries APIs. Collect CPU sampling information. Profile any child processes. Generate the report#.qdstrm file in the default location, incrementing the report number if needed to avoid overwriting any existing output files.

Limited trace only run

nsys profile --trace=cuda,nvtx -d 20 --sample=none 
-o my_test <application> [application-arguments]

Effect: Launch the application using the given arguments. Start collecting immediately and end collection after 20 seconds or when the application ends. Trace CUDA and NVTX APIs only. Do not collect CPU sampling information. Profile any child processes. Generate the output file as my_test.qdstrm in the current working directory.

Delayed start run

nsys profile -e TEST_ONLY=0 -y 20 <application> [application-arguments]

Effect: Set environment variable TEST_ONLY=0. Launch the application using the given arguments. Start collecting after 20 seconds and end collection at application exit. Trace CUDA, OpenGL, NVTX, and OS runtime libraries APIs. Collect CPU sampling information. Profile any child processes. Generate the report#.qdstrm file in the default location, incrementing if needed to avoid overwriting any existing output files.

Collect ftrace events

nsys profile --ftrace=drm/drm_vblank_event -d 20

Effect: Collect ftrace drm_vblank_event events for 20 seconds. Generate the report#.qdstrm file in the current working directory. Note that ftrace event collection requires running as root. To get a list of ftrace events available from the kernel, run the following:

sudo cat /sys/kernel/debug/tracing/available_events

Typical case: profile a Python script that uses CUDA

nsys profile --trace=cuda,cudnn,cublas,osrt,nvtx --delay=60 python my_dnn_script.py

Effect: Launch a Python script and start profiling it 60 seconds after the launch, tracing CUDA, cuDNN, cuBLAS, OS runtime APIs, and NVTX.

Typical case: profile an app that uses Vulkan

nsys profile --trace=vulkan,osrt,nvtx --delay=60 ./myapp

Effect: Launch an app and start profiling it 60 seconds after the launch, tracing Vulkan, OS runtime APIs, and NVTX.

Collect from beginning of application, end manually

nsys start --stop-on-exit=false
nsys launch --trace=cuda,nvtx --sample=none <application> [application-arguments]
nsys stop

Effect: Create interactive CLI process and set it up to begin collecting as soon as an application is launched. Launch the application, set up to allow tracing of CUDA and NVTX only. Stop only when explicitly requested. Generate the report#.qdstrm in the default location.

Note:

If you start a collection and fail to stop the collection (or if you are allowing it to stop on exit, and the application runs for too long) your system’s storage space may be filled with collected data causing significant issues for the system. Nsight Systems will collect a different amount of data/sec depending on options, but in general Nsight Systems does not support runs of more than 5 minutes duration.

Run application, begin collection manually, run until process ends

nsys launch -w true <application> [application-arguments]
nsys start

Effect: Create interactive CLI and launch an application set up for default analysis. Send application output to the terminal. No data is collected until you manually start collection at area of interest. Profile until the application ends. Generate the report#.qdstrm in the default location.

Note:

If you launch an application and that application and any descendants exit before start is called Nsight Systems will create a fully formed .qdstrm file containing no data.

Run application, start/stop collection using cudaProfilerStart/Stop

nsys start -c cudaProfileApi
nsys launch -w true <application> [application-arguments]

Effect: Create interactive CLI process and set it up to begin collecting as soon as a cudaProfileStart() is detected. Launch application for default analysis, sending application output to the terminal. Stop collection at next call to cudaProfilerStop, when the user calls nsys stop, or when the root process terminates. Generate the report#.qdstrm in the default location.

Note:

If you call nsys launch before nsys start -c cudaProfilerApi and the code contains a large number of short duration cudaProfilerStart/Stop pairs, Nsight Systems may be unable to process them correctly, causing a fault. This will be corrected in a future version.

Note:

The Nsight Systems CLI does not support multiple calls to the cudaProfilerStart/Stop API at this time.

Run application, start/stop collection using NVTX

nsys start -c nvtx
nsys launch -w true -p MESSAGE@DOMAIN <application> [application-arguments]

Effect: Create interactive CLI process and set it up to begin collecting as soon as an NVTX range with given message in given domain (capture range) is opened. Launch application for default analysis, sending application output to the terminal. Stop collection when all capture ranges are closed, when the user calls nsys stop, or when the root process terminates. Generate the report#.qdstrm in the default location.

Note:

The Nsight Systems CLI only triggers the profiling session for the first capture range.

NVTX capture range can be specified:

• Message@Domain: All ranges with given message in given domain are capture ranges. For example:

  nsys launch -w true -p profiler@service ./app

  This would make the profiling start when the first range with message "profiler" is opened in domain "service".

• Message@*: All ranges with given message in all domains are capture ranges. For example:

  nsys launch -w true -p profiler@* ./app

  This would make the profiling start when the first range with message "profiler" is opened in any domain.

• Message: All ranges with given message in default domain are capture ranges. For example:

  nsys launch -w true -p profiler ./app

  This would make the profiling start when the first range with message "profiler" is opened in the default domain.

• By default only messages, provided by NVTX registered strings are considered to avoid additional overhead. To enable non-registered strings check please launch your application with NSYS_NVTX_PROFILER_REGISTER_ONLY=0 environment:

  nsys launch -w true -p profiler@service -e NSYS_NVTX_PROFILER_REGISTER_ONLY=0 ./app

Run application, start/stop collection multiple times

The interactive CLI supports multiple sequential collections per launch.

nsys launch <application> [application-arguments]
nsys start
nsys stop
nsys start
nsys stop
nsys shutdown --kill sigkill

Effect: Create interactive CLI and launch an application set up for default analysis. Send application output to the terminal. No data is collected until the start command is executed. Collect data from start until stop requested, generate report#.qdstrm in the current working directory. Collect data from second start until the secont stop request, generate report#.qdstrm (incremented by one) in the current working directory. Shutdown the interactive CLI and send sigkill to the target application's process group.

Note:

Calling nsys cancel after nsys start will cancel the collection without generating a report.

You use the --stats option with the nsys profile or nsys start command to generate a set of useful summary statistics.

If your run traces CUDA, these include CUDA API, Kernel, and Memory Operation statistics:

If your run traces OS runtime events or NVTX push-pop ranges:

Recipes for these statistics as well as documentation on how to create your own metrics will be available in a future version of the tool.

The CLI generates a .qdstrm file. The .qdstrm file is an intermediate result file, not intended for multiple imports. It needs to be processed, either by importing it into the GUI or by using the standalone QdstrmImporter to generate an optimized .qdrep file. Use this .qdrep file when re-opening the result on the same machine, opening the result on a different machine, or sharing results with teammates.

This version of Nsight Systems will attempt to automatically convert the .qdstrm file to a .qdrep file with the same name after the run finishes if the required libraries are available. The ability to turn off auto-conversion will be added in a later version.

Import Into the GUI

The CLI and host GUI versions must match to import a .qdstrm file successfully. The host GUI is backward compatible only with .qdrep files.

Copy the .qdstrm file you are interested in viewing to a system where the Nsight Systems host GUI is installed. Launch the Nsight Systems GUI. Select File->Import... and choose the .qdstrm file you wish to open.

The import of really large, multi-gigabyte, .qdstrm files may take up all of the memory on the host computer and lock up the system. This will be fixed in a later version.

Create .qdrep Using QdstrmImporter

The CLI and QdstrmImporter versions must match to convert a .qdstrm file into a .qdrep file. This .qdrep file can then be opened in the same version or more recent versions of the GUI.

To run QdstrmImporter on the host system, find the QdstrmImporter binary in the Host-x86_64 directory in your installation. QdstrmImporter is available for all host platforms. See options below.

To run QdstrmImporter on the target system, copy the Linux Host-x86_64 directory to the target Linux system or install Nsight Systems for Linux host directly on the target. The Windows or MacOS host QdstrmImporter will not work on a Linux Target. See options below.

The nvprof command of the Nsight Systems CLI is intended to help former nvprof users transition to nsys. Many nvprof switches are not supported by nsys, often because they are now part of NVIDIA Nsight Compute.

The full nvprof documentation can be found at https://docs.nvidia.com/cuda/profiler-users-guide.

The nvprof transition guide for Nsight Compute can be found at https://docs.nvidia.com/nsight-compute/NsightComputeCli/index.html#nvprof-guide.

Any nvprof switch not listed below is not supported by the nsys nvprof command. No additional nsys functionality is available through this command. New features will not be added to this command in the future.

After choosing the nvprof command switch, the following options are available. When you are ready to move to using Nsight Systems CLI directly, see Command Line Options documentation for the nsys switch(es) given below. Note that the nsys implementation and output may vary from nvprof.

Usage.

nsys nvprof [options]

NVIDIA Visual Profiler (NVVP) and NVIDIA nvprof are deprecated. New GPUs and features will not be supported by those tools. We encourage you to make the move to Nsight Systems now. For additional information, suggestions, and rationale, see the blog series in Other Resources.