Getting Started

This section describes the sample and tutorial applications included in Isaac SDK and how to run an application.

Tutorial and Sample Applications

There are over 30 tutorials and samples provided with Isaac SDK to get you started. The tutorials in //apps/tutorials demonstrate basic concepts in a clean environment. The sample applications in //apps/samples demonstrate various GEMs. Many samples have hardware requirements.

Most samples and tutorials come with a BUILD file and a single JSON file. The JSON file contains a compute graph which defines the functionality of the application. For example, a camera sample application has a camera node and a viewer node. The camera node acquires images and sends them to the viewer node for visualization. The JSON file may also contain a configuration section for specification of various configuration parameters. Sometimes you might have to adapt the configuration based on your hardware setup or platform.

How to run an application on the desktop is explained in the section Running an Application. Deploying an application to a Jetson device and running it on the Jetson is explained in the section Deploying and Running on Jetson.

The following sections describe a selection of tutorials and samples to get you started.

ping

A tutorial which introduces the concept of apps and codelets. It shows how to create a simple application with a codelet which ticks periodically. You can find a step-by-step tutorial for this application in the Developing Codelets in C++ section of this documentation.

Platforms: Desktop, Jetson TX2, Jetson Xavier, Jetson Nano

Hardware requirements: none

ping_pong

A development of the basic ping tutorial which shows how messages can be passed from one node to another node.

Platforms: Desktop, Jetson TX/2, Jetson Xavier, Jetson Nano

Hardware requirements: none

proportional_control_cpp

An intermediate tutorial which explains how to write a basic proportional controller for a differential robot base.

Platforms: Jetson TX2, Jetson Xavier, Jetson Nano

Hardware requirements: Segway

v4l2_camera

A basic sample application which connects to a camera using video for linux (V4L2). The camera image can be seen in Sight. (You must add a window in Sight to see the image. See Windows Menu (Top Left) for more information.) Make sure to set a valid resolution in the configuration section in the application file.

Platforms: Desktop, Jetson TX2, Jetson Xavier, Jetson Nano

Hardware: Requires a camera, for example a Realsense or a ZED camera. A normal webcam might also work.

april_tags

Demonstrates the GPU accelerated AprilTag fiducial detection GEM. A connected Realsense camera and a printed AprilTag are required. (The ApriTag can be displayed on your screen, if necessary.) In case you do not have a Realsense camera you can switch to a different camera similar that used for the v4l2_camera tutorial.

Platforms: Desktop, Jetson TX2, Jetson Xavier, Jetson Nano

Hardware: Requires a camera and an AprilTag fiducial.

stereo_vo

A basic sample application which connects to a ZED camera to provide stereo visual odometry.

Platforms: Desktop, Jetson TX2, Jetson Xavier, Jetson Nano

Hardware: Requires a camera.

realsense_camera

A basic sample application which connects to a Realsense camera. The camera image can be seen in Sight. Make sure to set a valid resolution in the configuration section in the application file.

Platforms: Desktop, Jetson TX2, Jetson Xavier, Jetson Nano

Hardware: Requires a Realsense camera.

zed_camera

A basic sample application which connects to a ZED camera. The camera image can be seen in Sight. Make sure to set a valid resolution in the configuration section in the application file.

Platforms: Desktop, Jetson TX2, Jetson Xavier, Jetson Nano

Hardware: Requires a ZED camera.

image_undistortion

A sample application that corrects distortion in images captured with a ZED camera. The effects of the correction can be seen in Sight. (You must add a window in Sight to see the image. See Windows Menu (Top Left) for more information.)

Platforms: Desktop, Jetson Xavier

Hardware: Requires a camera.

hgmm_matching

A sample application that demonstrates point cloud matching using a Realsense camera.

Platforms: Desktop, Jetson Xavier

Hardware: Requires a Realsense camera.

stereo_matching_depth

A sample application which demonstrates the coarse-to-fine stereo matching GEMs. It works with a stereo image and computes depth images.

Platforms: Desktop, Jetson TX2, Jetson Xavier

Hardware: Requires a ZED camera.

Running an Application

This section explains how to run the stereo_dummy sample application on your desktop. All other applications can be run in a similar matter by using the corresponding bazel target name of the application.

A bazel target name for example has the following form: //app/samples/stereo_dummy. This refers to the application stereo_dummy in the folder app/samples/stereo_dummy. If you want to run a different application you have to change the target name correspondingly.

Note that all bazel build and bazel run commands should be executed at the root folder of your repository. For example if your root folder is /home/bob/isaac you first go to the directory /home/bob/isaac and then run the commands mentioned below.

  1. Build the sample application with the following command on the host system:
bob@desktop:~/isaac$ bazel build //apps/samples/stereo_dummy
  1. Run the sample application with the following command:
bob@desktop:~/isaac$ bazel run //apps/samples/stereo_dummy

The bazel run command first builds and then runs the application. If you want to run an application bazel run is enough. Remember to run bazel run at the root folder of your repository as explained above.

3. Once you have an application running, open the visualization frontend by opening http://localhost:3000 in a browser.

In case you want to run an application with additional configuration parameters you can pass these configuration files on the command line as follows:

bob@desktop:~/isaac$ bazel run //apps/samples/stereo_dummy -- --config more_config.json

Note the double dash -- to separate arguments to bazel from arguments to the application.

The following bazel commands are useful:

  • Build and run an application in GDB for debugging (in this example, pose_tensorrt_inference) with the following commands:

    bob@desktop:~/isaac$ bazel build -c dbg //apps/samples/pose_tensorrt_inference
bob@desktop:~/isaac$ gdb --args bazel-bin/engine/alice/tools/main --app apps/samples/pose_tensorrt_inference/pose_tensorrt_inference.app.json

  • Build everything:

    bob@desktop:~/isaac$ bazel build ...

  • Build only one target:

    bob@desktop:~/isaac$ bazel build //engine/gems/filters/examples:ekf_sin_exp

  • Run all tests:

    bob@desktop:~/isaac$ bazel test ... --jobs=1

  • Run one test:

    bob@desktop:~/isaac$ bazel test //engine/gems/optimization/tests:pso_ackley

  • Run linter checks:

    bob@desktop:~/isaac$ bazel test --config=lint ...

    The linter requires Python 2.7. If all files fail the linter tests, the Python path is probably the issue. Try:

    bob@desktop:~/isaac$ bazel test --python_path=/usr/bin/python2.7 --config=lint ...

Deploying and Running on Jetson

This section briefly explains how to deploy an application from your desktop machine to the robot and how to run it.

The Isaac SDK fully supports cross-compilation for Jetson. Compiling the source code on Jetson itself is not recommended.

The following two steps need to be run only once:

  1. Make sure you have an SSH key on your desktop machine.

2. Copy your SSH identity to the robot using the user name and IP you use to login on the robot with a command similar to the following:

bob@desktop:~/isaac$ ssh-copy-id ROBOTUSER@ROBOTIP

You might need to connect the robot to a screen to get its IP address.

  1. To run the stereo_dummy sample application on the robot, first deploy the package to the robot with the following command:

    bob@desktop:~/isaac$  ./engine/build/deploy.sh --remote-user ROBOTUSER -p //apps/samples/stereo_dummy:stereo_dummy-pkg -d jetpack42 -h ROBOTIP

    The -d jetpack42 option specifies that we are building and deploying to a Jetson device with Jetpack version 4.2.

  2. Login to the robot to run the application:

bob@jetson:~/isaac$ ssh ROBOTUSER@ROBOTIP
  1. Go to the deployment folder and run the application:
bob@jetson:~/$ cd deploy/bob/stereo_dummy-pkg
bob@jetson:~/deploy/bob/stereo_dummy-pkg$ ./apps/samples/stereo_dummy/stereo_dummy

Here “bob” is the user name you use on your host system. You can deploy under a different folder by specifying -u OTHER_USER to deploy.sh in step 3.

  1. Once the application is running, connect to it in your browser and inspect the running application with websight. To do so navigate to http://ROBOTIP:3000 in your browser.

In case you want to run an application with additional configuration parameters you can pass these configuration files on the command line as follows:

bob@desktop:~/isaac$ ./apps/samples/stereo_dummy/stereo_dummy --config more_config.json

Note that configuration files are passed directly via the --config command line argument.

Python Application Support

The Isaac SDK provides basic Python support. However support for Python is in an experimental state in this release. The API is not stable or feature complete.

The tutorial in //apps/samples/ping_python is similar to the ping tutorial for C++ but implemented in Python. It is available on all platforms and does not require any hardware.

Running a Python application on a desktop system is identical to running a C++ application as explained in section Running an Application. Deploying a Python app to Jetson is identical to deploying a C++ application as explained in Deploying and Running on Jetson.

However running a Python application on Jetson is slightly different. Using the run script is necessary to set certain environment variables required for Python. In step 5 of Deploying and Running on Jetson, use the following commands to run an application:

bob@jetson:~/$ cd deploy/bob/ping_python-pkg
bob@jetson:~/deploy/bob/ping_python-pkg$ ./run ./apps/tutorials/ping_python/ping_python.py

Where “bob” is you username on your desktop system.