Getting Started¶
This section describes the sample and tutorial applications included in Isaac SDK and how to run an application.
Tutorial and Sample Applications¶
There are over 30 tutorials and samples provided with Isaac SDK to get you started.
The tutorials in //apps/tutorials
demonstrate basic concepts in a clean environment. The
sample applications in //apps/samples
demonstrate various GEMs. Many samples have hardware
requirements.
Most samples and tutorials come with a BUILD file and a single JSON file. The JSON file contains a compute graph which defines the functionality of the application. For example, a camera sample application has a camera node and a viewer node. The camera node acquires images and sends them to the viewer node for visualization. The JSON file may also contain a configuration section for specification of various configuration parameters. Sometimes you might have to adapt the configuration based on your hardware setup or platform.
How to run an application on the desktop is explained in the section Running an Application. Deploying an application to a Jetson device and running it on the Jetson is explained in the section Deploying and Running on Jetson.
The following sections describe a selection of tutorials and samples to get you started.
ping¶
A tutorial which introduces the concept of apps and codelets. It shows how to create a simple application with a codelet which ticks periodically. You can find a step-by-step tutorial for this application in the Developing Codelets in C++ section of this documentation.
Platforms: Desktop, Jetson TX2, Jetson Xavier, Jetson Nano
Hardware requirements: none
ping_pong¶
A development of the basic ping tutorial which shows how messages can be passed from one node to another node.
Platforms: Desktop, Jetson TX/2, Jetson Xavier, Jetson Nano
Hardware requirements: none
proportional_control_cpp¶
An intermediate tutorial which explains how to write a basic proportional controller for a differential robot base.
Platforms: Jetson TX2, Jetson Xavier, Jetson Nano
Hardware requirements: Segway
v4l2_camera¶
A basic sample application which connects to a camera using video for linux (V4L2). The camera image can be seen in Sight. (You must add a window in Sight to see the image. See Windows Menu (Top Left) for more information.) Make sure to set a valid resolution in the configuration section in the application file.
Platforms: Desktop, Jetson TX2, Jetson Xavier, Jetson Nano
Hardware: Requires a camera, for example a Realsense or a ZED camera. A normal webcam might also work.
april_tags¶
Demonstrates the GPU accelerated AprilTag fiducial detection GEM. A connected Realsense camera and a printed AprilTag are required. (The ApriTag can be displayed on your screen, if necessary.) In case you do not have a Realsense camera you can switch to a different camera similar that used for the v4l2_camera tutorial.
Platforms: Desktop, Jetson TX2, Jetson Xavier, Jetson Nano
Hardware: Requires a camera and an AprilTag fiducial.
stereo_vo¶
A basic sample application which connects to a ZED camera to provide stereo visual odometry.
Platforms: Desktop, Jetson TX2, Jetson Xavier, Jetson Nano
Hardware: Requires a camera.
realsense_camera¶
A basic sample application which connects to a Realsense camera. The camera image can be seen in Sight. Make sure to set a valid resolution in the configuration section in the application file.
Platforms: Desktop, Jetson TX2, Jetson Xavier, Jetson Nano
Hardware: Requires a Realsense camera.
zed_camera¶
A basic sample application which connects to a ZED camera. The camera image can be seen in Sight. Make sure to set a valid resolution in the configuration section in the application file.
Platforms: Desktop, Jetson TX2, Jetson Xavier, Jetson Nano
Hardware: Requires a ZED camera.
image_undistortion¶
A sample application that corrects distortion in images captured with a ZED camera. The effects of the correction can be seen in Sight. (You must add a window in Sight to see the image. See Windows Menu (Top Left) for more information.)
Platforms: Desktop, Jetson Xavier
Hardware: Requires a camera.
hgmm_matching¶
A sample application that demonstrates point cloud matching using a Realsense camera.
Platforms: Desktop, Jetson Xavier
Hardware: Requires a Realsense camera.
stereo_matching_depth¶
A sample application which demonstrates the coarse-to-fine stereo matching GEMs. It works with a stereo image and computes depth images.
Platforms: Desktop, Jetson TX2, Jetson Xavier
Hardware: Requires a ZED camera.
Running an Application¶
This section explains how to run the stereo_dummy sample application on your desktop. All other applications can be run in a similar matter by using the corresponding bazel target name of the application.
A bazel target name for example has the following form: //app/samples/stereo_dummy
. This refers to
the application stereo_dummy
in the folder app/samples/stereo_dummy
. If you want to run a
different application you have to change the target name correspondingly.
Note that all bazel build and bazel run commands should be executed at the root folder of your
repository. For example if your root folder is /home/bob/isaac
you first go to the directory
/home/bob/isaac
and then run the commands mentioned below.
- Build the sample application with the following command on the host system:
bob@desktop:~/isaac$ bazel build //apps/samples/stereo_dummy
- Run the sample application with the following command:
bob@desktop:~/isaac$ bazel run //apps/samples/stereo_dummyThe
bazel run
command first builds and then runs the application. If you want to run an applicationbazel run
is enough. Remember to runbazel run
at the root folder of your repository as explained above.
3. Once you have an application running, open the visualization frontend by opening
http://localhost:3000
in a browser.
In case you want to run an application with additional configuration parameters you can pass these configuration files on the command line as follows:
bob@desktop:~/isaac$ bazel run //apps/samples/stereo_dummy -- --config more_config.json
Note the double dash --
to separate arguments to bazel from arguments to the application.
The following bazel commands are useful:
Build and run an application in GDB for debugging (in this example, pose_tensorrt_inference) with the following commands:
bob@desktop:~/isaac$ bazel build -c dbg //apps/samples/pose_tensorrt_inference bob@desktop:~/isaac$ gdb --args bazel-bin/engine/alice/tools/main --app apps/samples/pose_tensorrt_inference/pose_tensorrt_inference.app.json
Build everything:
bob@desktop:~/isaac$ bazel build ...
Build only one target:
bob@desktop:~/isaac$ bazel build //engine/gems/filters/examples:ekf_sin_exp
Run all tests:
bob@desktop:~/isaac$ bazel test ... --jobs=1
Run one test:
bob@desktop:~/isaac$ bazel test //engine/gems/optimization/tests:pso_ackley
Run linter checks:
bob@desktop:~/isaac$ bazel test --config=lint ...
The linter requires Python 2.7. If all files fail the linter tests, the Python path is probably the issue. Try:
bob@desktop:~/isaac$ bazel test --python_path=/usr/bin/python2.7 --config=lint ...
Deploying and Running on Jetson¶
This section briefly explains how to deploy an application from your desktop machine to the robot and how to run it.
The Isaac SDK fully supports cross-compilation for Jetson. Compiling the source code on Jetson itself is not recommended.
The following two steps need to be run only once:
- Make sure you have an SSH key on your desktop machine.
2. Copy your SSH identity to the robot using the user name and IP you use to login on the robot with a command similar to the following:
bob@desktop:~/isaac$ ssh-copy-id ROBOTUSER@ROBOTIPYou might need to connect the robot to a screen to get its IP address.
To run the stereo_dummy sample application on the robot, first deploy the package to the robot with the following command:
bob@desktop:~/isaac$ ./engine/build/deploy.sh --remote-user ROBOTUSER -p //apps/samples/stereo_dummy:stereo_dummy-pkg -d jetpack42 -h ROBOTIP
The
-d jetpack42
option specifies that we are building and deploying to a Jetson device with Jetpack version 4.2.Login to the robot to run the application:
bob@jetson:~/isaac$ ssh ROBOTUSER@ROBOTIP
- Go to the deployment folder and run the application:
bob@jetson:~/$ cd deploy/bob/stereo_dummy-pkg bob@jetson:~/deploy/bob/stereo_dummy-pkg$ ./apps/samples/stereo_dummy/stereo_dummy
Here “bob” is the user name you use on your host system. You can deploy under a different folder by specifying
-u OTHER_USER
todeploy.sh
in step 3.
- Once the application is running, connect to it in your browser and inspect the running
application with websight. To do so navigate to
http://ROBOTIP:3000
in your browser.
In case you want to run an application with additional configuration parameters you can pass these configuration files on the command line as follows:
bob@desktop:~/isaac$ ./apps/samples/stereo_dummy/stereo_dummy --config more_config.json
Note that configuration files are passed directly via the --config
command line argument.
Python Application Support¶
The Isaac SDK provides basic Python support. However support for Python is in an experimental state in this release. The API is not stable or feature complete.
The tutorial in //apps/samples/ping_python
is similar to the ping tutorial for C++ but
implemented in Python. It is available on all platforms and does not require any hardware.
Running a Python application on a desktop system is identical to running a C++ application as explained in section Running an Application. Deploying a Python app to Jetson is identical to deploying a C++ application as explained in Deploying and Running on Jetson.
However running a Python application on Jetson is slightly different. Using the run
script
is necessary to set certain environment variables required for Python. In step 5 of Deploying and
Running on Jetson, use the following commands to run an application:
bob@jetson:~/$ cd deploy/bob/ping_python-pkg
bob@jetson:~/deploy/bob/ping_python-pkg$ ./run ./apps/tutorials/ping_python/ping_python.py
Where “bob” is you username on your desktop system.