This section provides answers to frequently asked questions about NVIDIA Isaac SDK™. Use it as the first step in troubleshooting problems.
To ask and answer questions in the Isaac SDK developer community, visit the Isaac SDK Forum.
When I run my application that uses a ZED Camera, I get the following error:
engine/alice/backend/modules.cpp@74: dlopen failed: libhidapi-libusb.so.0: cannot open shared
object file: No such file or directory
The libusb-dev
library is not installed. Install libusb-dev
and all other required
dependencies on the robot with the following command:
bob@desktop:~/isaac/engine/$ ./engine/build/scripts/install_dependencies_jetson.sh -h <jetson_ip> -u <jetson_username>
Where <jetson_ip>
is the IP address of the robot.
I am running my app on Kaya. Everything runs fine, but the joystick does not move the robot.
Make sure you press the “deadman switch” L1 on the joystick while using the direction knobs. Pressing L1 is required to prevent unwanted movements.
When I run my application that uses a BMI160 IMU, I get the following error:
I2C Error: Device or resource busy (errno 16)
The kernel module nvs_bmi160
is using the I2C resource needed by the application. Unload
the module with the command sudo rmmod nvs_bmi160
. You can also run the
install_dependencies_jetson.sh
script, which prevents nvs_bmi160
from loading.
bob@desktop:~/isaac/engine/$ ./engine/build/scripts/install_dependencies_jetson.sh -h <jetson_ip> -u <jetson_username>
I don’t see images in Sight. Or the images are blank. Or the labels displayed are red/yellow.
There are a number of reasons why a channel may not update correctly:
The channel is not enabled in the bottom left menu. If the channel is not enabled, the robot does not send the update to the frontend and the channel appears in red in the renderer’s legend. Enable this channel from the channel menu directly or right click on the renderer and click enable all channels.
The channel is enabled, but there is no valid transformation between the channel and the reference frame used by the renderer. If this is the case, the channel appears in yellow in the renderer’s legend. If you hover over the channel name, additional information is provided about the missing transformation: ‘[reference_T_channel] is not defined’. You can check the PoseTree widget to figure out if there is a path between both coordinate frames.
Note‘’#####’’ is used as a default frame. It means that no frame has been provided for the channel.)
Your image/channel is rendered, but a later channel is overriding it. Channels are rendered in the order they appear in the legend: the top one first, followed by second one, and so on. If you render an image the size of the renderer, it covers all the previous channels. Make sure your channels are in the right order. You can change the order by right clicking the renderer and clicking settings. From there you can use the arrows on the left side of the channels to update the order.
The timestamp of the channel is in the future. Each show operation in Sight has its own timestamp (if none is provided, the current time is used). In general, it is good practice to use the acquisition time to render images or objects; it helps to synchronize the channels. Sight uses the application time (seconds since the beginning) to render the different channels. If the timestamp of a channel is in the future (either because the acquisition time used is not relative to the start of the application or because the wrong unit is used), then the channel is not updated until the application time catches up with the channel.
This case can be detected by refreshing the web page. If the image/channel updates but then freezes, it means that Sight received the latest message but is waiting for the time to catch up before rendering.
If you own the codelet rendering the image, make sure that you are using the right time in your show call. If you do not own the codelet, then the acquisition time of one of the channels is most likely used to render. Make sure the publisher provides the right acquisition time.
I am having lag issues in Sight.
See below for different types of lag issue:
- All the Sight widgets are lagging. You are most likely streaming too much data to Sight and your network connection can’t handle the load. First, try enabling only the channels you are interested in visualizing. To achieve this quickly, first unselect all the channels by clicking on the root of the channel list (left panel). You can then re-enable all the channels of a given renderer by right clicking on it and selecting Enable all channels.
- I enabled only the channels I need, but Sight is still lagging. You are probably looking at a channel that requires a lot information. It could come from either a channel with a lot of drawing primitives (PointCloudViewer for example) or large images. Most of the Isaac default viewers have an option to downscale the amount of data sent to Sight (CameraViewer has a reduce_scale parameter and PointCloudViewer has a`skip` parameter). One way to figure out how much bandwidth a given channel is using is to look at the Channel Statistics. To open it, click on Channels in the left panel. Once the widget is open, you can have a look at the bandwidth/frequency and other information about how much data are streamed to sight. Disable very demanding channels when they are not needed. If you need to visualize a channel with a high bandwidth, you may need to skip some messages. One way to achieve this result is to reduce the total allowed bandwidth from the websight config (type WebsightServer in the search box of the config panel on the right, then modify the bandwidth parameter).
- Sight is rendering with a decent framerate, but some channels seem to be lagging behind. By default, each renderer renders with a delay of 0.2s to allow all the channels to arrive. However, 0.2s might not be enough for channels where the duration from sensor acquisition to Sight is more than 0.2s. You can adjust the rendering delay of each renderer individually by right-clicking the renderer and hovering over the Change delay option: a slider will appear, allowing you to adjust the delay from 0 to 2s.
- The bounding box for ML detections do not match the image. If you have tried the solutions above and they did not help, check that each detection matches an image and each image has its own detection. If you reduced the frame rate of the CameraViewer, it is possible a detection is being rendered without its matching image. On the other hand, if the detection pipeline can’t keep up with the camera framerate, it is possible some frames are not being processed. Try reducing the frame rate of the camera to make sure no frames are skipped.
Unable to see the Sight webpage after opening http://localhost:3000 in a browser.
Make sure an application is running on your desktop. If you are running the application on the robot, you must use the robot IP address.
If the “Failed to start Webserver!” error message appears in the logs, the port may not have been released by a previous application.
Use the following command to determine the application using the port:
lsof -i TCP:3000
You can then kill the application with the following command:
killall -9 <app name>
If you need to keep the app running, update the configuration file to change the port to another open port.
Sight visualization is very slow.
When network bandwidth is insufficient, channels displayed in Sight may exhibit latency or very low framerate. Follow these steps to improve bandwidth:
In Sight, under Channels, uncheck all unnecessary channels.
TipYou can disable all channels at once by clicking on the application name in the channel menu. Enable all channels you want to visualize by right clicking a renderer and selecting enable all channels.
Verify Wi-Fi antennas and cables are connected to the PCIe Wi-Fi card.
Determine whether power-saving mode is enabled on the Wi-Fi adapter:
cat /etc/NetworkManager/conf.d/default-wifi-powersave-on.conf [connection] wifi.powersave = 3
Set the
wifi.powersave
value to 2 to disable power-saving mode:wifi.powersave = 2
Try a wired connection to rule out Wi-Fi bandwidth issues.
If Sight has been running for a long time, it might have accumulated a lot of data, which slows down the frontend. Try refreshing from time to time to see if it improves performance.
The global planner is taking forever to start or to run
If the global planner is too slow, you can try to reduce some parameters:
graph_initialization_steps
: The number of random samplings done in the start function.graph_in_tick_steps
: The number of random samplings done each tick. If the graph is already dense enough, this parameter might be set to 0.
If reducing the number of random samplings compromises the quality of the navigation graph produced, consider pre-computing the graph and loading it from a file. To produce this graph, follow these steps:
- Run the application with
graph_initialization_steps
set to a high value (you can also setgraph_in_tick_steps
to a high value to keep increasing the graph each tick). - Choose a destination to save the current graph and update the
graph_file_out
parameter. - Wait for the graph to be large enough. Then kill the app.
- You can now restart the app and set the
graph_file_in
parameter to the file containing the graph.
The global planner can’t find a path or produces a suboptimal path
If your map is large, you probably need to generate a large graph. You can increase the
graph_initialization_steps
parameter or load the graph from a file (see the steps described
in the question above).
What is the easiest way to generate a capnp id for a new message?
Every capnp file requires a unique ID at the beginning of the file. If you create a new capnp file and attempt to build Isaac without an ID in your new file, an ID is generated and printed in the error message, similar to the following:
messages/my_new.capnp:1:1: error: File does not declare an ID. I've generated one for you.
Add this line to your file: @0xcdeac1e381086f01;
As instructed by the error message above, add an “@” symbol, then the generated ID, then a semicolon to the top of your capnp file (if you don’t add the semicolon, you will recieve a parse error).
When running an application that requires TensorFlow, I get the following error: PANIC engine/core/buffers/algorithm_cuda.cpp@55: Could not copy memory. Error: 35
To resolve this error, install CUDA 10.0 using the instructions at NVIDIA CUDA Installation Guide for Linux.
When using a USB Camera on a TX2 I see no image or a heavily distorted image.
This happens when the CPU clock speed is too low on the TX2 and it cannot process the USB frames in
time. Try running the jetson_clocks
command to increase the CPU clock speed. See this NVIDIA Developer Forums topic for more details.
What is the syntax for setting a Pose3d in json?
Earlier versions of Isaac SDK expected an array of 7 numbers to set a Pose3d in JSON. The old syntax, which is still supported, looks as follows:
[qw, qx, qy, qz, x, y, z]
The first four numbers form the quaternion while the latter three are the translation values.
Some users may find the new format more convenient. For details on the syntax, please see
engine/gems/serialization/json_formatter.hpp
and for examples please check
engine/gems/serialization/tests/json_formatter.cpp
. Below are some equivalent poses:
old syntax (supported) Use square brackets | new syntax Use curly brackets |
---|---|
|
|
|
|
|
|
|
|
How can I convert my log files that contain deprecated types like ColorCameraProto?
//packages/cask/apps:cask_converter can be used to convert log files with deprecated types. Please see help from this application.
What should I check first if my build is failing?
Make sure you are running Ubuntu 18.04 on your host system, and that you have run the install_dependencies.sh script.
Why am I seeing “crosstool” errors when running Bazel build?
If you see the following errors during Bazel build, your Bazel version is out of date:
ERROR: @bazel_tools//tools/build_defs/cc:action_names.bzl' does not contain symbol 'ACTION_NAMES
ERROR: error loading package '@toolchain//crosstool': Extension file 'crosstool/cc_toolchain_config.bzl' has errors
Use the engine/build/scripts/install_dependencies.sh
script inside engine
directory
to upgrade Bazel to the current version required by Isaac SDK. To perform a manual update, see the
GitHub page for the Bazel installer.
When using a USB Camera on a TX2 I see no image or a heavily distorted image.
This happens when the CPU clock speed is too low on the TX2 and it cannot process the USB frames in
time. Try running the jetson_clocks
command to increase the CPU clock speed. See this NVIDIA Developer Forums topic
for more details.
I am seeing an error similar to “libnppc.so.10.0: cannot open shared object file: No such file or directory”
Some Isaac SDK code requires CUDA 10.0 to execute properly. Ensure CUDA 10.0 is installed
locally and the /usr/local/cuda-10.0/lib64
value is part of the LD_LIBRARY_PATH
environment variable.
I am seeing “Could not deserialize configuration parameter” errors when I run an application.
Configuration values must match the data type specified in the component API. See the Component API Overview or the component .hpp file for the expected data type.
Note that an integer value is accepted as a type of double value.
Your packages or components may require external dependencies not yet available in Isaac. The right strategy for integrating a new external dependency can vary case by case. There are several examples in the third_party folder. This tutorial explains how to add the zlib library.
Find a reliable source for your dependency. In the case of zlib, get it from the official zlib webpage at zlib.net. To avoid unexpected problems in your codebase when the external dependency is updated, fix a specific version of the library. This tutorial chooses the version 1.2.11.
Add the dependency to your WORKSPACE file using the Bazel rule
isaac_http_archive
:isaac_http_archive( name = "zlib", build_file = clean_dep("//zlib.BUILD"), sha256 = "c3e5e9...cb1a1", strip_prefix = "zlib-1.2.11", url = "https://zlib.net/zlib-1.2.11.tar.gz", )
Importing external dependencies as an archive is generally preferred over using the rules
new_git_repository
orgit_repository
, which import git repositories, because archives are smaller and have less overhead than git repositories.For the purposes of this tutorial, assume that you have to write a BUILD file for the new external dependency (this may not always be required). In the case of zlib, this is quite straightforward:
cc_library( name = "zlib", srcs = glob(["*.c", "*.h"], exclude="zlib.h"), hdrs = ["zlib.h"], copts = [ "-Wno-shift-negative-value", "-Wno-implicit-function-declaration", ], includes = ["."], )
This BUILD file defines a single C++ library named
zlib
using the Bazel rulecc_library
. The library includes all C source and header files from zlib and compiles them into a single library. The only external header file iszlib.h
, which is excluded from the sources so that the same file does not appear as a source and a header.In general,
srcs
is used for files internal to the library and can be used for both source and header files, whilehdrs
is used for publicly facing header files that are required in applications that use the library.A couple of compiler flags are also required. Add them using the
copts
option to avoid compiler warnings that are treated as errors due to use of the -Wall option by the Isaac compilation tool.
The new library is ready for use as a dependency in one of your libraries or binaries, similar to the following example:
cc_library(
name = "foo",
srcs = ["foo.cpp"]
hdrs = ["foo.hpp"],
deps = ["@zlib"]
)
Every external dependency creates its namespace with the same name as the one used for the
isaac_http_archive
or corresponding rule. In this case, the name of both the dependency and the
library is zlib and thus the shortcut @zlib
can be used to refer to the library.
If there were a second library foo
inside the same archive, the explicit form @zlib//:foo
would
be needed to reference it.
In case of a crash, Isaac SDK applications attempt to collect information with breakpad. This information is written into a minidump file in binary format. This section explains how to extract human-readable information from minidump files.
Prepare the Minidump Toolkit
Prepare the minidump toolkit in /tmp/minidump/
with the following command:
bob@desktop:~/isaac/engine/$ ./engine/build/scripts/prepare_minidump_tools.sh
Prepare Symbols
Prepare symbols before testing applications by passing the argument -s
when deploying applications
as shown in the following command:
bob@desktop:~/isaac/sdk/$ ./../engine/engine/build/deploy.sh -s -h <robot_ip> -d <device> -p <target>
--remote_user <username_on_robot>
where <robot_ip>
is the IP address of the robot and <username_on_robot>
is your username on the
robot.
Analyzing Minidumps
When a crash occurs, Isaac application minidump file paths are reported on the console similar to the following:
Minidump written to: /tmp/28db00a1-e756-4c47-62f6a7b6-fc26c1a0.dmp
To make the minidump data human-readable, run the following command on a desktop that has symbols prepared:
bob@desktop:~/isaac/engine/$ ./engine/build/scripts/process_jetson_minidump.sh -h <robot_ip> -d
/tmp/28db00a1-e756-4c47-62f6a7b6-fc26c1a0.dmp --remote_user <username_on_robot>
where <robot_ip>
is the IP address of the robot and <username_on_robot>
is your user name on the
robot. If the minidump file is locally available, run the following command:
bob@desktop:~/isaac/engine/$ ./engine/build/scripts/process_minidump.sh <minidump>
Crash information is presented on the console, and is similar to the following:
Crash reason: SIGSEGV /SEGV_MAPERR
Crash address: 0x0
Process uptime: not available
Thread 11 (crashed)
0 realsense_camera!isaac::RealsenseCamera::tick() [RealsenseCamera.cpp : 533 + 0x4]
x0 = 0x0000000000000000 x1 = 0x0000007f962b2180
x2 = 0x0000007fa0000080 x3 = 0x0000007f800008d0
x4 = 0x0000000000000007 x5 = 0x00000000007e8ac8
x6 = 0x00000000007e89f0 x7 = 0x0000000000063762
x8 = 0x00000000000000d7 x9 = 0x001dcd6500000000
x10 = 0x000000005c787d6b x11 = 0x000000000f171c90
x12 = 0x0000000000000017 x13 = 0x000000005c787d6b
x14 = 0x00076f61bec52c49 x15 = 0x00001c2c8948c9df
x16 = 0x0000007fa7708688 x17 = 0x0000007fa7471838
x18 = 0x0000007fa005bdcc x19 = 0x0000000037cc4610
x20 = 0x0000007fa004bbb0 x21 = 0x0000007fa0248c01
x22 = 0x0000000037cc47e0 x23 = 0x00000006556cdce8
x24 = 0x000000000091f000 x25 = 0x0000007fa53dd330
x26 = 0x0000000000000000 x27 = 0x0000007f962b31b0
x28 = 0x0000000000000001 fp = 0x0000007f962b20a0
lr = 0x0000000000459f6c sp = 0x0000007f962b20a0
pc = 0x0000000000459f74