Frequently Asked Questions¶
This section provides answers to frequently asked questions about your release. Use it as the first step in troubleshooting problems.
Issues with Hardware Components¶
When I run my application that uses a ZED Camera I get the following error:
engine/alice/backend/modules.cpp@74: dlopen failed: libhidapi-libusb.so.0: cannot open shared object file: No such file or directory
libusb-dev library is not installed. Install
libusb-dev and all other required
dependencies on the robot with the following command:
bob@desktop:~/isaac$ ./engine/build/scripts/install_dependencies_jetson.sh -h <Jetson_IP>
Where <Jetson_IP> is the IP address of the robot.
I am running my app on Kaya. Everything runs fine, but the joystick does not move the robot.
Make sure you press the “deadman switch” L1 on the joystick while using the direction knobs. Pressing L1 is required to prevent unwanted motions.
When I run my application that uses a BMI160 IMU I get the following error:
I2C Error: Device or resource busy (errno 16)
The kernel module
nvs_bmi160 is using the I2C resource needed by the application. Unload
the module with the command
sudo rmmod nvs_bmi160. You can also run the
install_dependencies_jetson.sh script, which prevents
nvs_bmi160 from loading.
bob@desktop:~/isaac$ ./engine/build/scripts/install_dependencies_jetson.sh -h <Jetson_IP>
Issues with the Visualization and WebSight¶
I don’t see images in Sight. Or the images are blank. Or the labels displayed are red/yellow.
Reasons why a channel does not update correctly:
The first thing to check is whether the channel is enabled in the bottom left menu. If the channel is not enabled, the robot does not send the update to the frontend and the channel appears in red in the renderer’s legend. Enable this channel either from the channel menu directly or right click on the renderer and click ‘enable all channels’.
The channel might be enabled, but there is no valid transformation between the channel and the reference frame used by the renderer. If this is the case, the channel appears in yellow in the renderer’s legend. If you hover over the channel name, additional information is provided about the missing transformation: ‘[reference_T_channel] is not defined’. You can check the PoseTree widget to figure out if there is a path between both coordinate frames. (Note: ‘#####’’ is used as a default frame. It means that no frame has been provided for the channel.)
Your image/channel is rendered, but a later channel is overriding it. Channels are rendered in the order they appear in the legend: the top one first, followed by second one, and so on. If you render an image the size of the renderer, it covers all the previous channels. Make sure your channels are sorted in the right order. You can change the order by right clicking on the renderer and clicking on ‘settings’. From there you can use the arrows on the left side of the channels to update the order.
Each show operation in Sight has its own timestamp (if none is provided, the current time is used). In general, it is good practice to use the acquisition time to render images or objects; it helps to synchronize the channels. Sight uses the application time (seconds since the beginning) to render the different channels. If the timestamp of a channel is in the future (either because the acquisition time used is not relative to the start of the application or because the wrong unit is used), then the channel is not updated until the application time catches up with the channel.
This case can be detected by refreshing the web page. If the image/channel updates but then freezes, it means that Sight received the latest message but is waiting for the time to catch up before rendering.
If you own the codelet rendering the image, make sure that you are using the right time in your show call. If you do not own the codelet, then the acquistion time of one of the channels is most likely used to render. Make sure the publisher provides the right acquisition time.
Unable to see the Sight webpage after opening http://localhost:3000 in a browser.
Make sure an application is running on your desktop. If you are running the application on the robot, you must use the robot IP address.
If the “Failed to start Webserver!” error message appears in the logs, the port may not have been released by a previous application.
Determine the application using the port with the following command:
lsof -i TCP:3000
Kill the application:
killall -9 <app name>
If you must run a different app at the same time, update the configuration file to change the port to another open port.
Sight visualization is very slow.
When network bandwidth is insufficient, channels displayed in Sight may show with latency or at very low framerates.
- In Sight, under Channels, uncheck all unnecessary channels. (Tip: You can disable all channels at once by clicking on the application name in the channel menu. Enable all channels you want to visualize by right clicking on a renderer and clicking on ‘enable all channels’).
- Verify Wi-Fi antennas and cables connected to the PCIe Wi-Fi card.
- Change the power mode of the Wi-Fi adapter by checking the file.
cat /etc/NetworkManager/conf.d/default-wifi-powersave-on.conf [connection] wifi.powersave = 2
must be moved to:
wifi.powersave = 3
- Try a wired connection to rule out Wi-Fi bandwidth issues.
- If Sight has been running for a long time, it might have accumulated a lot of data, which slows down the frontend. Try refreshing from time to time to see if it improves performance.
What should I check first if my build is failing?
Make sure you are running Ubuntu 18.04 on your host system, and that you have run the install_dependencies.sh script.
What is the easiest way to generate a capnp id for a new message?
Every capnp file requires a unique ID at the beginning of the file. If you create a new capnp file, attempt to build Isaac without an ID in your new file. An ID is generated and printed in the error message, similar to the following:
messages/my_new.capnp:1:1: error: File does not declare an ID. I've generated one for you. Add this line to your file: @0xcdeac1e381086f01;
As instructed by the error message, add a line with an ‘@’ symbol, the generated ID, and a semicolon to the top of your capnp file. (If you forget the semi-colon, you receive a parse error.)
When running an application that requires TensorFlow, I get the following error: PANIC engine/core/buffers/algorithm_cuda.cpp@55: Could not copy memory. Error: 35
To resolve this error, install CUDA 10.0 using the instructions at NVIDIA CUDA Installation Guide for Linux.
When using a USB Camera on a TX2 I see no image or a heavily distorted image.
This happens when the CPU clock speed is too low on the TX2 and it cannot process the USB frames in
time. A fix for this issue is expected in the next release of Jetpack. In the meantime, try running
jetson_clocks command to increase the CPU clock speed. See
How to Add a New External Dependency¶
Your packages or components may require external dependencies not yet available in Isaac. The right strategy for integrating a new external dependency can vary case by case. There are several examples in the third_party folder. This tutorial explains how to add the zlib library.
First, find a reliable source for your dependency. In the case of zlib, get it from the official zlib webpage at zlib.net. To avoid unexpected problems in your codebase when the external dependency is updated, fix a specific version of the library. This tutorial chooses the version 1.2.11.
Second, add the dependency to your WORKSPACE file using the Bazel rule new_http_archive, as shown in the following code:
new_http_archive( name = "zlib", build_file = clean_dep("//zlib.BUILD"), sha256 = "c3e5e9...cb1a1", strip_prefix = "zlib-1.2.11", url = "https://zlib.net/zlib-1.2.11.tar.gz", )
The main difference between new_http_archive and http_archive is that new_http_archive is for dependencies that are not accompanied by a BUILD file. http_archive is for dependencies that are accompanied by a BUILD file.
See the official Bazel documentation for information on the various parameters for new_http_archive and documentation of all Bazel rules.
Importing external dependencies as an archive is generally preferred over using the rules new_git_repository or git_repository, which import git repositories, because archives are smaller and have less overhead than git repositories.
Third, for the purposes of this tutorial, assume that you have to write a BUILD file for the new external dependency (this may not always be required). In the case of zlib, this is quite straight-forward:
cc_library( name = "zlib", srcs = glob(["*.c", "*.h"], exclude="zlib.h"), hdrs = ["zlib.h"], copts = [ "-Wno-shift-negative-value", "-Wno-implicit-function-declaration", ], includes = ["."], )
This BUILD file defines a single C++ library named zlib using the Bazel rule cc_library. The library includes all C source and header files from zlib and compiles them into a single library. The only external header file is zlib.h, which is excluded from the sources so that the same file does not appear as a source and a header.
In general, srcs is used for files internal to the library and can be used for both source and header files. hdrs is used for publicly facing header files that are required in applications that use the library.
A couple of compiler flags are also required. Add them using the copts option to avoid compiler warnings that are treated as errors due to use of the -Wall option by the Isaac compilation tool.
The new library is ready for use as a dependency in one of your libraries or binaries, similar to the following example:
cc_library( name = "foo", srcs = ["foo.cpp"] hdrs = ["foo.hpp"], deps = ["@zlib"] )
Every external dependency creates its namespace with the same name as the one used for the new_http_archive or corresponding rule. In this case, the name of both the dependency and the library is zlib and thus the shortcut @zlib can be used to refer to the library. If there were a second library foo inside the same archive, the explicit form @zlib//:foo would be needed to reference it.
How To Analyze a Crash using Minidump¶
In the case of a crash, Isaac SDK applications attempt to collect information with breakpad. This information is written into a minidump file in binary format. This section explains how to extract human-readable information from minidump files.
Prepare the Minidump Toolkit¶
Prepare the minidump toolkit in
/tmp/minidump/ with the following command:
Prepare symbols before testing applications by passing the argument ‘-s’ when deploying applications as shown in the following command:
bob@desktop:~/isaac$ ./engine/build/deploy.sh -s -h <robot_ip> -d <device> -p <target> --remote_user <username_on_robot>
where <robot_ip> is the IP address of the robot and <username_on_robot> is your user name on the robot.
When a crash happens, Isaac application minidump file paths are reported on the console similar to the following:
Minidump written to: /tmp/28db00a1-e756-4c47-62f6a7b6-fc26c1a0.dmp
To make the minidump data human-readable, at a desktop that has symbols prepared, run the following command:
bob@desktop:~/isaac$ ./engine/build/scripts/process_jetson_minidump.sh -h <robot_ip> -d /tmp/28db00a1-e756-4c47-62f6a7b6-fc26c1a0.dmp --remote_user <username_on_robot>
where <robot_ip> is the IP address of the robot and <username_on_robot> is your user name on the robot. If the minidump file is locally available, run the following command:
bob@desktop:~/isaac$ ./engine/build/scripts/process_minidump.sh <minidump>
Crash information is presented on the console, and is similar to the following:
Crash reason: SIGSEGV /SEGV_MAPERR Crash address: 0x0 Process uptime: not available Thread 11 (crashed) 0 realsense_camera!isaac::RealsenseCamera::tick() [RealsenseCamera.cpp : 533 + 0x4] x0 = 0x0000000000000000 x1 = 0x0000007f962b2180 x2 = 0x0000007fa0000080 x3 = 0x0000007f800008d0 x4 = 0x0000000000000007 x5 = 0x00000000007e8ac8 x6 = 0x00000000007e89f0 x7 = 0x0000000000063762 x8 = 0x00000000000000d7 x9 = 0x001dcd6500000000 x10 = 0x000000005c787d6b x11 = 0x000000000f171c90 x12 = 0x0000000000000017 x13 = 0x000000005c787d6b x14 = 0x00076f61bec52c49 x15 = 0x00001c2c8948c9df x16 = 0x0000007fa7708688 x17 = 0x0000007fa7471838 x18 = 0x0000007fa005bdcc x19 = 0x0000000037cc4610 x20 = 0x0000007fa004bbb0 x21 = 0x0000007fa0248c01 x22 = 0x0000000037cc47e0 x23 = 0x00000006556cdce8 x24 = 0x000000000091f000 x25 = 0x0000007fa53dd330 x26 = 0x0000000000000000 x27 = 0x0000007f962b31b0 x28 = 0x0000000000000001 fp = 0x0000007f962b20a0 lr = 0x0000000000459f6c sp = 0x0000007f962b20a0 pc = 0x0000000000459f74