Frequently Asked Questions

This section provides answers to frequently asked questions about your release. Use it as the first step in troubleshooting problems.

Issues with Hardware Components

When I run my application that uses a ZED Camera I get the following error:

engine/alice/backend/modules.cpp@74: dlopen failed: libhidapi-libusb.so.0: cannot open shared
object file: No such file or directory

The libusb-dev library is not installed. Install libusb-dev and all other required dependencies on the robot with the following command:

bob@desktop:~/isaac$ ./engine/build/scripts/install_dependencies_jetson.sh -h <Jetson_IP>

Where <Jetson_IP> is the IP address of the robot.

I am running my app on Kaya. Everything runs fine, but the joystick does not move the robot.

Make sure you press the “deadman switch” L1 on the joystick while using the direction knobs. Pressing L1 is required to prevent unwanted motions.

When I run my application that uses a BMI160 IMU I get the following error:

I2C Error: Device or resource busy (errno 16)

The kernel module nvs_bmi160 is using the I2C resource needed by the application. Unload the module with the command sudo rmmod nvs_bmi160. You can also run the install_dependencies_jetson.sh script, which prevents nvs_bmi160 from loading.

bob@desktop:~/isaac$ ./engine/build/scripts/install_dependencies_jetson.sh -h <Jetson_IP>

Issues with the Visualization and WebSight

I don’t see images in Sight. Or the images are blank. Or the labels displayed are red/yellow.

Reasons why a channel does not update correctly:

  • The first thing to check is whether the channel is enabled in the bottom left menu. If the channel is not enabled, the robot does not send the update to the frontend and the channel appears in red in the renderer’s legend. Enable this channel either from the channel menu directly or right click on the renderer and click ‘enable all channels’.

  • The channel might be enabled, but there is no valid transformation between the channel and the reference frame used by the renderer. If this is the case, the channel appears in yellow in the renderer’s legend. If you hover over the channel name, additional information is provided about the missing transformation: ‘[reference_T_channel] is not defined’. You can check the PoseTree widget to figure out if there is a path between both coordinate frames. (Note: ‘#####’’ is used as a default frame. It means that no frame has been provided for the channel.)

  • Your image/channel is rendered, but a later channel is overriding it. Channels are rendered in the order they appear in the legend: the top one first, followed by second one, and so on. If you render an image the size of the renderer, it covers all the previous channels. Make sure your channels are sorted in the right order. You can change the order by right clicking on the renderer and clicking on ‘settings’. From there you can use the arrows on the left side of the channels to update the order.

  • Each show operation in Sight has its own timestamp (if none is provided, the current time is used). In general, it is good practice to use the acquisition time to render images or objects; it helps to synchronize the channels. Sight uses the application time (seconds since the beginning) to render the different channels. If the timestamp of a channel is in the future (either because the acquisition time used is not relative to the start of the application or because the wrong unit is used), then the channel is not updated until the application time catches up with the channel.

    This case can be detected by refreshing the web page. If the image/channel updates but then freezes, it means that Sight received the latest message but is waiting for the time to catch up before rendering.

    If you own the codelet rendering the image, make sure that you are using the right time in your show call. If you do not own the codelet, then the acquistion time of one of the channels is most likely used to render. Make sure the publisher provides the right acquisition time.

Unable to see the Sight webpage after opening http://localhost:3000 in a browser.

Make sure an application is running on your desktop. If you are running the application on the robot, you must use the robot IP address.

If the “Failed to start Webserver!” error message appears in the logs, the port may not have been released by a previous application.

Determine the application using the port with the following command:

lsof -i TCP:3000

Kill the application:

killall -9 <app name>

If you must run a different app at the same time, update the configuration file to change the port to another open port.

Sight visualization is very slow.

When network bandwidth is insufficient, channels displayed in Sight may show with latency or at very low framerates.

  • In Sight, under Channels, uncheck all unnecessary channels. (Tip: You can disable all channels at once by clicking on the application name in the channel menu. Enable all channels you want to visualize by right clicking on a renderer and clicking on ‘enable all channels’).
  • Verify Wi-Fi antennas and cables connected to the PCIe Wi-Fi card.
  • Change the power mode of the Wi-Fi adapter by checking the file.
cat /etc/NetworkManager/conf.d/default-wifi-powersave-on.conf
[connection]
wifi.powersave = 2

must be moved to:

wifi.powersave = 3
  • Try a wired connection to rule out Wi-Fi bandwidth issues.
  • If Sight has been running for a long time, it might have accumulated a lot of data, which slows down the frontend. Try refreshing from time to time to see if it improves performance.

Issues with the navigation stack

The global planner is taking forever to start or to run?

If the global planner is too slow, you can try to reduce some parameters:

  • graph_initialization_steps: this parameter controls the number of random sampling done in the start function
  • graph_in_tick_steps: this parameter controls the number of random sampling done each tick. If the graph is already dense enough, this parameter might be set to 0.

If reducing the number of random sampling is not possible without compromising the quality of the navigation graph produced, you should consider pre-computing the graph and load it from a file. To produce this graph, you can follow this steps:

  • Run the application with graph_initialization_steps set to a high value (you can also set graph_in_tick_steps to a high value to keep increasing the graph each tick).
  • Choose a destination to save the current graph and update the parameter: graph_file_out. Wait for the graph to be big enough and kill the app.
  • You can now restart the app setting the parameter graph_file_in to the file containing the graph.

The global planner can’t find a path or produce a suboptimal path?

If your map is big, you probably need to generate a big graph. You can increase the parameter graph_initialization_steps, or you should consider loading the graph from a file. To generate a graph, you can follow the steps described the question above.

Other Issues

What should I check first if my build is failing?

Make sure you are running Ubuntu 18.04 on your host system, and that you have run the install_dependencies.sh script.

What is the easiest way to generate a capnp id for a new message?

Every capnp file requires a unique ID at the beginning of the file. If you create a new capnp file, attempt to build Isaac without an ID in your new file. An ID is generated and printed in the error message, similar to the following:

messages/my_new.capnp:1:1: error: File does not declare an ID. I've generated one for you.
Add this line to your file: @0xcdeac1e381086f01;

As instructed by the error message, add a line with an ‘@’ symbol, the generated ID, and a semicolon to the top of your capnp file. (If you forget the semi-colon, you receive a parse error.)

When running an application that requires TensorFlow, I get the following error: PANIC engine/core/buffers/algorithm_cuda.cpp@55: Could not copy memory. Error: 35

To resolve this error, install CUDA 10.0 using the instructions at NVIDIA CUDA Installation Guide for Linux.

When using a USB Camera on a TX2 I see no image or a heavily distorted image.

This happens when the CPU clock speed is too low on the TX2 and it cannot process the USB frames in time. A fix for this issue is expected in the next release of Jetpack. In the meantime, try running the jetson_clocks command to increase the CPU clock speed. See https://devtalk.nvidia.com/default/topic/1055732/jetson-tx2/usb-port-disabled-when-no-hdmi-connected-/post/5362568/#5362568.

How to Add a New External Dependency

Your packages or components may require external dependencies not yet available in Isaac. The right strategy for integrating a new external dependency can vary case by case. There are several examples in the third_party folder. This tutorial explains how to add the zlib library.

First, find a reliable source for your dependency. In the case of zlib, get it from the official zlib webpage at zlib.net. To avoid unexpected problems in your codebase when the external dependency is updated, fix a specific version of the library. This tutorial chooses the version 1.2.11.

Second, add the dependency to your WORKSPACE file using the Bazel rule new_http_archive, as shown in the following code:

new_http_archive(
    name = "zlib",
    build_file = clean_dep("//zlib.BUILD"),
    sha256 = "c3e5e9...cb1a1",
    strip_prefix = "zlib-1.2.11",
    url = "https://zlib.net/zlib-1.2.11.tar.gz",
)

The main difference between new_http_archive and http_archive is that new_http_archive is for dependencies that are not accompanied by a BUILD file. http_archive is for dependencies that are accompanied by a BUILD file.

See the official Bazel documentation for information on the various parameters for new_http_archive and documentation of all Bazel rules.

Importing external dependencies as an archive is generally preferred over using the rules new_git_repository or git_repository, which import git repositories, because archives are smaller and have less overhead than git repositories.

Third, for the purposes of this tutorial, assume that you have to write a BUILD file for the new external dependency (this may not always be required). In the case of zlib, this is quite straight-forward:

cc_library(
  name = "zlib",
  srcs = glob(["*.c", "*.h"], exclude="zlib.h"),
  hdrs = ["zlib.h"],
  copts = [
    "-Wno-shift-negative-value",
    "-Wno-implicit-function-declaration",
  ],
  includes = ["."],
)

This BUILD file defines a single C++ library named zlib using the Bazel rule cc_library. The library includes all C source and header files from zlib and compiles them into a single library. The only external header file is zlib.h, which is excluded from the sources so that the same file does not appear as a source and a header.

In general, srcs is used for files internal to the library and can be used for both source and header files. hdrs is used for publicly facing header files that are required in applications that use the library.

A couple of compiler flags are also required. Add them using the copts option to avoid compiler warnings that are treated as errors due to use of the -Wall option by the Isaac compilation tool.

The new library is ready for use as a dependency in one of your libraries or binaries, similar to the following example:

cc_library(
  name = "foo",
  srcs = ["foo.cpp"]
  hdrs = ["foo.hpp"],
  deps = ["@zlib"]
)

Every external dependency creates its namespace with the same name as the one used for the new_http_archive or corresponding rule. In this case, the name of both the dependency and the library is zlib and thus the shortcut @zlib can be used to refer to the library. If there were a second library foo inside the same archive, the explicit form @zlib//:foo would be needed to reference it.

How To Analyze a Crash using Minidump

In the case of a crash, Isaac SDK applications attempt to collect information with breakpad. This information is written into a minidump file in binary format. This section explains how to extract human-readable information from minidump files.

Prepare the Minidump Toolkit

Prepare the minidump toolkit in /tmp/minidump/ with the following command:

bob@desktop:~/isaac$ ./engine/build/scripts/prepare_minidump_tools.sh

Prepare Symbols

Prepare symbols before testing applications by passing the argument ‘-s’ when deploying applications as shown in the following command:

bob@desktop:~/isaac$ ./engine/build/deploy.sh -s -h <robot_ip> -d <device> -p <target>
--remote_user <username_on_robot>

where <robot_ip> is the IP address of the robot and <username_on_robot> is your user name on the robot.

Analyzing Minidumps

When a crash happens, Isaac application minidump file paths are reported on the console similar to the following:

Minidump written to: /tmp/28db00a1-e756-4c47-62f6a7b6-fc26c1a0.dmp

To make the minidump data human-readable, at a desktop that has symbols prepared, run the following command:

bob@desktop:~/isaac$ ./engine/build/scripts/process_jetson_minidump.sh -h <robot_ip> -d
/tmp/28db00a1-e756-4c47-62f6a7b6-fc26c1a0.dmp --remote_user <username_on_robot>

where <robot_ip> is the IP address of the robot and <username_on_robot> is your user name on the robot. If the minidump file is locally available, run the following command:

bob@desktop:~/isaac$ ./engine/build/scripts/process_minidump.sh <minidump>

Crash information is presented on the console, and is similar to the following:

Crash reason:  SIGSEGV /SEGV_MAPERR
Crash address: 0x0
Process uptime: not available

Thread 11 (crashed)
 0  realsense_camera!isaac::RealsenseCamera::tick() [RealsenseCamera.cpp : 533 + 0x4]
     x0 = 0x0000000000000000    x1 = 0x0000007f962b2180
     x2 = 0x0000007fa0000080    x3 = 0x0000007f800008d0
     x4 = 0x0000000000000007    x5 = 0x00000000007e8ac8
     x6 = 0x00000000007e89f0    x7 = 0x0000000000063762
     x8 = 0x00000000000000d7    x9 = 0x001dcd6500000000
    x10 = 0x000000005c787d6b   x11 = 0x000000000f171c90
    x12 = 0x0000000000000017   x13 = 0x000000005c787d6b
    x14 = 0x00076f61bec52c49   x15 = 0x00001c2c8948c9df
    x16 = 0x0000007fa7708688   x17 = 0x0000007fa7471838
    x18 = 0x0000007fa005bdcc   x19 = 0x0000000037cc4610
    x20 = 0x0000007fa004bbb0   x21 = 0x0000007fa0248c01
    x22 = 0x0000000037cc47e0   x23 = 0x00000006556cdce8
    x24 = 0x000000000091f000   x25 = 0x0000007fa53dd330
    x26 = 0x0000000000000000   x27 = 0x0000007f962b31b0
    x28 = 0x0000000000000001    fp = 0x0000007f962b20a0
     lr = 0x0000000000459f6c    sp = 0x0000007f962b20a0
     pc = 0x0000000000459f74