Running Isaac SDK on Kaya

Installation

This guide takes you through the software setup necessary for running Isaac SDK after you have finished building your own NVIDIA Kaya robot.

1. Install the operating system for your Jetson Nano. Follow the procedures in the Getting Started With Jetson Nano guide to complete installation of your Jetson Nano Developer Kit.

  1. Follow the Setup guide for installing Isaac SDK along with all its dependencies.

3. Obtain the IP address of the robot. Connect a keyboard, mouse, display, and boot the robot. At a terminal prompt, enter the following command:

bob@jetson:~/$ ip addr show
  1. Install all the software dependencies on the robot. Run the following command from the desktop:
bob@desktop:~/isaac$ engine/build/scripts/install_dependencies_jetson.sh -u <jetson_username> -h <jetson_ip>

where <jetson_username> is your username on the robot, and <jetson_ip> is the IP address of the robot.

5. This step is optional but makes development easier. Make sure you have an SSH key on your desktop machine. Then copy your SSH identity to the robot using the user name and IP you use to login on the robot with a command similar to the following:

bob@desktop:~/isaac$ ssh-copy-id <username_on_robot>@<robot_ip>

where <username_on_robot> is your user name on the robot, and <robot_ip> is the IP address of the robot.

Running your first application : Joystick

The following steps deploy a simple joystick application that uses a paired Joystick to control the robot. The same steps can be used for deploying and running other applications on your robot.

1. To run the joystick application (//apps/kaya:joystick) on the robot, deploy the package to the robot from your desktop with the following command:

bob@desktop:~/isaac$ ./engine/build/deploy.sh --remote_user <username_on_robot> -p //apps/kaya:joystick-pkg -d jetpack43 -h <robot_ip>

where <robot_ip> is the IP address of the robot and <username_on_robot> is your user name on the robot.

Note

If a username is not specified with the –remote-user option, the default username used is nvidia.

The -d jetpack43 option specifies that you are building and deploying to a Jetson device with Jetpack version 4.3.

  1. Login to the robot to run the application:
bob@jetson:~/isaac$ ssh ROBOTUSER@ROBOTIP
  1. Go to the deployment folder on the robot and run the application:
bob@jetson:~/$ cd deploy/<bob>/joystick-pkg/
bob@jetson:~/deploy/<bob>/joystick-pkg$ ./apps/kaya/joystick

Where <bob> is your username on the host system.

A button on the Joystick functions as a deadman trigger. The robot will follow joystick commands only when the trigger is pressed.

The Follow Me Application

The following steps deploy an application to have the Kaya robot autonomously move towards a designated AprilTag fiducial. This application combines AprilTag detection, path planning, control, and the Kaya driver. It requires a working realsense camera for AprilTag detection and path planning, and a joystick to enable autonomous mode.

  1. Deploy //apps/kaya:follow_me-pkg to the robot as explained in Deploying and Running on Jetson.

  2. Change to the directory on your Nano and run the application with the following commands:

    cd deploy/<your_username>/follow_me-pkg/
    ./apps/kaya/follow_me
    
  3. Open Isaac Sight on the desktop browser at <nano_ip>:3000. In the Application Configuration panel on the right, click on fiducial_as_goal, and change the target_fiducial_id (default is “tag36h11_9”) to the id of the AprilTag in use.

    If you put the AprilTag within the realsense camera’s field of view, you should see in the Follower Kaya - Camera window that the april tag is detected, and a planned path shown in a blue line from the Kaya robot to the april tag.

  4. If you hold down the R1 button on the joystick, the Kaya robot enters autonomous mode, and moves towards the april tag. Move the april tag around while keeping it within the camera’s view, and Kaya follows it around.

The Mapping Application

The gmapping application demonstrates use of the Kaya robot to build an occupancy map of its environment. Kaya uses the Intel RealSense camera to perceive depth of obstacles, and wheel odometry and IMU to update its own state. Obstacle depth and robot state information are published to the host to build up a map.

This application has two parts: gmapping_distributed_kaya that runs on the Kaya robot, and gmapping_distributed_host that runs on the host machine.

  1. Deploy //apps/kaya:gmapping_distributed_kaya-pkg to the robot as explained in Deploying and Running on Jetson.

  2. Change to the directory on your Nano and run the application with the following commands:

    cd deploy/<your_username>/gmapping_distributed_kaya-pkg/
    ./apps/kaya/gmapping_distributed_kaya
    

    Open Isaac Sight on the desktop browser at <nano_ip>:3000. You should be able to see the camera RGB and depth image, as well as plot of the robot state.

  3. Use the joystick to drive Kaya around and you should see the robot state being updated.

  4. On the desktop system, open app/kaya/gmapping_distributed_host.app.json and change the tcp-subscriber host from “YOUR_NANO_IP_HERE” to the IP address of the Jetson Nano on the robot (<nano_ip>).

  5. Build and run the host application with the following commands:

    bazel build //apps/kaya:gmapping_distributed_host
    bazel run //apps/kaya:gmapping_distributed_host
    
  6. Open Isaac Sight on the desktop at localhost:3000. You should be able to see the map Kaya is building. Use the joystick to drive Kaya around and observe the map updates.

    If you see performance issues with the map update, go to the Kaya page of Isaac Sight webpage and disable camera RGB and depth visualization (uncheck Channels - viewer).