Running the Navigation Stack in Isaac Sim

This section will guide you through the process of installing the Navigation Stack on your machine and running a simple demo with the Nova Carter platform in Omniverse Isaac Sim.

Isaac SDK provides a prebuilt Docker image that contains the entire Navigation Stack. Omniverse Isaac Sim currently supports Isaac AMR in a specific prebuilt Docker image, not its latest release image. The instructions below will specify which Omniverse Isaac Sim container to use.

Follow these instructions to install Docker and access the images through the NVIDIA Container Registry.

The Navigation Stack has many configuration options. This tutorial focuses on running the Navigation Stack with lidar-based localization and wheel-inertial odometry. To run the navigation stack, first create a configuration file at /tmp/config.yaml, then copy the following configuration to the file.

            

            
robot: carter
physics_engine: isaac-sim
localization: lidar
waypoint_graph_generator: grid-map
route_planner: onboard
path_planner: grid-map
omap_path: apps/assets/maps/full_warehouse.png
omap_cell_size: 0.05
environment: full_warehouse
linear_speed_limit: 1.111
robot_goal_pose:
   translation: [56.57, 11.35, 0.0]
   rotation_rpy: [0.0, 0.0, 135.0]
world_T_map:
   translation: [41.37500152587891, 7.975, 0.0]
   rotation_rpy: [0.0, 0.0, -90.0]
        

After creating the configuration file, run the following command:

            

            
docker run -it --gpus all --rm --network=host -v /tmp:/tmp \
   nvcr.io/<your_staging_area>/navigation_stack:isaac_2.0-k8 \
   -c /tmp/config.yaml
        

After running the above command, the Navigation Stack will initialize and begin searching for a TCP connection to Isaac Sim, displaying output like the following:

            

            
2023-07-13 23:16:57.305 INFO  external/com_nvidia_gxf/gxf/network/tcp_client_socket.cpp@53: TCP close 32
2023-07-13 23:16:57.305 WARN  external/com_nvidia_gxf/gxf/network/tcp_client_socket.cpp@179: Failed to connect to TCP server 127.0.0.1:7000
        

The next section will cover running Isaac Sim and connecting it to the Navigation Stack, at which point these messages will stop and the Navigation Stack will drive the robot in the simulation.

Use a web browser to navigate to http://<ip_address>:3000, where <ip_address> is the IP address of the machine on which the Docker container is running. This webpage contains the Websight visualization of the Navigation Stack.

Until Isaac Sim is connected to the Navigation Stack and the simulation is running, this webpage will be blank. Once the simulation is running, you should see the occupancy map, along with the robot moving autonomously towards the goal point, which is visualized with a red circle. You can reposition the goal by clicking and dragging the marker, causing the robot to replan its path and move to the new goal. Note that once you have moved the marker to a new position, you have to click elsewhere for the new goal to be registered.

You should expect to see a webpage similar to the following.

Running Isaac Sim on AWS (optional)

To learn how to deploy Isaac Sim on Amazon Web Services (AWS), follow the official Isaac Sim documentation here, with two minor modifications:

• In Section 4.1.1, Requirements, Step 2, replace the referenced AMI with the AMI at this link. This will enable live web streaming and Isaac Sight when deploying the Isaac Sim container in the cloud instance.

Note

When you open AMI on AWS, it may default to “p2.2xlarge” (i.e. V100). Instead, make sure to specify “g4dn.2xlarge” or better (for T4) or “g5.2xlarge” or better (for A10G).

• Follow the Isaac Sim documentation until Section 4.1.4, Container Deployment, at which point you should return to this document to learn how to launch and configure Isaac Sim to connect to the Isaac AMR navigation stack.

Deploying the Isaac Sim Container

From the terminal run the Omniverse Isaac Sim container using the following commands:

            

            
xhost +
docker login nvcr.io
docker run --entrypoint bash -it --gpus all --rm --network=host --privileged \
-v $HOME/.Xauthority:/root/.Xauthority \
-e DISPLAY \
-v /dev:/dev \
-v /sys:/sys \
-v ~/docker/isaac-sim/kit/cache/Kit:/isaac-sim/kit/cache/Kit:rw \
-v ~/docker/isaac-sim/cache/ov:/root/.cache/ov:rw \
-v ~/docker/isaac-sim/cache/pip:/root/.cache/pip:rw \
-v ~/docker/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw \
-v ~/docker/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw \
-v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw \
-v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw \
-v ~/docker/isaac-sim/documents:/root/Documents:rw \
nvcr.io/<your_staging_area>/isaac-sim:isaac_2.0-k8
        

Sample assets provided with this container include the Nova Carter robot, the NVIDIA Hawk sensor, a “walled room” environment with the Nova Carter robot, and a large warehouse environment with the Nova Carter robot. This documentation will describe how to run the Navigation Stack to move the robot in the provided sample scenes.

The Omniverse Isaac Sim documentation is available at this link. It covers how to use Isaac Sim in general, but does not currently include instructions for the GXF Bridge Extension, which is necessary to run Isaac Sim in-the-loop with the Isaac AMR Navigation Stack. Instructions specific to GXF and Isaac AMR support are provided below.

Isaac Sim can be run manually or via the Python API as part of an automated workflow. If run manually, you will be presented with a GUI to control the simulation and run the Navigation Stack. If run via Python, Isaac Sim can run with a GUI or without a GUI (i.e. headless). For ease of use, the container includes a Python script to load a robot into a scene at a specified initial pose; this script is intended to run in parallel with the Navigation Stack, with or without the Isaac Sim GUI. More complex functionality is enabled via the Isaac Sim Python API, documented here.

Running Isaac Sim manually is recommended for first-time demonstration and/or workflow testing.

Running Isaac Sim Manually

Follow these steps to connect Isaac Sim to the Navigation Stack and move the virtual robot.

1. Launch Isaac Sim:

   1. If running Isaac Sim from a local desktop, launch Isaac Sim manually from the container terminal as follows:

      Copy

      Copied!

                  
      
                  
      ./isaac-sim.sh --allow-root
              

   2. If running Isaac Sim from AWS, instead follow steps 5-7 of Section 4.14, Container Deployment in the Isaac Sim AWS Deployment documentation.

2. If a tab labeled GXF Bridge does not appear at the bottom of the window (see image below), follow these steps to enable the GXF Bridge extension:

   1. Select Window > Extension from the menu at the top of the Isaac Sim window.

      

   2. Search for “GXF Bridge” in the extensions list.

   3. Toggle the switch next to “GXF Bridge” to enable it. The switch should turn green.

      

3. If not already on a blank stage, click File > New to open a blank stage.

   

4. Open the sample scene corresponding to the Navigation Stack environment selected for the client app:

   1. Select the Isaac Assets (Beta) tab located at the bottom of the window.

   2. Click on Samples in the drop-down menu situated on the left side of the tab.

   3. Allow a moment for the Samples window to populate.

   4. Scroll down to find Isaac_AMR and double-click either full_warehouse_carter_v2_4_gxf.usd or walled_room_carter_v2_4_gxf.usd, depending on the environment you chose in the navigation stack configuration above.

   Note

   You must select the Isaac Sim scene corresponding to the omap_path and environment specified in the navigation stack configuration to ensure the navigation stack uses the occupancy map generated from the Isaac Sim scene you are using. Otherwise, navigation will not succeed.

   

5. Select the GXF Bridge tab, and confirm the Graph Path is set to the tcp_server.yaml file. You can modify the graph here to anything on your local system; by default, this file will launch a TCP server application designed to connect with the Navigation Stack.

   

6. Click “Create Application” to start the graph in Isaac Sim.

   

7. When the Navigation Stack has finished initializing, it will look for a TCP connection, displaying output like:

   Copy

   Copied!

               
   
               
   2023-07-13 23:16:57.305 INFO  external/com_nvidia_gxf/gxf/network/tcp_client_socket.cpp@53: TCP close 32
   2023-07-13 23:16:57.305 WARN  external/com_nvidia_gxf/gxf/network/tcp_client_socket.cpp@179: Failed to connect to TCP server 127.0.0.1:7000
           

   After this happens, click the Play button in Isaac Sim. The server and client will connect, and the Navigation Stack will begin.

   

8. Refresh the Isaac Sight browser page you opened earlier. You should see the robot visualized on top of a 2D occupancy map.

   Additionally, you should see a visualized circle with the subscript robot/goal. This is the marker that represents the goal position of the robot. If the goal position is not on top of the robot, you should also see a visualized path starting at the robot and ending at the goal.

   You can update the robot goal position by clicking and dragging the circle representing the goal position.

   Note

   If the simulation is paused (see below), Isaac Sight will also pause.

9. At any point, you can pause the simulation by clicking the Pause button in Isaac Sim. This will also pause the Navigation Stack . You can then click the Play button to resume simulation and navigation.

   

10. At any point, you can stop the simulation by clicking Destroy Application in the GXF Bridge tab. This will automatically stop the simulation, reset the robot to its initial position in the scene, and end the TCP server application. To restart the simulation, close the Navigation Stack app via Ctrl-C, restart the Navigation Stack app following the instructions above, then follow steps 2-8.

    

    Note

    If the simulation is stopped by clicking the Stop button, the server application will persist. After this point, the Navigation Stack will end fatally if you click Play in Isaac Sim, as the simulation clock has been reset while the Navigation Stack clock has merely paused. If you click Stop, ensure you also click Destroy Application to avoid this issue.

Running Isaac Sim via Python

From the container terminal, examine the bundled run_scene.py script:

            

            
./python.sh tools/isaac_amr/run_scene.py -h
        

The run_scene.py script will then show command-line interface options. You must specify, at minimum, the path to the scene matching the environment that the navigation stack is using.

Isaac AMR sample assets can be found in /Isaac/Samples/Isaac_AMR, with the following directory structure:

• Environments: Contains custom environments developed for testing Nova Carter in Isaac Sim.

  • walled_room.usd: A simple enclosed space with two block obstacles

• Robots: Contains the Nova Carter robot

  • carter_v2_4_gxf.usd: The Nova Carter robot with 3D PhysX-based lidar, IMU, differential wheel base, and 2x Hawk sensor. It also includes Omnigraph-enabled sensor/control message exchange with Navigation Stack.

• Scenarios: Contains sample scenarios that are built from existing environments and include the Nova Carter robot.

  • full_warehouse_carter_v2_4_gxf.usd: A sample warehouse with multiple shelves, a forklift, and Nova Carter

  • walled_room_carter_v2_4_gxf.usd: The walled room environment described above, with Nova Carter in the center

• Sensors: Contains the Hawk sensor.

  • hawk.usd: The Isaac Sim model of the Hawk sensor, including distortion effects

After specifying the path to the USD scene, specify the YAML server graph that ships with Isaac Sim: exts/omni.isaac.gxf_bridge/data/config/tcp_server.yaml. If running a custom scene that includes an ActionGraph that will generate the TCP server application, you do not need to specify additional YAML files on the command line.

The following is an example command to run the large warehouse scene with Nova Carter at 60Hz for 60,0000 frames (~1000 seconds).

            

            
./python.sh tools/isaac_amr/run_scene.py \
/Isaac/Samples/Isaac_AMR/Scenarios/full_warehouse_carter_v2_4_gxf.usd \
exts/omni.isaac.gxf_bridge/data/config/tcp_server.yaml \
-r 60 \
-n 60000
        

The following is an example command to run the large warehouse scene and place the Carter v2.3 asset at a new location/orientation.

            

            
./python.sh tools/isaac_amr/run_scene.py \
/Isaac/Environments/Simple_Warehouse/full_warehouse.usd \
exts/omni.isaac.gxf_bridge/data/config/tcp_server.yaml \
--run_indefinitely \
--robot carter_v2_3 \
--initial_pos 5.0 2.0 \
--initial_yaw 90
        

You may now open Isaac Sight as described above to visualize the Navigation Stack.

Using a Custom Environment (Optional)

You may wish to test the Navigation Stack in a different scene than those packaged with Isaac AMR. To do so, you must first generate a new occupancy map, then configure the Navigation Stack to use the new occupancy map.

1. Open the scene in Isaac Sim.

2. Temporarily remove any dynamic objects (eg. moving obstacles, robots, etc.) from the scene.

3. Follow the instructions here to generate and save a new occupancy map for your scene. When you reach the Visualization Window, record the information in the box above the occupancy map image, then save the map to (eg) /tmp/occupancy_map.png.

4. Restore the dynamic objects to the scene, or simply re-open the scene without saving it.

5. Generate a configuration file for the Navigation Stack as described here, or copy an existing configuration provided above into /tmp/config.yaml.

6. Make the following additions or changes to /tmp/config.yaml:

   Copy

   Copied!

               
   
               
   omap_path: /tmp/occupancy_map.png
   omap_cell_size: 0.05
   omap_to_world_transform:
      translation: [17.975, 11.924999809265136, 0.0]
      rotation_rpy: [0.0, 0.0, -90.0]
           

   1. Replace the omap_path value with the path to the occupancy map you generated.

   2. Replace the omap_cell_size value with the Cell Size you set when generating the occupancy map, if you modified it from the default value of 0.05.

   3. Replace the omap_to_world_transform/translation x- and y-coordinates with the Coordinates of the top left of the image (pixel 0, 0) you recorded in Step 3.

7. Optionally, specify the initial goal pose of the robot in the scene:

   Copy

   Copied!

               
   
               
   goal_frame_initial_pose:
      translation: [56.57, 11.35, 0.0]
      rotation_rpy: [0.0, 0.0, 135.0]
           

   The x- and y- values of translation will be the desired goal position in omap coordinates, and the yaw value in rotation_rpy will be the desired goal yaw orientation in degrees.

8. Follow the steps above to run the Navigation Stack with the new configuration and occupancy map.

Limitations

There is a known memory leak in Isaac Sim when the Quick Search and Curve Editor extensions are enabled. These extensions are automatically disabled on Isaac Sim start when running manually or via Python. Note that, if either extension is enabled, Isaac Sim memory consumption may grow by 50% over 24 hours.

The Isaac Sim run_scene.py script’s –robot option only supports adding Nova Carter to a scene. However, the source code is readily available in the container and serves as a useful example for connecting Nova Carter to the Navigation Stack using the Isaac Sim Python API. The syntax for using the Nova Carter GXF asset is the same as what is provided in the run_scene.py script - only the USD path needs to be modified.