Running the Navigation Stack in Isaac Sim

This section will guide you through the process of installing the Navigation Stack on your machine and running a simple demo with the Carter platform in Omniverse Isaac Sim.

Isaac SDK provides a prebuilt Docker image that contains the entire Navigation Stack. Omniverse Isaac Sim currently supports Isaac AMR in a specific prebuilt Docker image, not its latest release image. The instructions below will specify which Omniverse Isaac Sim container to use.

Follow these instructions to install Docker and access the images through the NVIDIA Container Registry.

The Navigation Stack has many configuration options. This tutorial focuses on running the Navigation Stack with lidar-based localization and wheel-inertial odometry. To run the navigation stack, first create a configuration file at /tmp/config.yaml, then copy the following configuration to the file.

            

            
robot: carter
physics_engine: isaac-sim
localization: lidar
waypoint_graph_generator: grid-map
route_planner: onboard
path_planner: grid-map
omap_path: apps/assets/maps/full_warehouse.png
omap_cell_size: 0.05
environment: full_warehouse
linear_speed_limit: 1.111
robot_goal_pose:
   translation: [56.57, 11.35, 0.0]
   rotation_rpy: [0.0, 0.0, 135.0]
world_T_map:
   translation: [41.37500152587891, 7.975, 0.0]
   rotation_rpy: [0.0, 0.0, -90.0]
        

After creating the configuration file, run the following command:

            

            
docker run -it --gpus all --rm --network=host -v /tmp:/tmp \
   nvcr.io/<your_staging_area>/navigation_stack:isaac_2.0-k8 \
   -c /tmp/config.yaml
        

After running the above command, the Navigation Stack will initialize and begin searching for a TCP connection to Isaac Sim, displaying output like the following:

            

            
2023-07-13 23:16:57.305 INFO  external/com_nvidia_gxf/gxf/network/tcp_client_socket.cpp@53: TCP close 32
2023-07-13 23:16:57.305 WARN  external/com_nvidia_gxf/gxf/network/tcp_client_socket.cpp@179: Failed to connect to TCP server 127.0.0.1:7000
        

The next section will cover running Isaac Sim and connecting it to the Navigation Stack, at which point these messages will stop and the Navigation Stack will drive the robot in the simulation.

Use a web browser to navigate to http://<ip_address>:3000, where <ip_address> is the IP address of the machine on which the Docker container is running. This webpage contains the Websight visualization of the Navigation Stack.

Until Isaac Sim is connected to the Navigation Stack and the simulation is running, this webpage will be blank. Once the simulation is running, you should see the occupancy map, along with the robot moving autonomously towards the goal point, which is visualized with a red circle. You can reposition the goal by clicking and dragging the marker, causing the robot to replan its path and move to the new goal. Note that once you have moved the marker to a new position, you have to click elsewhere for the new goal to be registered.

You should expect to see a webpage similar to the following.

To learn how to deploy Isaac Sim on Amazon Web Services (AWS), follow the official Isaac Sim documentation here, with two minor modifications:

• In Section 4.1.1, Requirements, Step 2, replace the referenced AMI with the AMI at this link. This will enable live web streaming and Isaac Sight when deploying the Isaac Sim container in the cloud instance.

Note

When you open AMI on AWS, it may default to “p2.2xlarge” (i.e. V100). Instead, make sure to specify “g4dn.2xlarge” or better (for T4) or “g5.2xlarge” or better (for A10G).

• Follow the Isaac Sim documentation until Section 4.1.4, Container Deployment, at which point you should return to this document to learn how to launch and configure Isaac Sim to connect to the Isaac 2.0 navigation stack.

From the terminal run the Omniverse Isaac Sim container using the following commands:

            

            
xhost +
docker login nvcr.io
docker run --entrypoint bash -it --gpus all --rm --network=host --privileged \
-v $HOME/.Xauthority:/root/.Xauthority \
-e DISPLAY \
-v /dev:/dev \
-v /sys:/sys \
-v ~/docker/isaac-sim/kit/cache/Kit:/isaac-sim/kit/cache/Kit:rw \
-v ~/docker/isaac-sim/cache/ov:/root/.cache/ov:rw \
-v ~/docker/isaac-sim/cache/pip:/root/.cache/pip:rw \
-v ~/docker/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw \
-v ~/docker/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw \
-v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw \
-v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw \
-v ~/docker/isaac-sim/documents:/root/Documents:rw \
nvcr.io/<your_staging_area>/isaac-sim:isaac_2.0-k8
        

Sample assets provided with this container include the Carter v2.3 robot, the NVIDIA Hawk sensor, a “walled room” environment with the Carter v2.3 robot, and a large warehouse environment with the Carter v2.3 robot. This documentation will describe how to run the Navigation Stack to move the robot in the provided sample scenes.

The Omniverse Isaac Sim documentation is available at this link. It covers how to use Isaac Sim in general, but does not currently include instructions for the GXF Bridge Extension, which is necessary to run Isaac Sim in-the-loop with the Isaac AMR Navigation Stack. Instructions specific to GXF and Isaac AMR support are provided below.

Isaac Sim can be run manually or via the Python API as part of an automated workflow. If run manually, you will be presented with a GUI to control the simulation and run the Navigation Stack. If run via Python, Isaac Sim can run with a GUI or without a GUI (i.e. headless). For ease of use, the container includes a Python script to load a robot into a scene at a specified initial pose; this script is intended to run in parallel with the Navigation Stack, with or without the Isaac Sim GUI. More complex functionality is enabled via the Isaac Sim Python API, documented here.

Running Isaac Sim manually is recommended for first-time demonstration and/or workflow testing.

Follow these steps to connect Isaac Sim to the Navigation Stack and move the virtual robot.

1. Launch Isaac Sim:

   1. If running Isaac Sim from a local desktop, launch Isaac Sim manually from the container terminal as follows:

      Copy

      Copied!

                  
      
                  
      ./isaac-sim.sh --allow-root
              

   2. If running Isaac Sim from AWS, instead follow steps 5-7 of Section 4.14, Container Deployment in the Isaac Sim AWS Deployment documentation.

2. If a tab labeled GXF Bridge does not appear at the bottom of the window (see image below), follow these steps to enable the GXF Bridge extension:

   1. Select Window > Extension from the menu at the top of the Isaac Sim window.

      

   2. Search for “GXF Bridge” in the extensions list.

   3. Toggle the switch next to “GXF Bridge” to enable it. The switch should turn green.

      

3. If not already on a blank stage, click File > New to open a blank stage.

   

4. Open the sample scene corresponding to the Navigation Stack environment selected for the client app:

   1. Select the Isaac Assets (Beta) tab located at the bottom of the window.

   2. Click on Samples in the drop-down menu situated on the left side of the tab.

   3. Allow a moment for the Samples window to populate.

   4. Scroll down to find Isaac_AMR and double-click either full_warehouse_carter_v2_3_gxf.usd or walled_room_carter_v2_3_gxf.usd, depending on the environment you chose in the navigation stack configuration above.

   Note

   You must select the Isaac Sim scene corresponding to the omap_path and environment specified in the navigation stack configuration to ensure the navigation stack uses the occupancy map generated from the Isaac Sim scene you are using. Otherwise, navigation will not succeed.

   

5. Select the GXF Bridge tab, and confirm the Graph Path is set to the tcp_server.yaml file. You can modify the graph here to anything on your local system; by default, this file will launch a TCP server application designed to connect with the Navigation Stack.

   

6. Click “Create Application” to start the graph in Isaac Sim.

   

7. When the Navigation Stack has finished initializing, it will look for a TCP connection, displaying output like:

   Copy

   Copied!

               
   
               
   2023-07-13 23:16:57.305 INFO  external/com_nvidia_gxf/gxf/network/tcp_client_socket.cpp@53: TCP close 32
   2023-07-13 23:16:57.305 WARN  external/com_nvidia_gxf/gxf/network/tcp_client_socket.cpp@179: Failed to connect to TCP server 127.0.0.1:7000
           

   After this happens, click the Play button in Isaac Sim. The server and client will connect, and the Navigation Stack will begin.

   

8. Refresh the Isaac Sight browser page you opened earlier. You should see the robot visualized on top of a 2D occupancy map.

   Additionally, you should see a visualized circle with the subscript robot/goal. This is the marker that represents the goal position of the robot. If the goal position is not on top of the robot, you should also see a visualized path starting at the robot and ending at the goal.

   You can update the robot goal position by clicking and dragging the circle representing the goal position.

   Note

   If the simulation is paused (see below), Isaac Sight will also pause.

9. At any point, you can pause the simulation by clicking the Pause button in Isaac Sim. This will also pause the Navigation Stack . You can then click the Play button to resume simulation and navigation.

   

10. At any point, you can stop the simulation by clicking Destroy Application in the GXF Bridge tab. This will automatically stop the simulation, reset the robot to its initial position in the scene, and end the TCP server application. To restart the simulation, close the Navigation Stack app via Ctrl-C, restart the Navigation Stack app following the instructions above, then follow steps 2-8.

    

    Note

    If the simulation is stopped by clicking the Stop button, the server application will persist. After this point, the Navigation Stack will end fatally if you click Play in Isaac Sim, as the simulation clock has been reset while the Navigation Stack clock has merely paused. If you click Stop, ensure you also click Destroy Application to avoid this issue.

From the container terminal, examine the bundled run_scene.py script:

            

            
./python.sh tools/isaac_amr/run_scene.py -h
        

The run_scene.py script will then show command-line interface options. You must specify, at minimum, the path to the scene matching the environment that the navigation stack is using.

Isaac AMR sample assets can be found in /Isaac/Samples/Isaac_AMR, with the following directory structure:

• Environments: Contains custom environments developed for testing Carter v2.3 in Isaac Sim.

  • walled_room.usd: A simple enclosed space with two block obstacles

• Robots: Contains the Carter v2.3 robot

  • carter_v2_3_gxf.usd: The Carter v2.3 robot with 3D PhysX-based lidar, IMU, differential wheel base, and 2x Hawk sensor. It also includes Omnigraph-enabled sensor/control message exchange with Navigation Stack.

• Scenarios: Contains sample scenarios that are built from existing environments and include the Carter v2.3 robot.

  • full_warehouse_carter_v2_3_gxf.usd: A sample warehouse with multiple shelves, a forklift, and Carter v2.3

  • walled_room_carter_v2_3_gxf.usd: The walled room environment described above, with Carter v2.3 in the center

• Sensors: Contains the Hawk sensor.

  • hawk.usd: The Isaac Sim model of the Hawk sensor, including distortion effects

After specifying the path to the USD scene, specify the YAML server graph that ships with Isaac Sim: exts/omni.isaac.gxf_bridge/data/config/tcp_server.yaml. If running a custom scene that includes an ActionGraph that will generate the TCP server application, you do not need to specify additional YAML files on the command line.

The following is an example command to run the large warehouse scene with Carter v2.3 at 60Hz for 60,0000 frames (~1000 seconds).

            

            
./python.sh tools/isaac_amr/run_scene.py \
/Isaac/Samples/Isaac_AMR/Scenarios/full_warehouse_carter_v2_3_gxf.usd \
exts/omni.isaac.gxf_bridge/data/config/tcp_server.yaml \
-r 60 \
-n 60000
        

The following is an example command to run the large warehouse scene and place the Carter v2.3 asset at a new location/orientation.

            

            
./python.sh tools/isaac_amr/run_scene.py \
/Isaac/Environments/Simple_Warehouse/full_warehouse.usd \
exts/omni.isaac.gxf_bridge/data/config/tcp_server.yaml \
--run_indefinitely \
--robot carter_v2_3 \
--initial_pos 5.0 2.0 \
--initial_yaw 90
        

You may now open Isaac Sight as described above to visualize the Navigation Stack.

There is a known memory leak in Isaac Sim when the Quick Search and Curve Editor extensions are enabled. These extensions are automatically disabled on Isaac Sim start when running manually or via Python. Note that, if either extension is enabled, Isaac Sim memory consumption may grow by 50% over 24 hours.