Using OmniGraph to Drive a Robot in Isaac Sim#

Overview#

In this module, we will explore the use of OmniGraph, a visual programming tool in Omniverse, to drive a robot in Isaac Sim. OmniGraph enables easy integration of functions from various systems and supports custom nodes for personalized functionality. We will learn how to use OmniGraph to connect Isaac Sim to ROS 2, add controllers to the robot, and build a simple Action Graph to drive the robot.

This module is divided into three sections:

  • Setting up the Environment and Robot - We will set up the environment and robot in Isaac Sim.

  • Building the Action Graph using OmniGraph Nodes - We will learn how to build a simple Action Graph using OmniGraph nodes, integrating ROS 2 and controllers to drive the robot.

  • Driving the Robot via ROS 2 - We will use the Action Graph to drive the robot via ROS 2, demonstrating the application of SIL in robotics development.

By the end of this module, you will be able to apply visual programming techniques using OmniGraph to control a robot in Isaac Sim, integrating ROS 2 for simulation-based testing.

Installing Isaac Sim and ROS 2 Humble#

In this section, we will set up the simulation environment. This involves launching Isaac Sim with ROS 2 enabled, and loading the “Simple Room” environment.

Install Isaac Sim 4.5.0#

  1. If not already installed, follow the installation guide for Isaac Sim provided in the Isaac Sim Installation Documentation.

  2. For this module, we assume you are using the Workstation Installation method.

    • Instructions can be found here.

  3. Alternatively, if you prefer to install Isaac Sim using PIP in a virtual environment, refer to the PIP Installation Guide.

Install ROS 2 Humble#

  1. Install ROS 2 Humble on Ubuntu 22.04 following the steps outlined in the ROS and ROS 2 Installation Documentation.

  2. For this module, we recommend using the Running Native ROS method. Follow the instructions under:

    • Running Native ROS > ROS 2 > Ubuntu 22.04 > Humble

Launch Isaac Sim With ROS 2 Enabled#

  1. To enable communication between Isaac Sim and ROS 2, launch Isaac Sim with the ROS 2 bridge enabled.

    • If you are using ROS 2 Humble, Isaac Sim loads the Humble bridge by default, so no additional steps are required.

    • If you are using ROS Noetic or another version of ROS, follow the instructions under Enabling the ROS Bridge Extension in the documentation.

  2. Ensure that your ROS installation is sourced before launching Isaac Sim to avoid fallback to pre-packaged libraries.

  3. Verify that the ROS2 Bridge is enabled:

    • In Isaac Sim, go to Window > Extensions.

    • Search for ROS in the search bar.

    • Confirm that the ROS2 Bridge extension is toggled on

Open the Simple Room Environment#

Note

Follow the video to guide you through the next steps.

  1. In Isaac Sim, navigate to

Content > NVIDIA/Assets/Isaac/4.2/Isaac/Environments/Simple_Room/

  1. Drag and drop the simple_room.usd file into the main window.

  1. In Isaac Sim, switch to the Isaac Sim Assets [Beta] and open Environments

  2. On simple_room.usd press with the right button and click on “Add at current selection”

  • Wait a few seconds for it to load completely.

  • Place it at the origin by zeroing out all Translate components in the Transform Property on the right-hand pane.

  • Zoom into the scene until you can clearly see the table within the room.

We have successfully set up our environment in Isaac Sim. In the next section, we will bring in the Nova Carter and prepare it for simulation.

Preparing the Nova Carter#

In this section, we will add the Nova Carter robot in the environment we prepared in Isaac Sim. This involves adding the robot and testing its physics. These steps will prepare us for further configuration and control of the robot in subsequent sections.

Add the Nova Carter Robot#

  1. Navigate to Content > NVIDIA/Assets/Isaac/4.1/Isaac/Robots/Carter

  2. Drag and drop the nova_carter.usd file into the viewport near or around the table.

  1. Open the Isaac Sim Assets [Beta] tab and select Robots

  2. Right click on nova_carter.usd and select Add at current selection.

  • Select the Nova Carter using your mouse, then adjust its position as needed while it remains selected.

  • Raise it slightly above ground level to prepare for testing physics.

Test Robot Physics#

  1. Press Play in Isaac Sim to start simulation mode.

  2. Observe that the robot falls onto the floor due to gravity, confirming that physics is enabled.

We have successfully imported the Nova Carter and verified its physics properties. In the next section, we will configure controllers to enable movement for the Nova Carter.

Building the Action Graph Using OmniGraph Nodes#

In this section, we’ll use visual scripting with OmniGraph nodes to build an Action Graph that’ll allow us to drive the robot using ROS 2.

Open Visual Scripting#

  1. Right-click on nova_carter (as seen in the video) and navigate to Create > Visual Scripting > Action Graph

  2. An Action Graph window will appear at the bottom of the Isaac Sim interface.

Familiarize Yourself With OmniGraph Nodes#

  1. Inside of the Action Graph window, locate the panel on the left-hand side.

  • This panel contains all the OmniGraph nodes.

  • ROS2-related OmniGraph nodes are listed under Isaac Ros2.

  • You can search for nodes by name using the search bar.

Add Nodes to the Graph#

  1. To add a node to the graph, drag it from the node list into the graph window (see how in the video above).

  2. Add the following nodes to your graph:

    • ROS2 Context

    • On Playback Tick

    • ROS2 Subscribe Twist

    • Scale To/From Stage Units

    • Two Break 3-Vector nodes

    • Make Array

    • Differential Controller

    • Articulation Controller

    • Two Constant Token nodes

../_images/dev-omnigraph.png

The fully Connected OmniGraph#

Troubleshoot ROS Version (if necessary)#

If all ROS-related nodes are labeled ROS1 instead of ROS2:

  1. Go to Windows > Extensions
    7. Disable the ROS bridge
    8. Enable the ROS2 bridge

Connect the Nodes#

  1. Connect the nodes as shown in the video and image above.

  2. Ensure all links are correct.

Configure the Differential Controller Node#

  1. Click on the Differential Controller node.

  2. In the right pane, set the following properties:

    • Max Linear Speed: 2.0

    • Max Angular Speed: 3.0

    • Wheel Distance: 0.413

    • Wheel Radius: 0.14

Configure the Articulation Controller Node#

  1. Click on the Articulation Controller node.

  2. In its property tab on the right pane, click on Add Target.

  3. In the window that pops up, click on nova_carter > chassis_link > Select to set the target to the Carter robot.

Configure the Constant Token Nodes#

  1. Click on the first Constant Token node.

  2. Set its Input value to joint_wheel_left.

  3. Click on the second Constant Token node.

  4. Set its Input value to joint_wheel_right.

Key Points About the Graph#

  • The ROS 2 Subscribe Twist node listens for velocity commands on the specific /cmd_vel topic.

  • The Differential Controller converts linear and angular velocities to wheel velocities.

  • The Articulation Controller applies these velocities to the robot’s wheels.

  • The Constant Token nodes specify which joints (left and right wheels) should be controlled.

With this setup, we are now ready to communicate via ROS 2 and start driving the robot around.

Driving the Robot via ROS 2#

In this section, we’ll use the popular ROS 2 package teleop_twist_keyboard to drive the robot in simulation based on the Action Graph we built earlier.

Start the Simulation#

  1. Press Play in Isaac Sim to start the graph and simulation.

Verify ROS 2 Topics#

  1. Open a new terminal and source your ROS 2 installation.

    • For example, if you have ROS 2 Humble, use the command: source /opt/ros/humble/setup.bash

  2. Run the command: ros2 topic list

  3. Verify that /cmd_vel is listed along with /rosout and /parameter_events.

Test Robot Movement#

  1. To drive the robot forward, run the following command. You should observe the Carter robot moving forward slowly in Isaac Sim. We are publishing a /cmd_vel ROS topic here that specifies the robot’s linear and angular speeds.

ros2 topic pub /cmd_vel geometry_msgs/Twist "{'linear': {'x': 0.2, 'y': 0.0, 'z': 0.0}, 'angular': {'x': 0.0, 'y': 0.0, 'z': 0.0}}"

  1. To stop the robot, run:

ros2 topic pub /cmd_vel geometry_msgs/Twist "{'linear': {'x': 0.0, 'y': 0.0, 'z': 0.0}, 'angular': {'x': 0.0, 'y': 0.0, 'z': 0.0}}"

Install Teleop Package#

  1. Install the teleop_twist_keyboard package using the following command if using ROS 2 Humble:

sudo apt-get install ros-humble-teleop-twist-keyboard

Note

If you are using a different version of ROS, you must specify your ROS version in the ROS_DISTRO variable.
sudo apt-get install ros-$ROS_DISTRO-teleop-twist-keyboard

Drive the Robot#

  1. Start the teleop_twist_keyboard node:

ros2 run teleop_twist_keyboard teleop_twist_keyboard

  1. Use the keyboard to control the robot:

i: move forward
,: move backward
j: rotate left
l: rotate right
k: stop

  1. You can increase/decrease speed by following instructions that are printed on the terminal after running the node.

Explore#

Take some time to drive the robot around the simulated environment, testing its movement and responsiveness.

Review#

In this module, we explored the fundamentals of visual scripting using OmniGraph in Isaac Sim.

In this module:

  1. Environment Setup: We learned how to set up a simulated environment and import a robot (Carter) into Isaac Sim.

  2. OmniGraph Basics: We introduced OmniGraph, a visual programming tool in Omniverse, and learned how to create an Action Graph.

  3. Node Configuration: We explored how to add and configure various nodes, including ROS 2 Subscribe, Differential Controller, and Articulation Controller.

  4. ROS 2 Integration: We learned how to integrate ROS 2 with Isaac Sim, enabling communication between the simulation and ROS 2 nodes.

  5. Robot Control: We implemented a control system for a differential drive robot using OmniGraph nodes.

  6. Practical Application: We used the teleop_twist_keyboard package to drive the simulated robot, demonstrating the practical application of our OmniGraph setup.

It’s important to note that the concepts we learned in this module are not specific to the Carter robot. With different parameters, they can be applied to any wheeled robot imported into Isaac Sim, making this knowledge widely applicable in robotics simulation.#

For further exploration:

In the next module, we’ll build upon these concepts to explore more advanced applications of Software-in-the-Loop (SIL) in robotics development.

Quiz#

  1. What is OmniGraph used for in Isaac Sim?

    1. To visually program and control simulations.

    2. To replace Python scripting entirely.

    3. To create physical hardware for robots.

    4. To generate 3D assets for simulations.

Answer

A
OmniGraph is a visual programming tool in Isaac Sim that allows users to create and control simulations by connecting nodes, making it ideal for tasks like driving a robot.

  1. Why is the Articulation Controller node necessary for driving a robot?

    1. It generates images from the robot’s camera.

    2. It sends commands to the robot’s joint drives.

    3. It creates a virtual environment for the robot.

    4. It replaces the need for physics simulation.

Answer

B
The Articulation Controller node is essential for sending commands to the robot’s joint drives, ensuring that the wheels or other actuators respond correctly to input commands.

  1. What is the purpose of the ROS 2 bridge in Isaac Sim?

    1. To replace the need for OmniGraph nodes.

    2. To simulate physical hardware components.

    3. To enable communication between Isaac Sim and ROS-based systems.

    4. To create 3D assets for the simulation.

Answer

C
The ROS 2 bridge allows Isaac Sim to communicate with ROS-based systems, enabling seamless integration for testing and controlling robots in simulation.