Run Locomotion Policy With Teleoperation#

In this module we’ll launch both a full body controller for the humanoid robot, and a teleoperation script to run alongside our simulation.

In total, we will have three terminal windows open:

  • One running Isaac Sim

  • One for the full body robot controller through ROS 2

  • One for teleoperation through ROS 2

Set up the Controller#

Now that the asset is set up, let’s run our policy through ROS 2.

  1. Open a new terminal.

  2. Source the system ROS and the ROS 2 workspace using these commands:

source /opt/ros/humble/setup.sh
source ~/IsaacSim-ros_workspaces/humble_ws/install/setup.bash

  1. Launch the h1_fullbody_controller ROS 2 package by running the following command.

ros2 launch h1_fullbody_controller h1_fullbody_controller.launch.py

  1. Wait until the full body controller is initialized before continuing. This line will be printed in the terminal when ready to move on: Initializing H1FullbodyController

Note

Make sure to start the full body controller before starting the simulation, otherwise the robot will fall over.

Set up the Teleoperation#

Checkpoint

If you had any troubles with these steps, load the completed environment checkpoint file:
h1_ros_locomotion_warehouse/h1_ros_locomotion_policy_tutorial.usd

  1. In Isaac Sim, open the USD file with the robot and scenario you created earlier.

  2. Press the Play button to start the simulation.

  3. Open another terminal window.

  4. Run the commands below to source ROS, and use teleop_twist_keyboard to publish Twist messages using the commands below.

source /opt/ros/humble/setup.sh
ros2 run teleop_twist_keyboard teleop_twist_keyboard

Control the robot#

You can now control the H1 humanoid robot using your keyboard!

This ROS2 package computes observations and actions using the ROS messages that we are publishing above and the flat terrain locomotion policy. When no command velocities are received, the robot will stand still and maintain balance.

Try the controls and see if the robot moves as expected. Make sure you’re clicked into the teleoperation terminal before pressing these commands.

  • Forward: i

  • Forward + Turn Left: u

  • Forward + Turn Right: o

  • Turn Left: j

  • Turn Right: l

  • Stand Still: k

H1 robot teleoperation controls

Note: this video is sped up for demonstration purposes

Important

If your robot falls over or you stop the simulation, follow these steps to reset. In short, Stop the Isaac Sim install and re-run the full body controller.

  1. Press Stop on the Isaac Sim simulation.

  2. Restart the Full Body Controller terminal. In the terminal running that command, press Ctrl+C to stop the command, up arrow to see the most recent command, and hit enter to run it again.

Troubleshooting Tips#

  1. If the robot leans forward and backward, then falls face-first, the policy (full body controller) likely isn’t running. Make sure the full body controller is running before starting the Isaac Sim simulation.

  2. If the robot explodes or behaves wildly, make sure to stop Isaac Sim, then reset the full body controller, before starting simulation.

  3. Reset all terminals as shown above before playing the Simulation.

  4. Double check the OmniGraphs - particularly that On Playback Tick nodes were replaced with On Physics Step nodes.

Bonus#

Now that the teleoperation is working, you might want to observe the robot from another point of view.

If you’d like a challenge, consider adding another camera parented to the robot, to change the point of view.

At a high-level:

  1. Add a new camera.

  2. Parent it to a rigid body from the robot.

  3. Change the viewport to this camera.

Review#

Congratulations, you’ve completed the course! We hope this has given you a deeper understanding how leveraging SIL techniques to validate and test policies in simulation before deploying to hardware.