Teleoperation#

Teleoperation Witth Isaac Lab#

NVIDIA Isaac Lab is a unified and modular framework for robot learning. Isaac Lab also provides standalone scripts to launch applications. Isaac for Healthcare is an extension of Isaac Lab, and therefore follows its scene creation guidelines.

Tip

If you are new to Isaac Lab, we recommend taking these courses first. They provide a strong foundation in Isaac Lab concepts, scene creation, and simulation workflows. They are highly recommended as a reference for building custom applications and understanding the underlying principles used in this chapter.

When building a new application, it’s often helpful to start from a related template. This teleoperation example is a great choice, because it allows you to build simple interactive environments, and test your robot’s ability to interact with the scene.

Code Walkthrough#

Have a look at the following code files within the I4H workflows repo:

Key Files Structure#

workflows/robotic_ultrasound/scripts/simulation/environments/teleoperation/teleop_se3_agent.py - Contains our teleoperation application, with the default task Isaac-Teleop-Torso-FrankaUsRs-IK-RL-Rel-v0
Environment Configuration - Searching for this task will lead us to the environment configuration file and class called ModFrankaUltrasoundTeleopEnv
RoboticIkRlEnvCfg Class - Defines all Markov Decision Process (MDP) related components, and contains the scene for our application
Scene Configuration - A closer look at the scene reveals that we create a basic scene with:
- Ground plane
- Lights
- Rigid table
- Phantom model
- Robot

Any of these assets can be exchanged by overriding the variable in a derived scene. This is a good place to test your own assets.

Tip

For more details on gym environments and how to register them, see Isaac Lab’s tutorials.

Teleoperation Logic#

The core teleoperation logic reads the command from the input device and converts it to an action in the robot’s end-effector space.

Code Instructions#

Launching the Teleoperation Script#

The process launches:

Ultrasound simulation application - receiving pose information from IsaacSim, publishing simulated ultrasound B-mode frames
Isaac Sim application window - with our robotic ultrasound environment, listening to keyboard input to drive the robot
Visualization application - to show the three simulated visual data-streams: room camera, wrist camera and ultrasound simulation

From your i4h-workflows folder, run the command below to launch teleoperation:

conda activate robotic_ultrasound
(
  python -m simulation.examples.ultrasound_raytracing &
  python -m simulation.environments.teleoperation.teleop_se3_agent --enable_cameras &
  python -m utils.visualization &
  wait
)

Process Termination#

Process Termination: Use Ctrl+C followed by running the script below to cleanly terminate all distributed processes.

./workflows/robotic_ultrasound/reset.sh

Teleoperation With Keyboard#

Use the controls below to control the robot arm during teleoperation.

Keyboard Controller for SE(3): Se3Keyboard#

   Reset all commands: R
   Toggle gripper (open/close): K
   Move arm along x-axis: W/S
   Move arm along y-axis: A/D
   Move arm along z-axis: Q/E
   Rotate arm along x-axis: Z/X
   Rotate arm along y-axis: T/G
   Rotate arm along z-axis: C/V

Teleoperation Code Deep Drive & Video Walkthrough#

See the video below for an example of the intended workflow.

In this video, we explore the key objects that enable teleoperation in the application. We discuss registering our environment with gym, before we detail our environment and scene configuration classes. These classes allow to build the scene programatically, and define how assets in the scene can be manipulated and articulated. Finally, we discuss the logic of the teleoperation script, which allows us to move the robotic arm in six degrees of freedom about the tool center point.