# Teleoperation

## Teleoperation Witth Isaac Lab

NVIDIA Isaac Lab is a unified and modular framework for robot learning. Isaac Lab also provides standalone scripts to launch applications. Isaac for Healthcare is an extension of Isaac Lab, and therefore follows its [scene creation](https://isaac-sim.github.io/IsaacLab/main/source/api/lab/isaaclab.scene.html) guidelines.

When building a new application, it’s often helpful to start from a related template. This teleoperation example is a great choice, because it allows you to build simple interactive environments, and test your robot’s ability to interact with the scene.

---

### Code Walkthrough

Have a look at the following code files within the I4H workflows repo:

#### Key Files Structure

- [`workflows/robotic_ultrasound/scripts/simulation/environments/teleoperation/teleop_se3_agent.py`](https://github.com/isaac-for-healthcare/i4h-workflows/blob/920bccf90fd3fe765545c01696491b73ce58d0c7/workflows/robotic_ultrasound/scripts/simulation/environments/teleoperation/teleop_se3_agent.py) - Contains our teleoperation application, with the default task `Isaac-Teleop-Torso-FrankaUsRs-IK-RL-Rel-v0`
- Environment Configuration - Searching for this task will lead us to the [environment configuration file](https://github.com/isaac-for-healthcare/i4h-workflows/blob/8c8a055fefeaf25e31b3d9dcefd28f9a546f54aa/workflows/robotic_ultrasound/scripts/simulation/exts/robotic_us_ext/robotic_us_ext/tasks/ultrasound/approach/config/teleop/ik_rel_env_cfg.py#L36) and class called `ModFrankaUltrasoundTeleopEnv`
- `RoboticIkRlEnvCfg` Class - Defines all Markov Decision Process (MDP) related components, and contains the [scene](https://github.com/isaac-for-healthcare/i4h-workflows/blob/8c8a055fefeaf25e31b3d9dcefd28f9a546f54aa/workflows/robotic_ultrasound/scripts/simulation/exts/robotic_us_ext/robotic_us_ext/tasks/ultrasound/approach/config/franka/franka_manager_rl_env_cfg.py#L398C7-L398C24) for our application
- Scene Configuration - A closer look at the [scene](https://github.com/isaac-for-healthcare/i4h-workflows/blob/8c8a055fefeaf25e31b3d9dcefd28f9a546f54aa/workflows/robotic_ultrasound/scripts/simulation/exts/robotic_us_ext/robotic_us_ext/tasks/ultrasound/approach/config/franka/franka_manager_rl_env_cfg.py#L55) reveals that we create a basic scene with:
  - Ground plane
  - Lights
  - Rigid table
  - Phantom model
  - Robot

Any of these assets can be exchanged by overriding the variable in a derived scene. This is a good place to test your own assets.

---

### Teleoperation Logic

The core [teleoperation logic](https://github.com/isaac-for-healthcare/i4h-workflows/blob/8c8a055fefeaf25e31b3d9dcefd28f9a546f54aa/workflows/robotic_ultrasound/scripts/simulation/environments/teleoperation/teleop_se3_agent.py#L367) reads the command from the input device and converts it to an action in the robot’s end-effector space.

---

### Code Instructions

#### Launching the Teleoperation Script

The process launches:

1. **Ultrasound simulation application** - receiving pose information from IsaacSim, publishing simulated ultrasound B-mode frames
2. **Isaac Sim application window** - with our robotic ultrasound environment, listening to keyboard input to drive the robot
3. **Visualization application** - to show the three simulated visual data-streams: room camera, wrist camera and ultrasound simulation

From your i4h-workflows folder, run the command below to launch teleoperation:

```bash
conda activate robotic_ultrasound
(
  python -m simulation.examples.ultrasound_raytracing &
  python -m simulation.environments.teleoperation.teleop_se3_agent --enable_cameras &
  python -m utils.visualization &
  wait
)
```

#### Process Termination

**Process Termination**: Use **Ctrl+C** followed by running the script below to cleanly terminate all distributed processes.

```bash
./workflows/robotic_ultrasound/reset.sh 
```

---

### Teleoperation With Keyboard

Use the controls below to control the robot arm during teleoperation.

#### Keyboard Controller for SE(3): Se3Keyboard

```bash
   Reset all commands: R
   Toggle gripper (open/close): K
   Move arm along x-axis: W/S
   Move arm along y-axis: A/D
   Move arm along z-axis: Q/E
   Rotate arm along x-axis: Z/X
   Rotate arm along y-axis: T/G
   Rotate arm along z-axis: C/V
```

---

### Teleoperation Code Deep Drive & Video Walkthrough

See the video below for an example of the intended workflow.

[Video](https://cdnapisec.kaltura.com/p/2935771/sp/293577100/playManifest/entryId/1_ko08u5ay/format/url/protocol/https)

In this video, we explore the key objects that enable teleoperation in the application. We discuss registering our environment with gym, before we detail our environment and scene configuration classes. These classes allow to build the scene programatically, and define how assets in the scene can be manipulated and articulated. Finally, we discuss the logic of the teleoperation script, which allows us to move the robotic arm in six degrees of freedom about the tool center point.