Sim Teleop and Whole-Body Control (WBC)#

In this lesson, we’ll collect demonstrations for the Unitree G1 static apple-to-plate task using a Meta Quest 3 or PICO 4 Ultra headset with NVIDIA Isaac Teleop.

To do that, we’ll:

Start the CloudXR runtime and Arena teleoperation session.
Connect an XR headset or emulator to the running Arena scene.
Record successful OpenXR teleoperation demonstrations to HDF5.
Replay the recorded demonstrations to verify the dataset.

Note

For current supported input devices and system requirements, see the Isaac Teleop supported input devices and Isaac Teleop system requirements.

As a best practice, update your teleop headset before beginning this lesson.

Start the CloudXR Runtime#

On the host computer, let’s configure the firewall to allow CloudXR traffic. The required ports may change depending on the client type. This example uses ufw on Ubuntu.

In a terminal outside the Isaac Lab-Arena container, configure the firewall to allow CloudXR traffic.

sudo ufw allow 49100/tcp   # Signaling
sudo ufw allow 47998/udp   # Media stream
sudo ufw allow 48322/tcp   # Proxy (HTTPS mode only)

Use an existing Arena container terminal from Setup: Isaac Lab-Arena, or start a new one if needed.
```
./docker/run_docker.sh
```
Inside the container, start the CloudXR runtime.
```
python -m isaacteleop.cloudxr
```

The first time you run this, it will prompt you to accept the NVIDIA CloudXR License Agreement.

NVIDIA CloudXR EULA must be accepted to run. View: https://github.com/NVIDIA/IsaacTeleop/blob/main/deps/cloudxr/CLOUDXR_LICENSE

Accept NVIDIA CloudXR EULA? [y/N]: Yes

Once CloudXR is running, you will see output similar to this:

Activate CloudXR environment in another terminal: source /home/ubuntu/.cloudxr/run/cloudxr.env
Keep this terminal open, **Ctrl+C** to terminate.

Start Arena Teleop#

Let’s get comfortable with the teleoperation session in Arena. For now we won’t record demonstrations; we’ll use this run for teleoperation practice.

For this step, we will need a total of two terminals attached to the Arena container.

In a second terminal, start the Arena Docker container and launch the teleoperation session.

Important

Order of operation matters here. It’s important to run the following steps, as ordered.

For example, we run source ~/.cloudxr/run/cloudxr.env, after the CloudXR runtime is already running and before you start Isaac Lab-Arena.

This is because Arena needs to inherit the Isaac Teleop CloudXR environment variables.

If you need to restart, follow these steps in order again.

Attach another terminal to the Arena Docker container.
```
./docker/run_docker.sh
```
Activate Isaac Teleop CloudXR environment variables.
```
source ~/.cloudxr/run/cloudxr.env
```

Run the teleop script. This will not record demonstrations, it is simply for teleoperation practice.

python isaaclab_arena/scripts/imitation_learning/teleop.py \
--viz kit \
--device cpu \
galileo_g1_static_pick_and_place \
--object apple_01_objaverse_robolab \
--destination clay_plates_hot3d_robolab \
--teleop_device openxr

Once Isaac Lab is running, locate the XR tab.

Arena teleoperation session with XR running. The stereoscopic viewport is on the left, and the XR settings panel is open on the right.#
Start the CloudXR session from the XR tab in the application window.
Under OpenXR Settings, you can also find the IP of your host machine listed under Which is the server’s IP address?. This is needed for the next section to connect your XR device.

Alternatively, you can use the ifconfig command to find the IP address of your wireless network interface.

Connect From the XR Headset Device#

This section assumes your headset is on the same wireless network as the host machine.

From your XR headset, open its browser app.
Navigate to the Isaac Teleop client at https://nvidia.github.io/IsaacTeleop/client.

Teleop web client interface as seen in an XR headset browser. — Teleop web client interface on the XR headset.#

In the Server IP field, enter the IP address of your Isaac Lab host machine.
Below that field, click the link labeled Click https://<ip>:48322/ to accept cert.
Accept the certificate in the new page, then navigate back to the teleop client page.
Click Connect to begin teleoperation.

Tip

For more detailed setup instructions, see Connect an XR Device. A strong wireless connection is essential for high-quality streaming. Refer to the CloudXR Network Setup guide for router configuration.

Teleoperation Control#

After you press Connect in the web browser (upper left side of UI), you should see the control panel.

Isaac Sim view with the teleop control panel.#

Tip

If the control panel is not visible, put the headset on before clicking Start XR in Isaac Lab-Arena and drag the control panel to a better location. You can also set the default location of the control panel in the teleop web client settings under Control Panel Start Position.

Client menu option for default control pane location

Press Play in the virtual control panel to start teleoperation.
Press Reset in the virtual control panel to reset the scene during practice.
Assuming you already clicked Start XR in Isaac Lab-Arena, you should now see the simulated environment from the robot’s point of view in your headset. This is an immersive experience.

The teleoperation controls are:

Left joystick: Move the body forward, backward, left, and right.
Right joystick: Squat down or rotate the torso left and right.
Controllers: Move end-effector targets for the arms.

Tip

If the simulation runs at too low FPS and teleoperation feels laggy, reduce the XR resolution from XR > Advanced Settings > Render Resolution.

When you’ve done a few practice episodes, you can exit the teleoperation session by pressing Stop XR in the virtual control panel.
In the terminal where you ran the teleop script, press Ctrl+C to terminate the teleop script.

Record With the Headset Device#

Now that we’ve had a chance to practice teleoperation, let’s actually record demonstrations.

Tip

In later modules, we provide a dataset with demonstrations pre-collected for you.

You can either:

Collect all demonstrations yourself
Try a few to understand the workflow, then move on and use our pre-collected dataset

You can re-use the same terminal we just used for teleoperation. If you are starting the recording app from a different Arena terminal, activate Isaac Teleop CloudXR environment settings again.

Confirm DATASET_DIR is still set in the Arena recording terminal.
```
echo $DATASET_DIR
```

Run the recording script. Set --num_demos to the number of successful demonstrations you want to record.

# Record demonstrations with OpenXR teleop
python isaaclab_arena/scripts/imitation_learning/record_demos.py \
  --viz kit \
  --device cpu \
  --enable_cameras \
  --dataset_file $DATASET_DIR/arena_g1_static_apple_dataset_recorded.hdf5 \
  --num_demos 20 \
  --num_success_steps 10 \
  --disable_full_sim_buffer_reset \
  galileo_g1_static_pick_and_place \
  --object apple_01_objaverse_robolab \
  --destination clay_plates_hot3d_robolab \
  --teleop_device openxr

After each successful demonstration, a UI element will appear to show how many demonstrations you have recorded so far.

Teleop as seen in an XR headset browser. — Teleop web client interface on the XR headset.#

Monitor Recording With a Second Viewport (Optional)#

For higher-quality datasets, use a two-person workflow when collecting demonstrations: one person teleoperates from the headset, while a second person watches the host monitor to confirm each trajectory stays inside the robot’s head-camera field of view.

Anything that drifts outside the recording field of view is absent from the saved HDF5 file and absent from the policy’s view at training time. Catching field-of-view problems live reduces re-recording.

The Arena application’s default viewport shows the teleoperator’s stereoscopic perspective. This is what the headset wearer sees, not what record_demos.py stores. To watch both views side by side, open a second viewport bound to the robot’s head camera.

In the running Arena application, open the Window menu and toggle on Viewport 2.

Enable a second viewport from the Window menu.#
In the new Viewport 2, click the camera selector in the viewport toolbar and choose the robot’s head-mounted camera, RobotHeadCam. In the scene hierarchy, this is under /World/envs/env_0/Robot/head_link.

This is the camera that record_demos.py writes to the HDF5 file, so any motion that leaves this frame will be absent from the dataset.

Select the robot head-mounted camera for Viewport 2.#
Keep the stereoscopic XR view and the head-camera view visible side by side during recording.

Dual-viewport layout: the stereoscopic XR view on the left is the teleoperator’s perspective, and the head-camera view on the right is what the dataset captures.#

The observer keeps every grasp and placement inside the right viewport and gives the teleoperator live feedback, such as “move a touch to your right; your hand is at the edge of frame.”

Note

RobotHeadCam is only spawned when --enable_cameras is set. The record_demos.py command in this lesson enables it by default, so the camera appears in the camera selector once you are recording.

The smoke-test teleop.py command earlier in this lesson omits --enable_cameras for performance. Pass --enable_cameras there too if you want to validate the dual-viewport layout before entering VR.

Important

The command above records only 20 demos (see argument --num_demos 20) for a fast tutorial pass.

For better inference results, collect about 400 high-quality demonstrations and keep --num_success_steps 10 so each successful episode includes extra stable frames after the success condition triggers.

Policy success rate depends heavily on dataset quality and dataset size. For better success rates, collect more clean demonstrations with smooth actions, stable grasps, and no unnecessary collisions.

Follow this protocol while collecting data:

Complete about 5 practice runs before recording the main dataset so you’re used to XR latency and the apple’s contact behavior.
Move consistently and avoid jerky motions. Jerky seed demonstrations lead to poor synthetic augmentations and unstable policy behavior.
Keep the robot torso and body fixed during this static task. Use only the arms and hands for manipulation.
Include diverse grasp styles across the dataset, including top-down grasps and side grasps, so the policy does not overfit to one approach direction.
Save only runs with no unnecessary collisions, no dropped objects before placement, and no recovery motions that would confuse the policy.
After releasing the apple onto the plate, keep the scene stable and wait until the recording auto-terminates and freezes. Reset only after that happens.
Aim for demonstrations around 200-400 timesteps. Very long episodes slow downstream data processing, while very short episodes tend to contain abrupt motion.

Read more teleoperation tips in the Teleoperation Data Collection Guide.

In the running application, start the session from the XR tab, connect the headset again, and complete the task for each demo.
After a successful placement, hold the scene stable and wait for the demo to end automatically. The simulation resets automatically after success, and the script saves successful runs to the HDF5 file path specified by --dataset_file $DATASET_DIR/arena_g1_static_apple_dataset_recorded.hdf5.

Note

There is no manual save button in this recording flow. The success condition ends and saves the episode automatically. Use Reset when you need to restart the scene during practice or after an unusable attempt.

Tip

Suggested task sequence for teleoperation:

Move the right arm to the side and keep it still, resting near the shelf or table surface if possible, to reduce visual clutter and self-occlusion.
Approach the apple smoothly with the left arm, primarily along a horizontal path. A side approach is a good default trajectory for clean demonstrations.
Once the hand is aligned with the apple, close the gripper or fingers firmly to establish a stable grasp.
Lift the apple straight upward before translating toward the plate. Avoid backtracking along the original approach path because it makes it harder for GR00T to distinguish approach and retreat motions during training.
Lower the apple until it is slightly above the plate surface, pause briefly in a stable pose, then release cleanly so the apple drops naturally onto the plate.
Keep the scene stable and wait for the automatic success termination and reset.

You can collect 20 successful demos in a short tutorial pass once the pipeline is running.

Keep the release height low and the orientation stable.

These recordings feed directly into LeRobot conversion and policy post-training, so demo quality is what the policy learns from.

Tip

Keep the release height low and the hand orientation stable so the apple lands naturally on the plate instead of rolling away.

Merge Multiple Recording Sessions#

Collecting hundreds of clean demonstrations in one sitting is impractical because of operator fatigue and the need to stop and restart Arena for breaks. The recommended workflow is to record one HDF5 file per session, then merge the sessions into the training-ready dataset.

Record each session to a separate file using the commands from above. Example flow of two recording sessions - make sure to update the --dataset_file to a unique name, and change--num_demos based on desired number of demos to record per session

# Examples - not complete
# Session 1
python isaaclab_arena/scripts/imitation_learning/record_demos.py \
  ... \
  --dataset_file $DATASET_DIR/session_a.hdf5 \
  --num_demos 50 \
  ...

# Session 2
python isaaclab_arena/scripts/imitation_learning/record_demos.py \
  ... \
  --dataset_file $DATASET_DIR/session_b.hdf5 \
  --num_demos 50 \
  ...

Merge the per-session files:

python isaaclab_arena/scripts/imitation_learning/merge_demos.py \
  -o $DATASET_DIR/arena_g1_static_apple_dataset_recorded.hdf5 \
  $DATASET_DIR/session_a.hdf5 $DATASET_DIR/session_b.hdf5

The merge script validates that all inputs share the same format_version, action shape, observation keys, and camera geometry. It renumbers successful demonstrations sequentially in the order the input files are listed.

Tip

Use --dry_run to inspect the merge report without writing the output file. This returns a non-zero exit code if any input would block the merge.

Replay Recorded Demos#

Replay the recorded HDF5 to confirm that the demos look correct.

This also works as a no-XR sanity check on the environment because it drives the environment from recorded actions without launching CloudXR.

Note

Replay runs the collected actions back through the environment. It does not reproduce the exact trajectory you saw during data collection.

Sometimes the object falls during replay even though it did not fall during the original recording. Treat that as a replay-dynamics mismatch, not automatic proof that the demonstration is bad.

Do not recollect data only because replay shows a fall; first review the original recording quality and confirm the dataset contains the intended successful action sequence.

To replay the recorded HDF5 dataset, run the following command:

# Replay from the recorded HDF5 dataset
python isaaclab_arena/scripts/imitation_learning/replay_demos.py \
  --viz kit \
  --device cpu \
  --dataset_file $DATASET_DIR/arena_g1_static_apple_dataset_recorded.hdf5 \
  galileo_g1_static_pick_and_place \
  --object apple_01_objaverse_robolab \
  --destination clay_plates_hot3d_robolab

Key Takeaways#

We collected static apple-to-plate demonstrations with OpenXR teleoperation and saved successful episodes to HDF5. Next, we’ll convert the HDF5 dataset to LeRobot format for GR00T 1.7 post-training.