Sim-to-Real Strategy 1: Domain Randomization#

Now that you’ve done teleoperation on a real robot, let’s try it in simulation with Isaac Lab.

In this module, you’ll use the teleop arm to drive a simulated SO-101 robot, allowing us to collect demonstrations with Isaac Lab.

Because it’s simulation, we have control of the world and can manipulate it in interesting ways, like using domain randomization to ensure our dataset will be sufficiently varied.

Learning Objectives#

By the end of this session, you’ll be able to:

Explain domain randomization and why it improves sim-to-real transfer
Collect demonstration data through teleoperation, in simulation
Apply domain randomization to augment demonstrations

What Is Domain Randomization?#

Domain randomization (DR) is a sim-to-real strategy based on this idea: instead of making simulation perfectly match reality, randomize simulation parameters during training so the policy becomes robust to any value in the range, including real-world values.

Put in simple terms: think about how you might learn to catch a ball.

If you always catch it in the same pose, you might not learn to reach and catch the ball, or hold the glove in different orientations. By varying where the ball is thrown to you when you practice, you will likely learn a better “policy” for catching the ball.

Teleoperation: Collecting Human Demonstrations#

In this lesson we’ll apply domain randomization during teleoperation. We will use these to perform a kind of robot learning known as imitation learning.

Hands-On: Collecting Demonstrations#

Here is a video of the task:

Teleoperation example in the LeRobot Dataset Visualizer — Example: Teleoperation of SO-101, being replayed through the LeRobot Dataset Visualizer.#

On top are the observations from cameras, and below are the positions of robot joints.

See this dataset on Hugging Face, using the Dataset Visualizer

Tip

Having trouble with cameras or robot connection? See the Troubleshooting Guide.

Launch Simulation Environment (Docker)#

If you still have the teleop-docker container’s terminal open from the last module, you can skip this step. If not, expand the dropdown and run the command.

Practice Teleoperation in Simulation#

Let’s launch the simulation environment to practice teleoperation without recording.

This is a good way to get familiar with the teleop controls and camera views before collecting data.

(Optional) Run this quick sanity check to make sure your environment variables are set correctly.

echo "Teleop port is ${TELEOP_PORT} with id ${TELEOP_ID}"

If they aren’t set, find the ports using lerobot-find-port and assign them again:

Move the teleop arm to a packed position. If the robot is in a strange starting position, it may run into items in simulation on startup.
Run the following command to open Isaac Lab, with our pre-configured simulation environment. You can choose between two options: Lerobot-So101-Teleop-Vials-To-Rack which has no domain randomization or Lerobot-So101-Teleop-Vials-To-Rack-DR, which has domain randomization enabled.

lerobot_agent --task Lerobot-So101-Teleop-Vials-To-Rack-DR

This will launch Isaac Sim and load the training environment.

Note

The first time this launches, it will take about 2 minutes to load.

If it gets stuck, check the console for errors. It’s likely the robot isn’t fully connected. Power cycle the robot (plug/replug power on the back) if you have issues.

Keep Isaac Lab open for the next step.

Setup Cameras#

We need our simulation to show us the same camera views our AI model will use.

When doing teleoperation for training VLAs, it’s crucial that we use the same camera views for teleoperation that the model will use for autonomous operation.

Otherwise, we may introduce biases or advantages the model won’t have.

Important

Only look through the gripper and external cameras when teleoperating.

When looking at the scene with your own eyes, or other cameras in the simulation scene, you may introduce perceptual affordances that the model will not have access to during inference.

The policy will only see what the cameras see. Train yourself to rely solely on the camera views displayed on your screen. This ensures your demonstrations reflect what the policy can actually perceive.

By default you’ll just see the general perspective camera. Let’s fix that.

Go to Window > Viewports, and enable both viewport Viewport 1 and Viewport 2 so we can see two cameras rendered at once.

In one viewport, go to the camera menu, and choose the gripper_cam.

In the other viewport, go to the camera menu, and choose the Camera_OmniVision_9782_Color camera.

For each viewport, set the aspect ratio to 4:3 to match the cameras.

Go to the settings menu in the viewport.
Under Viewport > Aspect Ratio on the right side you’ll see 16:9. Change it to 4:3.
Now try teleoperating, and take some time to get familiar with the teleop controls and camera views before collecting data in episodic format.
Press R to reset the environment with domain randomization. If it doesn’t work, click on the viewport to give the application focus, and try again.
Notice in the terminal, you will see status updates about the subtask success, such as when the vials are grasped or placed in the rack.

Controls (click in Viewport to use these commands)

Press R to reset the environment (also stops recording)
Episodes are queued for processing while you continue working

When finished, stop Isaac Lab by pressing CTRL+C in the terminal.

Start Recording Demonstrations#

When ready to collect data, we’ll add a few extra arguments for where to save the data we collect.

Before launching the teleop agent, set your Hugging Face username as an environment variable. This is used to organize your datasets in a unique namespace.

If you don’t have one, or don’t want to login, you can make up a username for local data collection.

Run this, replacing your-hf-username with your actual Hugging Face username:

export HF_USER=your-hf-username

You only need to do this once per terminal session before running the following commands. Feel free to use a made up username if you don’t want to login and upload your demos.

Overall Flow#

For each episode we will:

Reset the environment: Press R to randomize vial positions, rack position, camera poses, and lighting.
Record: Press S to start recording
Execute: Immediately begin the demonstration.
Complete: Press S to stop recording

Demonstration Quality Guidelines:

Good demonstrations:

Smooth, deliberate motions
Clear grasp contact with vial
Successful placement in rack
Only looking at camera views while operating

Avoid:

Jerky, hesitant motions
Missed grasps or drops
Including more than the actual task execution

Launch recording session. This will be just like the environment before, but we have additional controls to cancel, start recording, and stop recording.

lerobot_agent --task Lerobot-So101-Teleop-Vials-To-Rack-DR \
    --repo_id ${HF_USER}/so101_teleop_vials \
    --repo_root $(pwd)/datasets/so101_teleop_vials \
    --task_name "Pick up the vial and place it in the rack"

Set up the window, viewports, and cameras (same as in Practice Teleoperation):
- Window > Viewport: Enable both viewports so you see two camera views at once.
- In one viewport, open the camera menu and choose gripper_cam.
- In the other viewport, open the camera menu and choose Camera_OmniVision_9782_Color.
- For each viewport: open the viewport settings, go to Viewport > Aspect Ratio, and set to 4:3 (instead of 16:9).
Recording Controls: Isaac Sim viewport must be in “focus” (click the app’s UI)

Press S to start/stop recording an episode
Press C to cancel the current recording (useful for mistakes)
Press R to reset the environment (also stops recording)
Episodes are queued for processing while you continue working

Example terminal output:

[INFO]: Started recording.
[INFO]: Stopped recording.
[INFO]: Copy episode to CPU...
[INFO]: Episode added to queue.
[INFO]: [ASYNC] received episode from queue...
[INFO]: Cleared buffers

When finished, make sure you see the message [INFO]: No More episodes in queue. Wait a few seconds if you don’t see it. This means all the episodes have been processed and saved.
Stop Isaac Lab by pressing CTRL+C in the terminal.

Step 4: Review Collected Data#

Optional: if you recorded a demonstration, use the LeRobot dataset visualizer to review your recorded episodes:

lerobot-dataset-viz \
    --repo-id ${HF_USER}/so101_teleop_vials \
    --root $(pwd)/datasets/so101_teleop_vials \
    --episode-index 0

Change --episode-index to view different episodes.

Domain Randomization in Simulation#

To maximize domain randomization benefits, collect demonstrations across multiple sessions. The environment randomizes conditions between episodes automatically.

Let’s take a look at the code.

Code Tour: Domain Randomization Implementation#

The Isaac Lab environment implements DR through reset event handlers. Here’s a tour of the key randomization methods from the teleop environment codebase.

In the workshop repo, these randomizations are applied in DR task variants (for example, Lerobot-So101-Teleop-Vials-To-Rack-DR). The base Lerobot-So101-Teleop-Vials-To-Rack task keeps the sky light off and uses a fixed orange robot color.

Lighting Randomization (randomize_sky_light)

File: sim_to_real_so101/source/sim_to_real_so101/mdp/resets.py

Randomizes the environment’s dome light on each reset—exposure, color temperature, and HDRI texture:

def randomize_sky_light(
    env,
    env_ids: torch.Tensor | None,
    exposure_range: tuple[float, float],
    temperature_range: tuple[float, float],
    textures_root: str,
    asset_cfg: SceneEntityCfg = None,
):
    # Sample random exposure and color temperature
    exposure = math_utils.sample_uniform(*exposure_range, (1,), device="cpu").item()
    temperature = math_utils.sample_uniform(*temperature_range, (1,), device="cpu").item()

    # Select random HDRI texture from available options
    textures = glob.glob(os.path.join(textures_root, "*.exr"))
    texture = textures[torch.randint(0, len(textures), (1,)).item()]

    # Apply to the dome light
    prim.GetAttribute("inputs:exposure").Set(exposure)
    prim.GetAttribute("inputs:colorTemperature").Set(temperature)
    prim.GetAttribute("inputs:texture:file").Set(Sdf.AssetPath(texture))

Camera Pose Randomization (randomize_camera_pose)

File: sim_to_real_so101/source/sim_to_real_so101/mdp/resets.py

Adds small position and rotation offsets to the external camera:

def randomize_camera_pose(
    env,
    env_ids: torch.Tensor | None,
    prim_path_pattern: str,
    pos_range: dict[str, tuple[float, float]] = None,  # e.g., {"x": (-0.02, 0.02)}
    rot_range: dict[str, tuple[float, float]] = None,  # e.g., {"pitch": (-0.05, 0.05)}
):
    # Sample random offsets relative to USD default pose
    x = base_pos[0] + math_utils.sample_uniform(*pos_range.get("x", (0, 0)), (1,)).item()
    y = base_pos[1] + math_utils.sample_uniform(*pos_range.get("y", (0, 0)), (1,)).item()
    z = base_pos[2] + math_utils.sample_uniform(*pos_range.get("z", (0, 0)), (1,)).item()
    
    # Combine base quaternion with random delta rotation
    delta_quat = math_utils.quat_from_euler_xyz(roll, pitch, yaw)
    final_quat = math_utils.quat_mul(base_quat_tensor, delta_quat)

Object Pose Randomization (reset_vials_rack)

File: sim_to_real_so101/source/sim_to_real_so101/mdp/resets.py

Randomizes vial and rack positions, with probability of pre-placing vials in slots:

def reset_vials_rack(
    env,
    env_ids: torch.Tensor,
    vials: list[str],
    rack: str,
    rack_pose_range: dict[str, tuple[float, float]],
    pose_range: dict[str, tuple[float, float]],
    rack_placement_prob: float = 0.33,
):
    # Randomize rack position and orientation
    new_rack_positions, new_rack_orientations = random_asset_pose(
        env, env_ids, rack, rack_pose_range, {}
    )
    
    # With some probability, pre-place a vial in a random slot
    if torch.rand(1).item() < rack_placement_prob:
        vial_idx = torch.randint(0, len(vial_objects), (1,)).item()
        slot_idx = torch.randint(0, total_slots, (1,)).item()
        # Transform slot position from rack local frame to world frame
        slot_position, slot_orientation = math_utils.combine_frame_transforms(
            new_rack_positions, new_rack_orientations, 
            slot_position_local, slot_orientation_local
        )
        vial.write_root_pose_to_sim(slot_pose, env_ids=env_ids)

Wiring It Up: Event Configuration

File: sim_to_real_so101/source/sim_to_real_so101/tasks/task_env_cfg.py

These randomization functions are registered as reset events in the environment config:

@configclass
class TaskEventCfg(EventCfg):
    
    reset_sky_light = EventTerm(
        func=randomize_sky_light,
        mode="reset",
        params={
            "exposure_range": (-4.0, 3.0),
            "temperature_range": (2500.0, 9500.0),
            "textures_root": f"{assets_path}/hdri",
            "asset_cfg": SceneEntityCfg("sky_light"),
        },
    )

    reset_camera_external_pose = EventTerm(
        func=randomize_camera_pose,
        mode="reset",
        params={
            "prim_path_pattern": "{ENV_REGEX_NS}/LightStudio/LightBox/camera_mount",
            "pos_range": {"x": (-0.02, 0.02), "y": (-0.02, 0.02), "z": (-0.01, 0.01)},
            "rot_range": {"roll": (-0.05, 0.05), "pitch": (-0.05, 0.05), "yaw": (-0.05, 0.05)},
        },
    )

Every time an episode resets, Isaac Lab calls each registered EventTerm with mode="reset", applying fresh randomization.

For this workshop migration, the mat yaw randomization range is tightened to (-0.1, 0.1) in DR task configs.

Tip

You can experiment with domain randomization by changing the ranges or which resets run. In task_env_cfg.py, the TaskEventCfg class registers each randomization as an EventTerm with a params dict. For example, adjust exposure_range or temperature_range in reset_sky_light, or pos_range / rot_range in reset_camera_external_pose, to widen or narrow variation. Commenting out an EventTerm disables that randomization.

Note where you’re editing - if inside the container, changes might be lost on restart.

Subtask Rating#

Notice in the terminal output, that our simulation can detect when the vial is grasped, and when it is placed in the rack.

[GRASP] Vial grasped in env(s): [0]
[RELEASE] Vial released in env(s): [0]
[RACK] vial_2 placed in rack in env(s): [0]

This strategy is useful when we start policy inference, because we can automatically score how well the policy is performing.

Sim vs. Real Teleoperation Comparison#

Aspect	Simulation	Real Robot
Domain randomization	Automatic	Manual, limited to what you can physically change in the environment
Data collection speed	Faster reset, parallel envs possible	Real-time only
Hardware wear	None	Accumulates over time
Visual diversity	Procedural generation	Requires manual variation
Physics accuracy	Approximated	Ground truth

When to Use Each#

Use simulation when:

Building initial dataset with DR
Hardware is limited or shared
Exploring task or policy variations quickly and safely
Real environment isn’t ready, accessible, or during development

Use real robot when:

Collecting high-quality ground truth
Validating sim-trained policies
Capturing real-world nuances (friction, lighting)

Key Takeaways#

Domain randomization makes policies robust by training on varied conditions
Teleoperation captures human expertise in demonstration form
Always teleoperate using only camera views—not your eyes
DR augmentation multiplies your dataset with varied conditions
Combined real demonstrations + DR augmentation is a powerful baseline

What’s Next?#

With augmented demonstrations collected, learn how policies are trained and served. In the next session, Isaac GR00T: Vision-Language-Action Models, you’ll study VLAs and the GR00T architecture before running evaluations.