Project Setup#

For this module, we’ll start with a working Isaac Lab project rather than build on the Isaac Lab Basics workspace. We’ll explore the Isaac Lab Tutorial together, one piece at a time.

Train a teacher policy using simulation state, then distill its action predictions into a student policy. The student uses two wrist-camera frames and robot state to control the SO-101 for vial pick-and-place. — A state teacher supplies action targets for a wrist-camera student on the same vial task.#

Tip

Refer back to Development Workflow for Isaac Lab for remote development, editor and container setup, and ways to preserve your changes as you complete each lesson.

Clone the Isaac Lab Tutorial#

The Isaac Lab Tutorial repository uses two branches for this course: main is the completed reference, and learning-path contains the scaffold you will complete. Stay on learning-path throughout the lessons and copy each lesson’s code into its marked locations.

Clone the repository on your GPU workstation or remote GPU host:
```
git clone https://github.com/isaac-sim/IsaacLabTutorial.git
cd IsaacLabTutorial
```
Take a look around the IsaacLabTutorial directory. You’ll recognize a project structure similar to the cartpole project, with a few more pieces to support the more challenging vial-placement task.
Switch to the branch created for this module:
```
git switch learning-path
```
Each lesson adds a few key components to this branch. Keep your changes on this branch, and run git diff main -- src/ to compare your current code with the complete reference without switching branches.
Ensure Git LFS is installed on the GPU host, then download the assets and sync the dependencies:
```
git lfs install --local
git lfs pull
uv sync --locked
```

Optional Packages for Recording and Experiment Tracking#

To record videos and track your training runs, you’ll need a few additional packages. The tutorial declares these as optional dependencies in pyproject.toml. The commands in these lessons use uv run --extra to install the packages needed for each command:

Use --extra video to save videos with record_video or --video. The tutorial’s video extra installs Isaac Lab’s MoviePy 1.x dependency.
Use --extra ovrtx with --visualizer newton_rtx to render an RTX view of your robot.
Use --extra wandb to track training runs with Weights & Biases in Managing Experiments.

You’ll record your first scenes and simple agents with --visualizer newton. We’ll also provide instructions for recording with RTX, but it is optional. See the RTX recording note before trying it.

The Project Layout#

Here is an overview of the tutorial project layout. Several functional pieces of code were removed in the learning-path branch to give you a chance to implement them, but all files and implementations are present in the main branch.

src/isaaclab_tutorial/
├── tasks/place_vial/
│   ├── config/so101/
│   │   ├── __init__.py       # Task registration
│   │   ├── env_cfg.py        # Scene and teacher task
│   │   ├── camera_env_cfg.py # Student camera observations
│   │   └── agents/           # PPO and distillation settings
│   ├── mdp/                  # Task behavior
│   └── reset/                # Reset data and curriculum
└── utils/                    # Supporting utilities

This lesson and the remaining lessons in this module build on each other as follows:

We’ll register the SO-101 vial-placement task with Gymnasium.
We’ll add the robot to an otherwise empty scene.
We’ll add a vial, rack, desk, and contact sensors to the scene.
We’ll add the actions and observations needed for simple agents.
We’ll train a teacher policy using privileged information from simulation.
We’ll train a student policy using only readily available observations.

The Robots and Tasks#

Like the cartpole template from the previous module, the tutorial project is an external Isaac Lab project: it lives in its own repository outside the Isaac Lab framework repository. Over the next several lessons, you’ll turn it into a vial-placement task, check that it runs, and then train two policies.

This section describes the robot and task you’ll be working with. You will not need to copy and paste any code until the Start With the Task ID section below.

The following snippets explain the configuration built across these lessons; you do not need to paste these overview snippets into learning-path. You’ll paste the complete registration in Start With the Task ID below. Once added, gym.register() in /so101/__init__.py tells Isaac Lab where to find the task configuration.

_PACKAGE = "isaaclab_tutorial.tasks.place_vial.config.so101"

gym.register(
    ...
    kwargs={
        ...
        "env_cfg_entry_point": f"{_PACKAGE}.env_cfg:SO101VialEnvCfg",
    },
)

The env_cfg_entry_point points to the SO101VialEnvCfg class inside the so101 package, which holds the SO-101 robot configuration for the place_vial task. This Gymnasium registration tells Isaac Lab which Python class to load when you request the task.

SO101VialEnvCfg is a ManagerBasedRLEnvCfg object. It contains the SO-101 scene, joint observations, a reset event, and simulation settings. Here is a simplified view of the configuration in env_cfg.py:

class SO101VialEnvCfg(ManagerBasedRLEnvCfg):
    scene       : SO101SceneCfg    = ...
    actions     : ActionsCfg       = ...
    observations: ObservationsCfg  = ...
    events      : EventsCfg        = ...
    rewards     : RewardsCfg       = ...
    terminations: TerminationsCfg  = ...
    sim         : SimulationCfg    = ...

These classes implement the MDP concepts from Problem Specification: ActionsCfg defines what the policy controls, ObservationsCfg defines its inputs, and RewardsCfg defines its training feedback. EventsCfg prepares resets and TerminationsCfg defines when an episode ends. The scene uses the SO-101 asset configuration supplied by Isaac Lab.

from isaaclab_assets.robots.so101 import SO101_CFG

class SO101SceneCfg(InteractiveSceneCfg):
    robot   = SO101_CFG.replace(...)
    vial    = ...
    rack    = ...
    desk    = ...
    sensors = ...

The SO101_CFG is an ArticulationCfg object that contains the robot’s articulation root, joint axes and limits, mass and inertia, collision geometry, and drives. The asset and its actuator settings must match the action interface: our SO-101 policy sends relative joint-position targets, while cartpole’s policy sends joint effort. For another robot, choose gains and limits that match its mechanics, actuator model, and physics backend. See Isaac Lab’s asset-import guide for the conversion workflow.

How the Task Configuration Fits Together#

The manager-based workflow assembles the scene and task pieces in one ManagerBasedRLEnvCfg. Each piece answers a different question:

Gymnasium registration points to ManagerBasedRLEnvCfg. The configuration contains SimulationCfg with PhysicsCfg, InteractiveSceneCfg with scene assets, and reward, observation, action, termination, and event term configurations. — Gymnasium registration names the task; `ManagerBasedRLEnvCfg` connects its simulation, scene, and task terms. These terms are added across lessons 2–5.#

Use the configuration to answer concrete questions about the vial task:

Scene: Which robot, vial, rack, and sensors exist?
Actions and observations: How does a policy output move a joint, and what measurements come back to the policy?
Rewards and terminations: What counts as progress, success, failure, or a timeout?
Events: Where do the robot and vial start on each attempt?
Simulation: Which physics backend advances the scene, and how often?

You will add these pieces as each lesson needs them. The environment’s physics time step multiplied by decimation gives the interval between policy actions. The separate agent configuration controls how PPO or distillation trains a policy from the resulting experience.

Start With the Task ID#

The project’s pyproject.toml exposes isaaclab_tutorial.tasks through the isaaclab.tasks entry-point group:

# pyproject.toml
[project.entry-points."isaaclab.tasks"]
so101-vial-place = "isaaclab_tutorial.tasks"

Importing the package loads the robot-specific configuration, where gym.register() connects a task ID to Isaac Lab’s ManagerBasedRLEnv and our SO101VialEnvCfg:

Replace the contents

File: src/isaaclab_tutorial/tasks/place_vial/config/so101/__init__.py
Markers: # BEGIN lesson-01-registration and # END lesson-01-registration

Paste the code below between the markers. Keep both marker comments.

gym.register(
    id="IsaacTutorial-Place-Vial-SO101",
    entry_point="isaaclab.envs:ManagerBasedRLEnv",
    disable_env_checker=True,
    kwargs={
        "env_cfg_entry_point": f"{_PACKAGE}.env_cfg:SO101VialEnvCfg",
        "rsl_rl_cfg_entry_point": f"{_PACKAGE}.agents.rsl_rl_ppo_cfg:SO101StatePPORunnerCfg",
        "default_agent": "rsl_rl",
    },
)

The id is the name you’ll pass to the command-line agents once the configuration has the actions and episode rules needed to run env.step(). The SO101VialEnvCfg configuration describes the task, including the robot. It gathers the scene, observations, events, and simulation timing.

The supplied inspection_env_cfg.py connects SO101SceneCfg to a ManagerBasedRLEnvCfg with empty actions, joint observations, and a reset event. Use its IsaacTutorial-Inspect-SO101 task after adding the robot in the next lesson. The placement task registration above will be used once you add controls in Running Simple Agents.

Prepare Joint Observations#

The mdp package brings together helper functions from three places:

Isaac Lab provides general helpers such as joint_pos and joint_vel.
Isaac Lab’s lift-task utilities provide helpers such as body_state_b and abnormal_robot_state.
This tutorial adds vial-placement behavior in tasks/place_vial/mdp/.

An observation term tells the environment how to collect a piece of information for the policy. Let’s follow two terms that read the robot’s joint positions and velocities. This configuration is already supplied in so101/lesson_scaffold.py, so you can read along without copying it into your project. The file supplies the imports, including ObsGroup and ObsTerm, which are short names for Isaac Lab’s observation group and observation term configuration classes.

@configclass
class JointObservationsCfg(ObsGroup):
    joint_pos = ObsTerm(
        func=mdp.joint_pos,
        params={"asset_cfg": SceneEntityCfg("robot")},
    )
    joint_vel = ObsTerm(
        func=mdp.joint_vel,
        params={"asset_cfg": SceneEntityCfg("robot")},
    )

    def __post_init__(self):
        self.enable_corruption = False
        self.concatenate_terms = True

Start with joint_pos. Its func field selects the helper that will read the positions. Notice that mdp.joint_pos has no parentheses: the configuration passes the function itself to the manager, which will call it later. The params dictionary supplies a keyword argument named asset_cfg. SceneEntityCfg("robot") selects the scene asset named robot; because this example does not specify joint names, it uses all of that robot’s joints.

Now look inside the helper. This excerpt from Isaac Lab’s source file mdp/observations.py shows its essential logic:

def joint_pos(env, asset_cfg):
    asset = env.scene[asset_cfg.name]
    return asset.data.joint_pos.torch[:, asset_cfg.joint_ids]

The manager supplies env, the running environment, and passes in asset_cfg from the configuration. The first line looks up the robot in env.scene. The second reads its joint positions as a PyTorch tensor. A tensor is a multidimensional array: six joint positions from each of 16 robots form a tensor with shape (16, 6). Each row belongs to one environment, and each column holds one joint position.

In [:, asset_cfg.joint_ids], the : selects every environment, while joint_ids selects the joints resolved by the manager. The same operation therefore reads all 16 robots, even when their positions differ. If you later select particular joint names in SceneEntityCfg, the manager resolves those names to indices before the helper uses them.

The joint_vel term follows the same pattern but reads asset.data.joint_vel instead. For the SO-101’s six joints, positions are in radians and velocities are in radians per second. With concatenate_terms=True, the observation manager joins the two outputs in term order: six positions followed by six velocities. For 16 robots, the resulting observation has shape (16, 12). enable_corruption=False disables observation noise for this group, so you can start by inspecting the simulated measurements directly.

These two helpers read current values without keeping their own history. Later, you’ll meet stateful terms that remember information between steps, such as earlier camera frames. For those terms, the manager creates a class instance, calls its __call__ method during operation, and calls reset(env_ids) to clear episode-specific memory when environments restart.

You can use these existing joint helpers through the tutorial’s mdp package, which re-exports them from Isaac Lab. Task-specific observations, such as placement features and wrist images, live in the tutorial’s mdp/observations.py.

The initial checkout will fail configuration validation because the robot has not been added yet. This is expected—you’ll fill in the missing pieces over the next lessons:

SO-101 Robot: Add the robot, then use the supplied inspection task to run the robot scene with empty actions.
Vial and Rack: Add the vial and rack.
Running Simple Agents: Add policy actions and episode rules.
Later lessons: Replace the same environment configuration with the teacher task and add the camera student.

The project’s tests check the completed task, so wait until you finish the distillation lesson to run the full test suite.

Review#

Check Your Understanding#

What does task registration connect a task ID to?

Apply What You Learned#

Starting from learning-path, what pieces must be added before a simple agent can step the SO-101 task? Distinguish the existing scaffold from the registration, scene, actions, observations, resets, and ending conditions needed for a runnable task.

Key Takeaways#

Stay on learning-path and compare your progress with the completed reference on main.
Paste the task registration into its marked region during this lesson.
Scene and MDP configuration connect the robot to observations, actions, rewards, and resets.
The inspection scene runs after adding the robot; simple agents need the later controls and episode rules.