# Project Setup[#](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/latest/train-a-robot-arm/01-project-setup.html.md#project-setup 'Link to this heading')

For this module, we’ll start with a working Isaac Lab project rather than build on the [Isaac Lab Basics](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/latest/isaac-lab-basics/02-isaac-lab-projects.html.md) workspace. We’ll explore the [Isaac Lab Tutorial](https://isaac-sim.github.io/IsaacLab/release/3.0.0/source/setup/tutorial.html) together, one piece at a time.

[Train a teacher policy using simulation state, then distill its action predictions into a student policy. The student uses two wrist-camera frames and robot state to control the SO-101 for vial pick-and-place.](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/latest/_images/teacher-to-camera-student.svg)

A state teacher supplies action targets for a wrist-camera student on the same vial task.[#](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/latest/train-a-robot-arm/01-project-setup.html.md#id1 'Link to this image')

Tip

Refer back to [Development Workflow for Isaac Lab](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/latest/isaac-lab-basics/03-development-workflow.html.md) for remote development, editor and container setup, and ways to preserve your changes as you complete each lesson.

## Clone the Isaac Lab Tutorial[#](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/latest/train-a-robot-arm/01-project-setup.html.md#clone-the-isaac-lab-tutorial 'Link to this heading')

The [Isaac Lab Tutorial repository](https://github.com/isaac-sim/IsaacLabTutorial) uses two branches for this course: `main` is the completed reference, and `learning-path` contains the scaffold you will complete. Stay on `learning-path` throughout the lessons and copy each lesson’s code into its marked locations.

1. Clone the repository on your GPU workstation or remote GPU host:

    ```
    git clone --branch main https://github.com/isaac-sim/IsaacLabTutorial.git
    cd IsaacLabTutorial
    ```

 Take a look around the `IsaacLabTutorial` directory. You’ll recognize a project structure similar to the cartpole project, with a few more pieces to support the more challenging vial-placement task.

2. Switch to the branch created for this module:

    ```
    git switch learning-path
    ```

 Each lesson adds a few key components to this branch. Keep your changes on this branch, and run `git diff main -- src/` to compare your current code with the complete reference without switching branches.

3. Ensure [Git LFS](https://git-lfs.com/) is installed on the GPU host, then download the assets and sync the dependencies:

    ```
    git lfs install --local
    git lfs pull
    uv sync --locked
    ```

### Optional Packages for Recording and Experiment Tracking[#](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/latest/train-a-robot-arm/01-project-setup.html.md#optional-packages-for-recording-and-experiment-tracking 'Link to this heading')

To record videos and track your training runs, you’ll need a few additional packages. The commands in these lessons use [`uv run --with`](https://docs.astral.sh/uv/concepts/projects/run/#requesting-additional-dependencies) to make additional packages available when you run the command:

- Use `--with 'moviepy<2'` to save videos with `record_video` or `--video`. These recording commands need MoviePy 1.x.

- Use `--extra ovrtx` with `--visualizer newton_rtx` to render an RTX view of your robot.

- Use `--with wandb==0.26.1` to track training runs with Weights & Biases in [Managing Experiments](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/latest/train-a-robot-arm/07-managing-experiments.html.md).

You’ll record your first scenes and simple agents with `--visualizer newton`. We’ll also provide instructions for recording with RTX, but it is optional. See the [RTX recording note](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/latest/train-a-robot-arm/05-teacher-policy.html.md#if-rtx-recording-fails) before trying it.

## The Project Layout[#](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/latest/train-a-robot-arm/01-project-setup.html.md#the-project-layout 'Link to this heading')

Here is an overview of the tutorial project layout. Several functional pieces of code were removed in the `learning-path` branch to give you a chance to implement them, but all files and implementations are present in the `main` branch.

```
src/isaaclab_tutorial/
├── tasks/place_vial/
│   ├── config/so101/
│   │   ├── __init__.py       # Task registration
│   │   ├── env_cfg.py        # Scene and teacher task
│   │   ├── camera_env_cfg.py # Student camera observations
│   │   └── agents/           # PPO and distillation settings
│   ├── mdp/                  # Task behavior
│   └── reset/                # Reset data and curriculum
└── utils/                    # Supporting utilities
```

This lesson and the remaining lessons in this module build on each other as follows:

1. We’ll register the SO-101 vial-placement task with Gymnasium.

2. We’ll add the robot to an otherwise empty scene.

3. We’ll add a vial, rack, desk, and contact sensors to the scene.

4. We’ll add the actions and observations needed for simple agents.

5. We’ll train a teacher policy using privileged information from simulation.

6. We’ll train a student policy using only readily available observations.

Full diff of the `learning-path` and `main` branches

The diff below was generated using [diff2html](https://diff2html.xyz/). You can skim through it to see the changes made across the lessons, or you can run `git diff main -- src/` in your local clone of the repository. Here is a link to open the diff in a separate browser tab: [learning-path and main branch diff](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/latest/_downloads/44736451fd77f31f565d6d570ea8964b/robot-arm-learning-path-to-main.html).

## The Robots and Tasks[#](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/latest/train-a-robot-arm/01-project-setup.html.md#the-robots-and-tasks 'Link to this heading')

Like the cartpole template from the previous module, the tutorial project is an external Isaac Lab project: it lives in its own repository outside the Isaac Lab framework repository. Over the next several lessons, you’ll turn it into a vial-placement task, check that it runs, and then train two policies.

This section describes the robot and task you’ll be working with. You will not need to copy and paste any code until the [Start With the Task ID](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/latest/train-a-robot-arm/01-project-setup.html.md#start-with-the-task-id) section below.

The following snippets explain the configuration built across these lessons; you do **not** need to paste these overview snippets into `learning-path`. You’ll paste the complete registration in **Start With the Task ID** below. Once added, `gym.register()` in `/so101/__init__.py` tells Isaac Lab where to find the task configuration.

```
1_PACKAGE = "isaaclab_tutorial.tasks.place_vial.config.so101"
2
3gym.register(
4    ...
5    kwargs={
6        ...
7        "env_cfg_entry_point": f"{_PACKAGE}.env_cfg:SO101VialEnvCfg",
8    },
9)
```

The `env_cfg_entry_point` points to the `SO101VialEnvCfg` class inside the `so101` package, which holds the SO-101 robot configuration for the `place_vial` task. This Gymnasium registration tells Isaac Lab which Python class to load when you request the task.

`SO101VialEnvCfg` is a [`ManagerBasedRLEnvCfg`](https://isaac-sim.github.io/IsaacLab/release/3.0.0/source/api/lab/isaaclab.envs.html#isaaclab.envs.ManagerBasedRLEnvCfg) object. It contains the SO-101 scene, joint observations, a reset event, and simulation settings. Here is a simplified view of the configuration in `env_cfg.py`:

```
1class SO101VialEnvCfg(ManagerBasedRLEnvCfg):
2    scene       : SO101SceneCfg    = ...
3    actions     : ActionsCfg       = ...
4    observations: ObservationsCfg  = ...
5    events      : EventsCfg        = ...
6    rewards     : RewardsCfg       = ...
7    terminations: TerminationsCfg  = ...
8    sim         : SimulationCfg    = ...
```

These classes implement the [MDP concepts from Problem Specification](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/latest/robot-learning-fundamentals/02-problem-specification.html.md): `ActionsCfg` defines what the policy controls, `ObservationsCfg` defines its inputs, and `RewardsCfg` defines its training feedback. `EventsCfg` prepares resets and `TerminationsCfg` defines when an episode ends. The scene uses the SO-101 asset configuration supplied by Isaac Lab.

```
1from isaaclab_assets.robots.so101 import SO101_CFG
2
3class SO101SceneCfg(InteractiveSceneCfg):
4    robot   = SO101_CFG.replace(...)
5    vial    = ...
6    rack    = ...
7    desk    = ...
8    sensors = ...
```

The `SO101_CFG` is an `ArticulationCfg` object that contains the robot’s articulation root, joint axes and limits, mass and inertia, collision geometry, and drives. The asset and its actuator settings must match the action interface: our SO-101 policy sends **relative joint-position targets**, while cartpole’s policy sends **joint effort**. For another robot, choose gains and limits that match its mechanics, actuator model, and physics backend. See Isaac Lab’s [asset-import guide](https://isaac-sim.github.io/IsaacLab/release/3.0.0/source/how-to/import_new_asset.html) for the conversion workflow.

## How the Task Configuration Fits Together[#](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/latest/train-a-robot-arm/01-project-setup.html.md#how-the-task-configuration-fits-together 'Link to this heading')

The [manager-based workflow](https://isaac-sim.github.io/IsaacLab/release/3.0.0/source/how-to/create_manager_rl_env.html) assembles the scene and task pieces in one `ManagerBasedRLEnvCfg`. Each piece answers a different question:

[Gymnasium registration points to ManagerBasedRLEnvCfg. The configuration contains SimulationCfg with PhysicsCfg, InteractiveSceneCfg with scene assets, and reward, observation, action, termination, and event term configurations.](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/latest/_images/isaac-lab-task-registration-components.svg)

Gymnasium registration names the task; `ManagerBasedRLEnvCfg` connects its simulation, scene, and task terms. These terms are added across lessons 2–5.[#](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/latest/train-a-robot-arm/01-project-setup.html.md#id2 'Link to this image')

Use the configuration to answer concrete questions about the vial task:

- **Scene:** Which robot, vial, rack, and sensors exist?

- **Actions and observations:** How does a policy output move a joint, and what measurements come back to the policy?

- **Rewards and terminations:** What counts as progress, success, failure, or a timeout?

- **Events:** Where do the robot and vial start on each attempt?

- **Simulation:** Which physics backend advances the scene, and how often?

You will add these pieces as each lesson needs them. The environment’s physics time step multiplied by `decimation` gives the interval between policy actions. The separate agent configuration controls how PPO or distillation trains a policy from the resulting experience.

## Start With the Task ID[#](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/latest/train-a-robot-arm/01-project-setup.html.md#start-with-the-task-id 'Link to this heading')

The project’s `pyproject.toml` exposes `isaaclab_tutorial.tasks` through the `isaaclab.tasks` entry-point group:

```
# pyproject.toml
[project.entry-points."isaaclab.tasks"]
so101-vial-place = "isaaclab_tutorial.tasks"
```

Importing the package loads the robot-specific configuration, where `gym.register()` connects a task ID to Isaac Lab’s `ManagerBasedRLEnv` and our `SO101VialEnvCfg`:

**Replace the contents**

- File: `src/isaaclab_tutorial/tasks/place_vial/config/so101/__init__.py`

- Markers: `# BEGIN lesson-01-registration` and `# END lesson-01-registration`

Paste the code below between the markers. Keep both marker comments.

```
gym.register(
    id="IsaacTutorial-Place-Vial-SO101",
    entry_point="isaaclab.envs:ManagerBasedRLEnv",
    disable_env_checker=True,
    kwargs={
        "env_cfg_entry_point": f"{_PACKAGE}.env_cfg:SO101VialEnvCfg",
        "rsl_rl_cfg_entry_point": f"{_PACKAGE}.agents.rsl_rl_ppo_cfg:SO101StatePPORunnerCfg",
        "default_agent": "rsl_rl",
    },
)
```

The `id` is the name you’ll pass to the command-line agents once the configuration has the actions and episode rules needed to run `env.step()`. The `SO101VialEnvCfg` configuration describes the task, including the robot. It gathers the scene, observations, events, and simulation timing.

The supplied `inspection_env_cfg.py` connects `SO101SceneCfg` to a `ManagerBasedRLEnvCfg` with empty actions, joint observations, and a reset event. Use its `IsaacTutorial-Inspect-SO101` task after adding the robot in the next lesson. The placement task registration above will be used once you add controls in Running Simple Agents.

## Prepare Joint Observations[#](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/latest/train-a-robot-arm/01-project-setup.html.md#prepare-joint-observations 'Link to this heading')

The robot inspection task uses Isaac Lab’s supplied `mdp.joint_pos` and `mdp.joint_vel` observation terms. They are re-exported through the tutorial’s `mdp` package; no custom joint-velocity implementation is needed. Use `SceneEntityCfg` to select the robot and joint order when defining observations. Task-specific observations, such as placement features and wrist images, live in `mdp/observations.py`.

The initial checkout fails configuration validation because the robot is missing; this is expected. After the SO-101 Robot lesson, use the supplied inspection task to run the robot scene with empty actions. The Vial and Rack lesson then adds the vial and rack. Running Simple Agents then adds policy actions and episode rules. Later lessons replace the same environment configuration with the teacher task and add the camera student. The project’s tests check the completed task, so run the full test suite after completing the distillation lesson.

## Review[#](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/latest/train-a-robot-arm/01-project-setup.html.md#review 'Link to this heading')

### Check Your Understanding[#](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/latest/train-a-robot-arm/01-project-setup.html.md#check-your-understanding 'Link to this heading')

What does task registration connect a task ID to?

Answer

Task registration connects a task ID to environment and agent configuration entry points. Those configurations may still be incomplete.

### Apply What You Learned[#](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/latest/train-a-robot-arm/01-project-setup.html.md#apply-what-you-learned 'Link to this heading')

Starting from `learning-path`, what pieces must be added before a simple agent can step the SO-101 task? Distinguish the existing scaffold from the registration, scene, actions, observations, resets, and ending conditions needed for a runnable task.

Agent Tutor Prompt

Copy this prompt and include your answer:

> Review my answer to this question: Starting from `learning-path`, what pieces must be added before a simple agent can step the SO-101 task? Distinguish the existing scaffold from the registration, scene, actions, observations, resets, and ending conditions needed for a runnable task.
> 
> My answer: \[paste your answer here\].
> 
> Check my reasoning about scaffold versus implemented behavior, registration, scene, actions, observations, resets, and endings. Ask for missing context rather than assuming it. Identify what I explained well, point out one gap or misconception, and ask a follow-up question that helps me improve my answer. Let me revise it before supplying a complete answer.

### Key Takeaways[#](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/latest/train-a-robot-arm/01-project-setup.html.md#key-takeaways 'Link to this heading')

- Stay on `learning-path` and compare your progress with the completed reference on `main`.

- Paste the task registration into its marked region during this lesson.

- Scene and MDP configuration connect the robot to observations, actions, rewards, and resets.

- The inspection scene runs after adding the robot; simple agents need the later controls and episode rules.