# Project Setup
For this module, we’ll start with a working Isaac Lab project rather than build
on the [Isaac Lab Basics](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/2026-10.0/isaac-lab-basics/02-isaac-lab-projects.md)
workspace. We’ll explore the [Isaac Lab
Tutorial](https://isaac-sim.github.io/IsaacLab/release/3.0.0/source/setup/tutorial.html)
together, one piece at a time.
#### TIP
Refer back to [Development Workflow for Isaac
Lab](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/2026-10.0/isaac-lab-basics/03-development-workflow.md) for remote development,
editor and container setup, and ways to preserve your changes
as you complete each lesson.
## Clone the Isaac Lab Tutorial
The [Isaac Lab Tutorial repository](https://github.com/isaac-sim/IsaacLabTutorial)
uses two branches for this course: `main` is the completed reference, and
`learning-path` contains the scaffold you will complete. Stay on `learning-path`
throughout the lessons and copy each lesson’s code into its marked locations.
1. Clone the repository on your GPU workstation or remote GPU host:
```bash
git clone https://github.com/isaac-sim/IsaacLabTutorial.git
cd IsaacLabTutorial
```
Take a look around the `IsaacLabTutorial` directory. You’ll recognize a
project structure similar to the cartpole project, with a few more pieces
to support the more challenging vial-placement task.
2. Switch to the branch created for this module:
```bash
git switch learning-path
```
Each lesson adds a few key components to this branch. Keep your changes on
this branch, and run `git diff main -- src/` to compare your current code
with the complete reference without switching branches.
3. Ensure [Git LFS](https://git-lfs.com/) is installed on the GPU host, then
download the assets and sync the dependencies:
```bash
git lfs install --local
git lfs pull
uv sync --locked
```
### Optional Packages for Recording and Experiment Tracking
To record videos and track your training runs, you’ll need a few additional
packages. The tutorial declares these as optional dependencies in
`pyproject.toml`. The commands in these lessons use `uv run --extra` to install
the packages needed for each command:
- Use `--extra video` to save videos with `record_video` or `--video`.
The tutorial’s `video` extra installs Isaac Lab’s MoviePy 1.x dependency.
- Use `--extra ovrtx` with `--visualizer newton_rtx` to render an RTX view of your robot.
- Use `--extra wandb` to track training runs with Weights & Biases in
[Managing Experiments](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/2026-10.0/train-a-robot-arm/07-managing-experiments.md).
You’ll record your first scenes and simple agents with `--visualizer newton`. We’ll also provide instructions for recording with RTX, but it is optional.
See the [RTX recording note](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/2026-10.0/train-a-robot-arm/05-teacher-policy.md#if-rtx-recording-fails) before trying it.
### Interpreting Startup Messages
Isaac Lab prints a lot of output when it starts, and some of it looks scarier
than it is. Many messages are just warnings: the simulation keeps running
normally. Here are the ones you’re most likely to see:
- `Failed to get root com velocity`: This is expected. The SO-101 is bolted to
the table (a fixed-base robot), so Isaac Lab just sets its root velocity to
zero and moves on.
- `Hub not found`: An optional asset-cache service isn’t available. As long as
the robot and scene still load, you’re fine.
- `FutureWarning` (about shape colors), `SyntaxWarning` (from MoviePy), or
PyTorch JIT deprecation notices: These are heads-ups for library maintainers,
not signs that your lesson broke.
- `Could not find git repository`: One of the installed packages doesn’t
include Git metadata. That’s harmless, but it’s a good reminder to note your
project’s commit and keep its lockfile so you can reproduce your run later.
If `uv` warns that its version is outside the project’s `uv-build` range, don’t
just delete the version constraint to make the warning go away. Copy the
message and check which tool versions the project supports first.
So how do you know when something actually went wrong? Look for a Python
traceback, a nonzero exit status, missing checkpoints, or output that doesn’t
make sense. When troubleshooting, save the *full* terminal output, not just the
last few lines.
You may also see some optional deployment and export tests get skipped if
their extra dependencies aren’t installed. That’s OK, but keep in mind that the
remaining tests passing doesn’t tell you whether those optional workflows work.
## The Project Layout
Here is an overview of the tutorial project layout. Several functional pieces of
code were removed in the `learning-path` branch to give you a chance to
implement them, but all files and implementations are present in the `main`
branch.
```text
src/isaaclab_tutorial/
├── tasks/place_vial/
│ ├── config/so101/
│ │ ├── __init__.py # Task registration
│ │ ├── env_cfg.py # Scene and teacher task
│ │ ├── camera_env_cfg.py # Student camera observations
│ │ └── agents/ # PPO and distillation settings
│ ├── mdp/ # Task behavior
│ └── reset/ # Reset data and curriculum
└── utils/ # Supporting utilities
```
This lesson and the remaining lessons in this module build on each other as follows:
1. We’ll register the SO-101 vial-placement task with Gymnasium.
2. We’ll add the robot to an otherwise empty scene.
3. We’ll add a vial, rack, desk, and contact sensors to the scene.
4. We’ll add the actions and observations needed for simple agents.
5. We’ll train a teacher policy using privileged information from simulation.
6. We’ll train a student policy using only readily available observations.
### Full diff of the `learning-path` and `main` branches
The diff below was generated using [diff2html](https://diff2html.xyz/). You can
skim through it to see the changes made across the lessons, or you can run `git diff main -- src/` in your local clone of the repository. Here is a link to open
the diff in a separate browser tab:
[learning-path and main branch diff](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/2026-10.0/_downloads/44736451fd77f31f565d6d570ea8964b/robot-arm-learning-path-to-main.html).
## The Robots and Tasks
Like the cartpole template from the previous module, the tutorial project is an
external Isaac Lab project: it lives in its own repository outside the Isaac Lab
framework repository. Over the next several lessons, you’ll turn it into a
vial-placement task, check that it runs, and then train two policies.
This section describes the robot and task you’ll be working with. You will not
need to copy and paste any code until the [Start With the Task
ID]() section below.
The following snippets explain the configuration built across these lessons;
you do **not** need to paste these overview snippets into `learning-path`.
You’ll paste the complete registration in **Start With the Task ID** below. Once added,
`gym.register()` in `/so101/__init__.py` tells Isaac Lab where to find the task
configuration.
```python
_PACKAGE = "isaaclab_tutorial.tasks.place_vial.config.so101"
gym.register(
...
kwargs={
...
"env_cfg_entry_point": f"{_PACKAGE}.env_cfg:SO101VialEnvCfg",
},
)
```
The `env_cfg_entry_point` points to the `SO101VialEnvCfg` class inside the
`so101` package, which holds the SO-101 robot configuration for the `place_vial` task.
This Gymnasium registration tells Isaac Lab which Python class to load when
you request the task.
`SO101VialEnvCfg` is a
[`ManagerBasedRLEnvCfg`](https://isaac-sim.github.io/IsaacLab/release/3.0.0/source/api/lab/isaaclab.envs.html#isaaclab.envs.ManagerBasedRLEnvCfg)
object. It contains the SO-101 scene, joint observations, a reset event, and
simulation settings. Here is a simplified view of the configuration in `env_cfg.py`:
```python
class SO101VialEnvCfg(ManagerBasedRLEnvCfg):
scene : SO101SceneCfg = ...
actions : ActionsCfg = ...
observations: ObservationsCfg = ...
events : EventsCfg = ...
rewards : RewardsCfg = ...
terminations: TerminationsCfg = ...
sim : SimulationCfg = ...
```
These classes implement the [MDP concepts from Problem Specification](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/2026-10.0/robot-learning-fundamentals/02-problem-specification.md):
`ActionsCfg` defines what the policy controls, `ObservationsCfg` defines its
inputs, and `RewardsCfg` defines its training feedback. `EventsCfg` prepares
resets and `TerminationsCfg` defines when an episode ends. The scene uses the
SO-101 asset configuration supplied by Isaac Lab.
```python
from isaaclab_assets.robots.so101 import SO101_CFG
class SO101SceneCfg(InteractiveSceneCfg):
robot = SO101_CFG.replace(...)
vial = ...
rack = ...
desk = ...
sensors = ...
```
The `SO101_CFG` is an `ArticulationCfg` object that contains the robot’s
articulation root, joint axes and limits, mass and inertia, collision geometry,
and drives. The asset and its actuator settings must match the action interface:
our SO-101 policy sends **relative joint-position targets**, while cartpole’s
policy sends **joint effort**. For another robot, choose gains and limits that
match its mechanics, actuator model, and physics backend. See Isaac Lab’s
[asset-import
guide](https://isaac-sim.github.io/IsaacLab/release/3.0.0/source/how-to/import_new_asset.html)
for the conversion workflow.
## How the Task Configuration Fits Together
The [manager-based
workflow](https://isaac-sim.github.io/IsaacLab/release/3.0.0/source/how-to/create_manager_rl_env.html)
assembles the scene and task pieces in one `ManagerBasedRLEnvCfg`. Each piece
answers a different question:
Use the configuration to answer concrete questions about the vial task:
- **Scene:** Which robot, vial, rack, and sensors exist?
- **Actions and observations:** How does a policy output move a joint, and what
measurements come back to the policy?
- **Rewards and terminations:** What counts as progress, success, failure, or
a timeout?
- **Events:** Where do the robot and vial start on each attempt?
- **Simulation:** Which physics backend advances the scene, and how often?
You will add these pieces as each lesson needs them. The environment’s physics
time step multiplied by `decimation` gives the interval between policy actions.
The separate agent configuration controls how PPO or distillation trains a
policy from the resulting experience.
## Start With the Task ID
The project’s `pyproject.toml` exposes `isaaclab_tutorial.tasks` through the
`isaaclab.tasks` entry-point group:
```toml
# pyproject.toml
[project.entry-points."isaaclab.tasks"]
so101-vial-place = "isaaclab_tutorial.tasks"
```
Importing the package loads the robot-specific configuration, where
`gym.register()` connects a task ID to Isaac Lab’s `ManagerBasedRLEnv` and our
`SO101VialEnvCfg`:
**Replace the contents**
- File: `src/isaaclab_tutorial/tasks/place_vial/config/so101/__init__.py`
- Markers: `# BEGIN lesson-01-registration` and `# END lesson-01-registration`
Paste the code below between the markers. Keep both marker comments.
```python
gym.register(
id="IsaacTutorial-Place-Vial-SO101",
entry_point="isaaclab.envs:ManagerBasedRLEnv",
disable_env_checker=True,
kwargs={
"env_cfg_entry_point": f"{_PACKAGE}.env_cfg:SO101VialEnvCfg",
"rsl_rl_cfg_entry_point": f"{_PACKAGE}.agents.rsl_rl_ppo_cfg:SO101StatePPORunnerCfg",
"default_agent": "rsl_rl",
},
)
```
The `id` is the name you’ll pass to the command-line agents once the
configuration has the actions and episode rules needed to run `env.step()`.
The `SO101VialEnvCfg` configuration describes the task, including the robot. It
gathers the scene, observations, events, and simulation timing.
The supplied `inspection_env_cfg.py` connects `SO101SceneCfg` to a
`ManagerBasedRLEnvCfg` with empty actions, joint observations, and a reset event.
Use its `IsaacTutorial-Inspect-SO101` task after adding the robot in the next lesson.
The placement task registration above will be used once you add controls in
Running Simple Agents.
## Prepare Joint Observations
The `mdp` package brings together helper functions from three places:
- Isaac Lab provides general helpers such as `joint_pos` and `joint_vel`.
- Isaac Lab’s lift-task utilities provide helpers such as `body_state_b` and
`abnormal_robot_state`.
- This tutorial adds vial-placement behavior in `tasks/place_vial/mdp/`.
An observation term tells the environment how to collect a piece of information
for the policy. Let’s follow two terms that read the robot’s joint positions and
velocities. This configuration is already supplied in
`so101/lesson_scaffold.py`, so you can read along without copying it into your
project. The file supplies the imports, including `ObsGroup` and `ObsTerm`,
which are short names for Isaac Lab’s observation group and observation term
configuration classes.
```python
@configclass
class JointObservationsCfg(ObsGroup):
joint_pos = ObsTerm(
func=mdp.joint_pos,
params={"asset_cfg": SceneEntityCfg("robot")},
)
joint_vel = ObsTerm(
func=mdp.joint_vel,
params={"asset_cfg": SceneEntityCfg("robot")},
)
def __post_init__(self):
self.enable_corruption = False
self.concatenate_terms = True
```
Start with `joint_pos`. Its `func` field selects the helper that will read the
positions. Notice that `mdp.joint_pos` has no parentheses: the configuration
passes the function itself to the manager, which will call it later. The
`params` dictionary supplies a keyword argument named `asset_cfg`.
`SceneEntityCfg("robot")` selects the scene asset named `robot`; because this
example does not specify joint names, it uses all of that robot’s joints.
Now look inside the helper. This excerpt from Isaac Lab’s source file
`mdp/observations.py` shows its essential logic:
```python
def joint_pos(env, asset_cfg):
asset = env.scene[asset_cfg.name]
return asset.data.joint_pos.torch[:, asset_cfg.joint_ids]
```
The manager supplies `env`, the running environment, and passes in `asset_cfg`
from the configuration. The first line looks up the robot in `env.scene`.
The second reads its joint positions as a PyTorch tensor. A *tensor* is a
multidimensional array: six joint positions from each of 16 robots form a
tensor with shape `(16, 6)`. Each row belongs to one environment, and each
column holds one joint position.
In `[:, asset_cfg.joint_ids]`, the `:` selects every environment, while
`joint_ids` selects the joints resolved by the manager. The same operation
therefore reads all 16 robots, even when their positions differ. If you later
select particular joint names in `SceneEntityCfg`, the manager resolves those
names to indices before the helper uses them.
The `joint_vel` term follows the same pattern but reads `asset.data.joint_vel`
instead. For the SO-101’s six joints, positions are in radians and velocities
are in radians per second. With `concatenate_terms=True`, the observation
manager joins the two outputs in term order: six positions followed by six
velocities. For 16 robots, the resulting observation has shape `(16, 12)`.
`enable_corruption=False` disables observation noise for this group, so you can
start by inspecting the simulated measurements directly.
These two helpers read current values without keeping their own history.
Later, you’ll meet *stateful terms* that remember information between steps,
such as earlier camera frames. For those terms, the manager creates a class
instance, calls its `__call__` method during operation, and calls
`reset(env_ids)` to clear episode-specific memory when environments restart.
You can use these existing joint helpers through the tutorial’s `mdp` package,
which re-exports them from Isaac Lab. Task-specific observations, such as
placement features and wrist images, live in the tutorial’s
`mdp/observations.py`.
The initial checkout will fail configuration validation because the robot has
not been added yet. This is expected—you’ll fill in the missing pieces over
the next lessons:
- **SO-101 Robot:** Add the robot, then use the supplied inspection task to run
the robot scene with empty actions.
- **Vial and Rack:** Add the vial and rack.
- **Running Simple Agents:** Add policy actions and episode rules.
- **Later lessons:** Replace the same environment configuration with the
teacher task and add the camera student.
The project’s tests check the completed task, so wait until you finish the
distillation lesson to run the full test suite.
## Review
### Check Your Understanding
What does task registration connect a task ID to?
### Answer
Task registration connects a task ID to environment and agent configuration entry points. Those configurations may still be incomplete.
### Apply What You Learned
Starting from `learning-path`, what pieces must be added before a simple agent can step the SO-101 task? Distinguish the existing scaffold from the registration, scene, actions, observations, resets, and ending conditions needed for a runnable task.
### Agent Tutor Prompt
Copy this prompt and include your answer:
> Review my answer to this question: Starting from `learning-path`, what pieces must be added before a simple agent can step the SO-101 task? Distinguish the existing scaffold from the registration, scene, actions, observations, resets, and ending conditions needed for a runnable task.
> My answer: [paste your answer here].
> Check my reasoning about scaffold versus implemented behavior, registration, scene, actions, observations, resets, and endings. Ask for missing context rather than assuming it. Identify what I explained well, point out one gap or misconception, and ask a follow-up question that helps me improve my answer. Let me revise it before supplying a complete answer.
### Key Takeaways
- Stay on `learning-path` and compare your progress with the completed reference on `main`.
- Paste the task registration into its marked region during this lesson.
- Scene and MDP configuration connect the robot to observations, actions, rewards, and resets.
- The inspection scene runs after adding the robot; simple agents need the later controls and episode rules.