SO-101 Reach Task - Part 1: Training the Policy
Create a reach environment in Isaac Lab, train an SO-101 policy with Proximal Policy Optimization (PPO), and export it as ONNX.
To use the supplied policy without training your own, skip to Part 2: Simulation or Part 3: Deployment.
Prerequisites
Before starting, ensure you have:
-
Isaac Lab installed — follow the Isaac Lab installation guide. The shipped policy was trained with Isaac Lab v2.3.1; later versions may need minor API changes.
-
SO-101 USD asset — a URDF/USD of the SO-101 arm imported into Isaac Lab. See Importing a New Asset if you need to convert from URDF.
-
Familiarity with manager-based RL environments — review Creating a Manager-Based RL Environment before proceeding.
-
A custom Isaac Lab project — create one with
./isaaclab.sh --newas described in Build your Own Project or Task.
Task Overview
The reach task is a goal-conditioned RL problem: a policy controls 4 arm joints to move the end-effector (EE) — the fingertip — to a target position in 3-D space.
The policy shipped with Isaac OS was trained with NVIDIA Isaac Lab, using some project-specific subclasses of the stock Isaac Lab classes. This tutorial builds the equivalent from the stock classes, so a few class names here differ from those recorded in the shipped IO descriptor; those differences are called out where they matter. The observation and action contracts are the same, and those are what Part 3 depends on.
Key characteristics:
- State-only — no vision; the policy operates on proprioceptive observations and a goal command.
- 4-DOF control — the policy outputs deltas for four
degrees of freedom (DOF):
shoulder_pan,shoulder_lift,elbow_flex, andwrist_flex. Thewrist_rollandgripperjoints are observed but not actuated. - Goal sampling — random or grid-based 3-D goals within the reachable workspace.
Isaac Lab ships reach environments under
isaaclab_tasks.manager_based.manipulation.reach which you can use as
a starting point. The sections below describe how to build a
SO-101-specific variant.
Defining the Scene
The scene consists of the SO-101 articulation, a table, and basic
lighting. Use InteractiveSceneCfg with your SO-101
ArticulationCfg:
See Creating an empty scene for more details on scene configuration.
Defining Actions
The policy controls 4 arm joints. The two remaining joints (wrist_roll, gripper) appear in the observation but are not actuated.
Actions are deltas against the joint position the robot is currently at, not targets in an absolute joint frame:
The scale parameter converts the raw policy output (clipped to
[-1, 1]) into radians of displacement per control step. At 0.2 the
policy can move any joint by at most 0.2 rad per tick, which at the
50 Hz control rate bounds joint speed at 10 rad/s. Lower it if the
real servos cannot track the commanded steps.
The deployment runtime implements the delta law and only the delta
law: it adds the policy output to the measured joint position on
every tick. A policy trained with JointPositionActionCfg — which
adds the action to the default pose instead — would be driven under
the wrong law and send the arm somewhere else entirely. The runtime
reads no training-time metadata, so it cannot detect the mismatch and
will not refuse the policy; matching the action law is your
responsibility. use_zero_offset=True matters for the same reason:
a delta has nowhere to put a per-joint offset.
The policy shipped with Isaac OS trains against
DelayedDeltaJointPositionActionCfg, a subclass that adds
randomized action delay and a per-tick displacement clamp. Both are
domain randomization for real servo behavior; the deployed control law
is the plain delta above.
Defining Observations
The policy receives a 20-dimensional observation vector assembled from four terms. The table below summarizes its structure:
ee_position_in_robot_root_frame is a custom observation term you
implement in your project using a FrameTransformerCfg to track
the fingertip position with an appropriate offset from the gripper
link.
❓ Why is there no joint velocity term?
The Feetech speed is differentiated from a coarse encoder and is too noisy for the policy to learn from. The runtime operator builds a fixed 20-D observation, so a 26-D policy needs a runtime operator change before it can run. See Part 3: Deployment for the layout the runtime builds and the tensor shapes it checks at startup.
Defining Rewards
The reward structure encourages the end-effector to reach the goal while keeping motions smooth and within joint limits:
Tune the weights based on convergence behavior. A well-trained reach policy typically converges in 3000–5000 iterations with 4096 environments.
❓ What reward does the shipped policy use?
The policy shipped with Isaac OS uses a multi-scale kernel reward that gives stronger signal as the end-effector approaches the goal:
Defining Terminations
Episodes need a termination condition so the environment resets and the agent can practice from varied starting states. For the reach task a simple time-out is sufficient:
The episode terminates after a fixed duration (e.g. 12 seconds). You
may also add a goal_reached termination if you want to reset early
on success.
See Defining termination criteria for more details.
Defining Commands
Goals are sampled uniformly within the SO-101 reachable workspace using
UniformPoseCommandCfg:
Adjust the position ranges to match your robot’s workspace. Deployment
ranges (the policy.goal_pos_*_range keys in Part 3) must stay within
these training bounds but may be a tighter subset for added safety on
real hardware.
Training Configuration
Tie all the components together into a ManagerBasedRLEnvCfg. This
single config class is what Isaac Lab uses to instantiate the
environment:
Next, configure the PPO training hyperparameters via the RSL-RL runner config. The table below lists recommended starting values:
Training logs are written to logs/rsl_rl/<experiment_name>/<timestamp>/.
Monitor reward curves via TensorBoard:
Evaluating and Exporting
Once training completes, evaluate the policy in simulation and export it to ONNX for deployment:
The play script exports:
exported/policy.onnx— the ONNX model used by Isaac OS for real-time inference via NVIDIA TensorRT.exported/policy.pt— the JIT-traced PyTorch model (useful for debugging but not used for deployment).
Both files are written next to the checkpoint in the
exported/ subdirectory.
Before deploying on real hardware, it is advisable to perform sim-to-sim validation by running the exported ONNX policy in an independent simulator such as MuJoCo. This catches discrepancies between the training and deployment observation pipelines (e.g. joint ordering, scale mismatches) in a safe, reproducible environment. Sim-to-sim validation is beyond the scope of this tutorial.
Exporting IO Descriptors
IO descriptors are YAML files that capture the exact observation and action layout of a trained policy — joint names, shapes, scales, offsets, and data types. They record the contract between training and deployment. The deployment runtime does not read them; it is the reference you transcribe into Part 3’s configuration so that the runtime assembles tensors in the same order and scale the policy was trained with.
See IO Descriptors 101 for a detailed explanation.
Export IO descriptors for your environment:
This produces a YAML file that looks like:
Check these fields before moving on:
actions[0].full_pathnames a delta class (Relative...or...Delta...).actions[0].offsetis zero.- The articulation’s
joint_namesstart with the action’s joint names in the same order.
Success criterion: exported/policy.onnx exists and the descriptor passes all three
checks.
Four descriptor values become configuration keys in Part 3:
actions[0].scale sets motion_control.action_scale,
default_joint_pos_limits narrowed to the controlled joints sets
motion_control.joint_pos_limits_lower and
motion_control.joint_pos_limits_upper,
joint_pos_offsets sets policy.default_joint_position, and
scene.dt sets policy.inference_period.
Keep both the policy.onnx and the IO descriptors YAML file. Part 3
loads the ONNX and uses the descriptor as the reference for those
configuration values.
Summary
In this part you learned how to:
- Define a reach RL environment with scene, actions, observations, rewards, terminations, and commands using Isaac Lab’s manager-based framework.
- Train a policy with PPO using 4096 parallel environments.
- Export the trained policy to ONNX format for real-time inference.
- Generate IO descriptors that capture the training contract for deployment.
Next Steps
- Part 2: Simulation — run the policy against a simulated arm in Isaac Sim before hardware deployment.
- Part 3: Deployment — configure, build, and run the trained policy on real SO-101 hardware.
- Sim2Real Deployment of Policies Trained in Isaac Lab — Isaac Lab’s deployment reference.