# Policy Distillation Imagine you train a strong policy in simulation that knows the exact pose of every object. How would you use this policy on a physical robot that only has access to camera images and joint measurements? **Policy distillation** lets you use the simulation-only policy as a teacher for a new student policy whose observations are available on the physical robot. ## Why Use a Teacher and a Student The **teacher** is first trained to solve the task. It may observe privileged simulation state such as exact object pose, contact state, or a target frame. These clean measurements make the learning problem easier, but they are not available on the physical robot. The **student** receives inputs available on the physical robot, such as camera images, joint positions, and joint velocities, along with its previous action. It must infer from those inputs what the teacher knows directly. Both policies produce the same action representation so that the teacher’s output can serve as a target. For example, both might output five arm-joint increments and one gripper increment. ![At the same simulation state, privileged state enters a frozen teacher while wrist RGB and proprioception enter a camera student. The student's action advances the environment, the teacher supplies a target action, and an action-matching loss updates only the student.](images/student-driven-distillation.svg) ## The Distillation Loop Distillation repeats the following steps: 1. The environment provides limited observations to the student and privileged observations to the teacher at the same simulation state. 2. The student produces an action $a_t^{S}$, and the frozen teacher produces a target action $a_t^{T}$ from their respective observations of that same state. “Frozen” means the teacher’s parameters stay fixed. 3. The student’s action advances the environment, so the next state reflects the student’s choices and mistakes. The teacher’s action is a training target; it does not move the robot. 4. The training algorithm measures the difference between the two actions. A common action-matching loss is mean squared error: $$ \mathcal{L}(\theta) = \frac{1}{d}\sum_{i=1}^{d} \left(a^{S}_{t,i} - a^{T}_{t,i}\right)^2, $$ where $d$ is the number of action components and $\theta$ represents the student’s parameters. 5. The optimizer updates the student to reduce this loss. The teacher remains unchanged. This is a form of **imitation learning** because the student learns from the teacher’s actions. PPO changes a policy using rewards and estimated advantages, while distillation changes the student using a supervised action target. For the action labels to be meaningful, keep the physical task, control frequency, action ordering, scaling, and clipping consistent between teacher and student. The observation spaces are intentionally different. The student should never receive a training-only variable that will disappear at deployment. A useful question for each input is: how will the physical robot measure it? ## Recognize the Limits A low action-matching loss does not prove that the student completes the task. The teacher may be imperfect, two observations may look identical while requiring different actions, or a small action error may compound over time. Evaluate the teacher and student with the same physical success criterion, starting-state distribution, episode limit, and action interface. Compare task success and dominant failures as well as the distillation loss. The difference between teacher and student performance tells you how much capability was lost when you changed the observation space. In the SO-101 task, for example, the student uses two consecutive 64 by 48 wrist RGB frames plus proprioception. ## Review ### Check Your Understanding Which policy’s weights change during the distillation loop in this lesson? ### Answer Only the student’s weights change. The teacher stays frozen and supplies target actions at the states visited by the student. ### Apply What You Learned How would you trace one student-driven distillation step and explain a failure that low action-matching loss might miss? Identify the student action, teacher target, inputs, and updated weights, then describe a possible rollout failure. ### Agent Tutor Prompt Copy this prompt and include your answer: > Review my answer to this question: How would you trace one student-driven distillation step and explain a failure that low action-matching loss might miss? Identify the student action, teacher target, inputs, and updated weights, then describe a possible rollout failure. > My answer: [paste your answer here]. > Check my reasoning about student action, teacher target, observation availability, loss, and rollout evaluation. Ask for missing context rather than assuming it. Identify what I explained well, point out one gap or misconception, and ask a follow-up question that helps me improve my answer. Let me revise it before supplying a complete answer. ### Key Takeaways - The teacher supplies action targets from privileged simulation observations. - The student acts using inputs intended to exist on the robot; only the student is updated. - The SO-101 module uses this distinction to move from a state teacher to a wrist-camera student. ### Resources - [Isaac Lab reinforcement-learning guide](https://isaac-sim.github.io/IsaacLab/release/3.0.0/source/concepts/reinforcement_learning.html) covers the RSL-RL teacher-student workflow and which checkpoints are compatible. - [RSL-RL algorithm overview](https://github.com/leggedrobotics/rsl_rl/blob/main/docs/guide/overview.rst) describes its student-driven behavior-cloning implementation. - [RSL-RL configuration guide](https://github.com/leggedrobotics/rsl_rl/blob/main/docs/guide/configuration.rst) lists the student, teacher, observation-group, and loss settings.