# Proximal Policy Optimization Proximal Policy Optimization (PPO) follows a repeating cycle: 1. Collect recent experience in a **rollout** (a batch of transitions gathered with the current policy). 2. Estimate which actions worked better than expected. 3. Compute a policy update that makes those actions more likely while making worse-than-expected actions less likely. 4. Update the policy while limiting how much one batch changes its behavior. Isaac Lab supplies the parallel robot environments and supports many RL algorithm implementations. In this learning path, we’ll use [RSL-RL](https://leggedrobotics.github.io/rsl_rl/), which supplies the PPO implementation. PPO is well suited to this robot-control task: it learns continuous actions from experience collected across Isaac Lab’s parallel environments, while its clipped update discourages abrupt policy changes. ![Four stages form one PPO iteration. The rollout policy collects experience from parallel Isaac Lab environments. Rewards and critic values produce return and advantage estimates. PPO compares the updated and rollout-policy action probabilities. A clipped objective updates the actor, a value loss updates the critic, and the updated actor collects a fresh rollout.](images/ppo-iteration.svg) Isaac Lab helps manage these stages. It runs the parallel environments, collects the rollout, and passes it to RSL-RL for return and advantage estimation, mini-batch updates, and checkpointing. The updated actor then collects a fresh rollout. ## Why Use Policy Optimization Let’s return to the stochastic policy from the previous lesson: $$ a_t \sim \pi_\theta(\cdot \mid o_t). $$ While training, the policy samples $a_t$ from that distribution. Sampling lets the robot try different actions instead of repeating the same motion in every environment. Each environment will produce a different outcome, and the policy can learn from that variation. Policy optimization uses the resulting rewards to adjust the policy directly: actions associated with useful outcomes become more likely, while actions associated with unhelpful outcomes become less likely. Over many rollouts, the probability distribution shifts toward action sequences that produce greater cumulative return, which *should* result in better behaviors. The rollout reflects the policy used to collect it. If the policy is **drastically** changed during an update, the rollout becomes a poor guide to how the new policy behaves. A large update can also discard useful behavior or reduce exploration before the policy has learned a reliable strategy. PPO addresses this problem by discouraging each update from changing the policy too abruptly. The result is a sequence of measured updates: collect experience, make useful actions more likely without changing the policy too much, and then collect fresh experience with the updated policy. ## Actor and Critic PPO is an **actor-critic** method. It trains two functions with different jobs: - The **actor**, $\pi_\theta$ (the end product of robot learning), produces the action distribution. Its parameters $\theta$ determine how the robot acts. - The **critic**, $V_\phi$, predicts the expected return from its current input. Its parameters $\phi$ are learned during training, but the critic does not command the robot. For continuous control, the actor (which is often used interchangeably with policy) commonly produces the mean and standard deviation of a Gaussian distribution. The mean represents its current action choice, while the standard deviation controls how widely training samples around that choice. The critic provides a baseline for evaluating an action. A positive return does not necessarily mean that the action was especially useful; the return may be typical for that situation. The critic estimates the return the policy can usually expect from that point, and PPO compares the observed outcome with this estimate. The actor and critic do not have to receive the same inputs. In the SO-101 task, for example, the actor uses the 60-value `policy` observation group; that count comes from the task’s configured observation terms, not from PPO itself. Sometimes it is useful to give the critic extra information about the environment that is not directly useful (or available) to the actor policy. We write these inputs as $o_t$ for the actor and $s_t$ for the critic: $$ \pi_\theta(\cdot \mid o_t) \qquad \text{and} \qquad V_\phi(s_t). $$ This extra critic information can help produce a better training signal. It does not become an input requirement for the trained actor. ![The actor receives observation o_t and produces an action distribution; a sampled action controls the environment. The critic receives its own observation s_t, which may include extra state information, and predicts expected return. During training, estimated advantages guide actor updates and estimated returns provide critic targets.](images/actor-critic-example.svg) ## Returns and Advantages After collecting a rollout, PPO uses the recorded rewards and the critic’s value predictions to estimate two quantities used in training: - The estimated return, $\hat{G}_t$, is the value that the critic should learn to predict. - The estimated advantage, $\hat{A}_t$, says whether the sampled action performed better or worse than expected. The previous lesson introduced the temporal-difference residual $$ \delta_t = r_{t+1} + \gamma V_\phi(s_{t+1}) - V_\phi(s_t). $$ It compares one observed reward plus the next value estimate with the current value estimate. PPO implementations commonly combine a sequence of these residuals with **generalized advantage estimation** (GAE): $$ \hat{A}_t = \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2\delta_{t+2} + \cdots. $$ The discount factor $\gamma$ controls how strongly future outcomes matter. The GAE parameter $\lambda$ controls how much information from later steps enters the estimate. Shorter estimates are usually less noisy but depend more on the imperfect critic. Longer estimates use more observed rewards but can vary more between rollouts. A positive $\hat{A}_t$ means the sampled action did better than the critic expected. The actor should make the action more likely. A negative $\hat{A}_t$ means it did worse than expected, so the actor should make it less likely. The critic is trained to predict the return target, improving the baseline used to evaluate future actions. Episode endings matter to this calculation. A true terminal state (i.e., success or failure) has no future reward to estimate, so the return does not bootstrap beyond it. A time-limit truncation may still have future value. The environment wrapper and RL library must preserve that distinction. ## The Clipped PPO Update PPO treats the actor that collected the rollout as the **old policy**, $\pi_{\theta_{\text{old}}}$. During optimization, it compares the new and old probabilities of each sampled action: $$ \rho_t(\theta) = \frac{\pi_\theta(a_t \mid o_t)} {\pi_{\theta_{\text{old}}}(a_t \mid o_t)}. $$ At the start of an update, the policies are the same and the ratio is 1. A ratio greater than 1 means the updated actor makes that sampled action more likely. A ratio below 1 means it makes the action less likely. An ordinary policy-gradient update would optimize $\rho_t(\theta)\hat{A}_t$. This has the right direction (it increases the probability of positive-advantage actions and decreases the probability of negative-advantage actions), but it does not prevent the probability from changing too far on one batch. PPO-Clip adds a guardrail for this: $$ L^{\text{clip}}(\theta) = \mathbb{E}_t\left[ \min\left( \rho_t(\theta)\hat{A}_t, \operatorname{clip}\left(\rho_t(\theta), 1-\epsilon, 1+\epsilon\right) \hat{A}_t \right) \right]. $$ Read this equation in three steps: 1. Use $\rho_t(\theta)$ to measure how the probability of the sampled action changed. 2. Multiply by $\hat{A}_t$ to reward changes that make positive-advantage actions more likely and negative-advantage actions less likely. 3. Compare this term with the clipped term and take the minimum so the optimizer gains no extra benefit from a helpful probability change beyond the clipping range. For example, with $\epsilon=0.2$ that range is 0.8 to 1.2. This does **not** mean that every probability is forced to remain inside those values, nor does it guarantee a fixed distance between the complete policies. Clipping removes the incentive for certain overly large changes. Implementations commonly monitor **Kullback-Leibler (KL) divergence** alongside clipping. KL divergence measures how much one probability distribution differs from another. A value of zero means the distributions are identical; a larger value means they differ more. In PPO, it summarizes how much the actor’s action distribution changed from the rollout policy to the updated policy. A sudden increase can reveal a larger-than-intended update even though the clipped objective was used. #### TIP PPO ratio clipping and action clipping are different. Ratio clipping limits the incentive to change the policy during learning. It is also common to clip the action outputs to the environment’s valid range before the robot uses them. ## One PPO Iteration in Isaac Lab In future lessons, one iteration of PPO follows this sequence: 1. **Collect a rollout.** RSL-RL runs the current actor in every Isaac Lab environment for a specified number of steps. Each transition records the actor and critic observations, sampled action and its old log probability, reward, value estimate, and episode-ending signals. 2. **Estimate returns and advantages.** RSL-RL uses the rewards, ending signals, critic values, $\gamma$, and $\lambda$ to calculate $\hat{G}_t$ and $\hat{A}_t$. 3. **Make mini-batches.** The rollout is flattened across environment and time dimensions, shuffled, and divided into mini-batches. 4. **Update the actor and critic.** RSL-RL makes several learning passes, or epochs, over the same rollout. Each mini-batch contributes the clipped policy objective, value loss, and entropy bonus before backpropagation. 5. **Record progress and repeat.** RSL-RL logs metrics and periodically saves a checkpoint. The updated actor then collects a fresh rollout. Using several epochs extracts more learning signal from an expensive rollout. Using too many would make the data increasingly stale because it came from the old policy. PPO is therefore an **on-policy** method: it reuses one recent batch for a limited number of updates, then replaces it with new experience. ## What to Monitor No single training curve proves that PPO learned the desired task. Analyze several kinds of evidence together. *The following figures use synthetic data to illustrate possible patterns.* A short trial run, often called a **smoke test**, answers a narrow question: can Isaac Lab create the environments, collect a rollout, run PPO updates, and write a checkpoint? A few iterations are not enough to judge policy quality. ### Return and Reward Terms Reward return and individual reward terms show what the optimizer is improving. A rising return can still describe reward exploitation rather than task completion. ![Illustrative curves show mean return rising strongly while evaluated task success remains almost flat.](images/ppo-monitor-return.svg) ### Task Metrics Task metrics such as grasp, lift, insertion, and stable-placement success show which parts of the behavior the policy completes. ![Illustrative task metrics show grasp improving first, followed by lift and insertion, while stable placement remains lowest.](images/ppo-monitor-task-metrics.svg) ### KL Divergence KL divergence shows how far the updated policy moved from the rollout policy. A sudden increase can indicate an unstable update. ![An illustrative KL-divergence curve stays near a run-specific target range before one sharp spike.](images/ppo-monitor-kl.svg) ### Entropy and Action Standard Deviation Entropy and action standard deviation describe exploration. A rapid collapse can leave the actor stuck; persistently high noise can prevent precise grasping or insertion. ![Illustrative normalized entropy and action-standard-deviation curves decline gradually and then collapse together before training ends.](images/ppo-monitor-exploration.svg) ### Value Loss Value loss shows how well the critic fits its return targets. A spike can follow a change in behavior, but persistent growth or non-finite values requires investigation. ![Two illustrative value-loss curves compare a temporary spike followed by recovery with persistent growth.](images/ppo-monitor-value-loss.svg) ### Episode Length and Ending Reasons Episode length and ending reasons distinguish faster success from early failure or timeout. ![Mean episode length falls across training, while aligned ending-reason bars show middle-training episodes ending mainly in failure and late-training episodes ending mainly in success.](images/ppo-monitor-episode-endings.svg) Policy and value losses are diagnostic signals, not scores to maximize or minimize in isolation. Their signs and scales also depend on how a library logs the objective. Interpret them together with behavior, task metrics, KL divergence, and action noise. ## Review ### Check Your Understanding What does the critic estimate? ### Answer The critic estimates the expected return from its current input. That estimate helps judge how an action performed relative to expectations. ### Apply What You Learned How would you explain one PPO iteration and decide whether an increase in reward reflects better task behavior? Describe rollout collection, actor and critic updates, and the metrics or behavior you would inspect. ### Agent Tutor Prompt Copy this prompt and include your answer: > Review my answer to this question: How would you explain one PPO iteration and decide whether an increase in reward reflects better task behavior? Describe rollout collection, actor and critic updates, and the metrics or behavior you would inspect. > My answer: [paste your answer here]. > Check my reasoning about actor and critic, rollout collection, returns and advantages, clipped objective, and task metrics. Ask for missing context rather than assuming it. Identify what I explained well, point out one gap or misconception, and ask a follow-up question that helps me improve my answer. Let me revise it before supplying a complete answer. ### Key Takeaways - PPO alternates collecting rollouts with updating an actor and critic. - The clipped update limits one incentive for large policy changes; it is not a guarantee of safe behavior. - Inspect task outcomes alongside return, KL divergence, exploration, and value loss in an Isaac Lab run. ### Resources - [Hugging Face Deep RL Course: Introduction to PPO](https://huggingface.co/learn/deep-rl-course/unit8/introduction) — An introduction to the PPO unit, including its clipped objective and a practical training exercise. - [OpenAI Spinning Up: Proximal Policy Optimization](https://spinningup.openai.com/en/latest/algorithms/ppo.html) — An explanation of PPO objectives, key equations, pseudocode, and implementation parameters. - [PyTorch TorchRL Tutorial: Reinforcement Learning With PPO](https://docs.pytorch.org/tutorials/intermediate/reinforcement_ppo.html) — A code walkthrough for implementing and training a PPO agent with TorchRL. - [PPO for LLMs: A Guide for Normal People](https://cameronrwolfe.substack.com/p/ppo-llm) — An explanation of how PPO is used to fine-tune language models with reinforcement learning from human feedback.