# Reinforcement Learning Reinforcement learning (RL) trains a policy through interaction. Instead of receiving the correct action for each observation, the policy tries actions, observes their consequences, and uses rewards to improve its future decisions. Here’s the standard RL loop: ![An agent sends an action to an environment containing an articulated robot arm. The environment returns a reward and an observation to the agent on separate paths.](images/robot-learning-loop.svg) ## Describing a Task as an RL Problem You’ve already seen how to describe a robot task in terms of observations, actions, a policy, and successful behavior. To turn that task into an RL problem, we connect those components in a repeated interaction and assign a reward to every possible outcome (i.e., change in state). This is the Markov decision process introduced in the lesson on [problem specifications](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/2026-10.0/robot-learning-fundamentals/02-problem-specification.md): the environment has a state, the policy receives an observation, and each action leads to a new state and reward. [Isaac Lab](https://isaac-sim.github.io/IsaacLab/release/3.0.0/index.html) builds on the [Gymnasium environment API](https://gymnasium.farama.org/introduction/basic_usage/) and uses a similar interface. The main difference is that an Isaac Lab RL environment contains many parallel sub-environments and passes around batches of tensors. To see the basic idea, start with an inference loop for one environment. Isaac Lab handles batches of environments, so this is a conceptual example rather than a complete Isaac Lab script: ```python observations, info = env.reset() episode_done = False while not episode_done: actions = policy(observations) observations, rewards, terminated, truncated, info = env.step(actions) episode_done = terminated or truncated env.close() ``` Think of this code snippet as a robot interacting with its environment. Each call to `.step()` moves forward through time. The **environment** (`env`) contains everything the action can affect, including the robot, other objects, physics, and task logic. An **episode** is a sequence of steps that ends when the task succeeds, fails, or reaches a limit. This could be the gear-assembly robot, where in each loop: 1. The policy selects arm-joint increments ($a_t$) given joint measurements and an estimated shaft pose ($o_t$). 2. The robot joints move and the environment advances to its next (internal) **state**. 3. The environment returns a reward $r_{t+1}$ and updated measurements $o_{t+1}$. The **reward** could indicate progress toward meshing the gear with the installed gear and seating it on the shaft. The `.step()` method returns additional information about whether the episode ended and general diagnostic information. **Termination** means that the task reached a defined ending, such as success or failure. **Truncation** stops the interaction because of an external limit, such as the maximum episode length. The environment’s **state** and the agent’s **observation** are related but not necessarily identical. The state contains the physical quantities needed to describe the scene and predict how it can change. The observation contains only the information exposed to the policy. For example, the environment might track the exact pose, mass, and contact state of a gear, while the insertion policy receives only an estimated shaft pose and the robot’s joint measurements. The policy must act from what it can observe, not from everything the environment knows. An **agent** is the decision-making system that interacts with the environment. Its policy determines how it selects actions from observations, sometimes using memory of earlier steps. The agent may also use other components, such as a value function to estimate expected future rewards. The snippet above shows its policy call without a separate `agent` object. During training, a learning algorithm updates the policy and any other learned components. During inference, the agent uses the trained policy to select actions, typically with its parameters held fixed. ![An agent may contain memory, a policy, a value function, and other components that improve learning or action selection.](images/agent-and-policy.svg) Let’s use standard RL notation to describe the agent’s interaction with the environment. Let $\pi_\theta$ represent a policy with learned parameters $\theta$. A **stochastic policy** samples an action from a probability distribution, so the same observation can lead to different actions. You can write the agent’s action selection as $$ a_t \sim \pi_\theta(\cdot \mid o_t), $$ where $o_t$ and $a_t$ are the observation and action at time $t$. The dot in $\pi_\theta(\cdot \mid o_t)$ leaves the action unspecified: given $o_t$, this expression represents the probability distribution over possible actions. The symbol $\sim$ means that $a_t$ is sampled from that distribution. During training, sampling enables **exploration**. The action with the highest probability is only the policy’s current best guess, based on the experience it has collected so far. If the agent always selected that action, it could repeat the same behavior and never discover alternatives that produce greater rewards. Sampling occasionally selects lower-probability actions, giving the agent evidence it can use to improve the policy. During evaluation or deployment, it is common to instead select the most likely action when you want consistent behavior rather than exploration. An RL problem is commonly modeled as a **Markov decision process** (MDP), and you will see references to MDPs in the following modules. In an MDP, the current state contains all information needed to predict the next-state distribution after an action. A robot policy that receives incomplete or noisy observations instead faces a **partially observable** problem (a partially observable Markov decision process, POMDP). It may need a history of observations, memory, or additional sensors to infer information that is hidden at a single step. This repeated dependence on earlier actions makes RL a **sequential decision problem**. Moving the gripper now changes the next observation, which changes the choices available later. The policy therefore needs to learn actions that support the entire task, not just the next instant. ## Episodes, Transitions, and Rollouts An action produces a **transition**. You can represent a transition as the following tuple: $$ (o_t, a_t, r_{t+1}, o_{t+1}, \text{terminated}_{t+1}, \text{truncated}_{t+1}). $$ The transition records what the policy observed, what it did, what reward followed, what it observed next, and why the interaction continued or ended. A sequence of transitions from a reset to a termination or truncation forms an **episode**. A **rollout** is the batch of recent transitions collected for updating the policy. It is not necessarily the same as one episode. A rollout can contain part of a long episode, several short episodes, or an episode boundary followed by data from a reset. When an environment ends early, it can reset and continue contributing transitions to the same rollout. Isaac Lab runs many independent copies of an environment in parallel. If $N$ environments each collect $T$ steps, the rollout contains $N \times T$ transitions. For example, 1,024 environments collecting 32 steps produce 32,768 transitions that are used to update the policy. The environments do not need to reach an episode boundary at the same time; each one resets when its own task ends. Parallel collection gives the learner a large, varied batch of recent experience. ![Three parallel environment timelines collect four steps each. Some episodes terminate and reset inside the shared rollout window, producing twelve transitions in total.](images/rl-transitions-rollouts.svg) ## Rewards and Behaviors A **reward** scores the outcome of one transition. A **return** combines rewards across time. Starting at time $t$, the *discounted* return is $$ G_t = r_{t+1} + \gamma r_{t+2} + \gamma^2 r_{t+3} + \cdots, $$ where the discount factor $\gamma$ is between 0 and 1. A smaller $\gamma$ emphasizes immediate rewards. A value close to 1 gives more weight to delayed outcomes, such as completing a placement after many control steps. The learning objective is to find policy parameters that maximize expected return: $$ \theta^* = \underset{\theta}{\operatorname{arg\,max}}\; \mathbb{E}_{\pi_\theta}\left[G_0\right]. $$ We use the expectation because initial conditions, environment dynamics, and the policy itself can be stochastic. A useful policy should perform well across many attempts, not only one favorable trajectory. For the gear-assembly task, a success reward alone may be too **sparse**: an untrained policy is unlikely to seat and mesh the gear by chance. You can add intermediate terms for alignment, tooth engagement, or seating depth. This is **reward shaping**. For example, a simplified reward might combine terms like these: $$ r = w_{\text{align}}r_{\text{align}} + w_{\text{mesh}}r_{\text{mesh}} + w_{\text{seat}}r_{\text{seat}} - w_{\text{effort}}c_{\text{effort}}, $$ where the weights determine how strongly each behavior affects the total. Those weights change the problem the policy actually solves. If hovering near the shaft earns more return than meshing and seating the gear, the policy may learn to hover. If the action penalty is too large, the safest way to preserve return may be to barely move. #### IMPORTANT A reward is a training signal, not proof that the task succeeded. You will always need to evaluate the finished policy with task-level criteria such as correct meshing, full seating, and the absence of damage. Report those results separately from training return. ## Value Functions and Advantage Estimation A return describes what followed one visited state. A **value function** describes what the policy expects on average. The *state-value* function (the policy’s expected return given the current state) is $$ V^\pi(s_t) = \mathbb{E}_\pi\left[G_t \mid s_t\right], $$ and the *action-value* function (the policy’s expected return given the current state and a specific action) is $$ Q^\pi(s_t, a_t) = \mathbb{E}_\pi\left[G_t \mid s_t, a_t\right]. $$ The difference between them is the **advantage**: $$ A^\pi(s_t, a_t) = Q^\pi(s_t, a_t) - V^\pi(s_t). $$ A positive advantage means the action led to a better outcome than the policy normally expects from that situation. A negative advantage means it led to a worse outcome. Policy-gradient methods use estimated advantages to decide which sampled actions should become more or less likely. The exact functions are unknown, so an **actor-critic** method (such as PPO described in the next lesson) learns a critic $V_\phi$. One way to measure whether a transition was better or worse than the critic expected is the **temporal-difference** (TD) residual $$ \delta_t = r_{t+1} + \gamma V_\phi(s_{t+1}) - V_\phi(s_t). $$ Here, $r_{t+1} + \gamma V_\phi(s_{t+1})$ is a **bootstrapped target** for $V_\phi(s_t)$: it combines the observed reward with the critic’s estimate of the return from the next state. The TD residual is the difference between this target and the critic’s current estimate, $V_\phi(s_t)$. Hats denote sample-based estimates, such as $\hat{G}_t$ and $\hat{A}_t$. ## Exploration, Training, and Inference ![Exploration tries alternative actions to discover better outcomes; exploitation repeats actions that currently appear best. Too much exploration can disrupt useful sequences, while too much exploitation can miss better alternatives.](images/exploration-vs-exploitation.svg) When training starts, the policy does not know which actions are useful. It needs **exploration** to encounter different outcomes. A stochastic policy supports exploration by sampling actions from a distribution. For continuous robot controls, that distribution might be centered on the policy’s predicted action and include adjustable noise. The opposite of exploration is **exploitation**: always selecting the action that currently appears best. Too little exploration can trap the policy in behavior it discovered early, a local optimum. Too much exploration can make its actions erratic and prevent it from repeating useful sequences. Exploration must also respect the action limits and safety constraints of the task. Simulation is valuable here because early mistakes can be frequent, resets can be automated, and many trials can run in parallel. During **training**, the learner alternates between collecting experience and changing the policy parameters. During **inference** or evaluation, the parameters stay fixed. An evaluation policy may choose the mean or most likely action instead of sampling, depending on the algorithm and evaluation protocol. Keeping the policy fixed allows you to measure its behavior without mixing evaluation with further learning. Return is useful for diagnosing training, but it is not enough for evaluation. A rising return can reflect genuine improvement, reward exploitation, or a change in episode length. Pair it with direct measures such as success rate, failure type, time to completion, and behavior across multiple initial conditions. ## The Learning Loop An actor-critic learning algorithm repeats four steps. The **actor** is the policy that selects actions, and the **critic** is the value function that provides value estimates: 1. **Collect:** Use the current actor to record a rollout of transitions from parallel environments. 2. **Estimate:** Use rollout rewards, episode endings, and critic value estimates to estimate returns and advantages. 3. **Update:** Update the actor using observations, actions, and estimated advantages so actions associated with better outcomes become more likely. Update the critic using observations and estimated returns as targets. 4. **Repeat:** Collect the next rollout with the updated actor and use the updated critic for the next round of value estimates. ![The actor collects a rollout from parallel environments. Rewards, episode endings, and critic value estimates support return and advantage estimation. Advantages guide actor updates, returns provide critic targets, and the updated actor and critic feed the next iteration.](images/rl-learning-loop.svg) A critic value estimate gives the learner a baseline for judging outcomes and calculating advantages. The following pseudocode shows this actor-critic loop without committing to a specific estimation or update method: ```python actor = initialize_policy_randomly() critic = initialize_value_function_randomly() for iteration in range(number_of_iterations): rollout = collect_parallel_experience(envs, actor) returns, advantages = estimate_returns_and_advantages(rollout, critic) update_actor(actor, rollout, advantages) update_critic(critic, rollout, returns) ``` Different actor-critic algorithms define `estimate_returns_and_advantages`, `update_actor`, and `update_critic` differently. The next page introduces Proximal Policy Optimization (PPO), which uses an actor, a critic, recent rollouts, and a clipped objective to keep each policy update from becoming too large. ## Review ### Check Your Understanding How many transitions do 1,024 environments collect in a 32-step rollout? ### Answer 32,768 transitions: 1,024 environments × 32 steps. A rollout can include both partial and complete episodes. ### Apply What You Learned How would you trace one transition through a rollout and policy update, including an episode ending? Identify the observation, action, reward, next observation, and ending signals, and explain when policy weights change. ### Agent Tutor Prompt Copy this prompt and include your answer: > Review my answer to this question: How would you trace one transition through a rollout and policy update, including an episode ending? Identify the observation, action, reward, next observation, and ending signals, and explain when policy weights change. > My answer: [paste your answer here]. > Check my reasoning about observation, action, reward, next observation, ending signals, rollout versus episode, and when weights change. Ask for missing context rather than assuming it. Identify what I explained well, point out one gap or misconception, and ask a follow-up question that helps me improve my answer. Let me revise it before supplying a complete answer. ### Key Takeaways - **RL learns from interaction.** A policy selects an action from its observation; the environment advances and returns a new observation, reward, and ending signals. - **Episodes and rollouts are different.** An episode ends when the task terminates or reaches a limit. A rollout is a batch of transitions collected for an update and may contain parts of several episodes. - **Rewards guide learning, while task metrics check behavior.** Higher return does not by itself establish that the robot completes the intended task. - **Value estimates help judge actions.** The critic predicts expected return; an advantage compares an action’s outcome with that expectation. - **Training repeats collect, estimate, and update.** During evaluation, the policy stays fixed so its behavior can be measured consistently.