# Cartpole Training ![Sixteen trained cartpoles recover from randomized starts and balance their poles.](images/cartpole-trained-policy.mp4) In the previous lesson, you verified task registration and observed zero-action and random-action behavior. Now let’s use the same task to train a policy with Proximal Policy Optimization (PPO), a popular reinforcement learning algorithm, inspect its training metrics, and compare saved checkpoints with those baselines. You’ll start with a short training session, then run training with the generated project’s default settings. You’ll use the plots and recordings together to see what improved and how well the policy performs under the conditions you tested. #### NOTE Before you begin, complete the baseline checks in [The Cartpole Example](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/2026-10.0/isaac-lab-basics/06-isaac-lab-cartpole.md). Run every command in this lesson from the same generated `isaac-lab-workspace/isaac_lab_tutorial` root. Keep the task and PPO configuration unchanged so you can follow the comparisons below. ## Choose a Starting Environment Count Start with **64 environments** for the short run below. The generated default run uses **4,096**, which you can try if your GPU has enough memory and you have time for a longer training session. Training needs room in GPU memory for rollouts, networks, and optimizer state alongside the simulation. If you run out of GPU memory, lower `--num_envs` and try again. ### Optional: Compare Runtime Throughput Before a Larger Run Use the [Isaac Lab benchmarking guide](https://isaac-sim.github.io/IsaacLab/release/3.0.0/source/developer-tools/benchmarking/run_benchmarks.html) if you want to tune the environment count. Compare candidate counts with the same task, backend, seed, and measurement settings. Look for a count that gives good overall throughput while leaving memory for training. Try a short training run with that count before committing to a longer run. A runtime benchmark measures environment stepping. It excludes the policy updates and rollout storage needed during training, so use its fastest count as a starting point rather than a final choice. With 16 steps per rollout, 64 environments collect 1,024 transitions per PPO iteration; 4,096 collect 65,536. This changes the amount of training experience, so compare runs by environment transitions as well as iterations. ## Train a Policy We’ll start with a short PPO training session and use `--visualizer none` to avoid the extra work of displaying a live simulation view while the policy trains: 1. Try a short session to check that training works. ```bash uv run isaaclab train \ --rl_library rsl_rl \ --task IsaacLabTutorial-Balance-Cartpole \ --num_envs 64 \ --max_iterations 50 \ --seed 42 \ --run_name cartpole_short \ --visualizer none ``` Here’s what the options do: - `--rl_library rsl_rl` selects RSL-RL to perform PPO updates. - `--task` selects the registered cartpole environment. - `--num_envs 64` and `--max_iterations 50` set the short run’s size/duration. - `--seed 42` sets the random seed. - `--run_name cartpole_short` labels the saved run. - `--visualizer none` trains without drawing a live viewer. The training log’s ETA gives you an idea of how long the run will take on your machine. This command uses `PPORunnerCfg` from `rsl_rl_ppo_cfg.py`. At each iteration, the policy collects a rollout from the environments, computes returns and advantages, and updates the actor (the policy) and critic (the value estimator) from mini-batches of that experience. This short run collects 51,200 environment transitions: 64 environments × 16 steps per rollout × 50 iterations. It gives you an initial training run and saved checkpoints to inspect. To explore the training settings, look for the configuration file `logs/.../params/agent.yaml` using `find`. 2. Next, use the defaults for a longer run with more parallel environments: ```bash uv run isaaclab train \ --rl_library rsl_rl \ --task IsaacLabTutorial-Balance-Cartpole \ --seed 42 \ --run_name cartpole_default \ --visualizer none ``` The two runs use the same task and seed, but use different numbers of parallel environments and amounts of training experience. Here’s a summary of the reference runs to help you compare them: | Setting | Short run | Full run | |-------------------------|------------------|--------------------| | Run name | `cartpole_short` | `cartpole_default` | | Parallel environments | 64 | 4,096 | | PPO iterations | 50 | 150 | | Steps per rollout | 16 | 16 | | Environment transitions | 51,200 | 9,830,400 | ## Analyze the Results Let’s look at the results from three angles: - **Terminal metrics:** inspect the final reward terms and episode endings. - **Learning curves:** compare progress over the full training run in TensorBoard. - **Policy playback:** watch whether the saved policy actually balances the pole. ### Inspect the Terminal Metrics Your terminal prints progress as training runs. Here’s the final output from the reference run, which you can compare with your own: ```text Learning iteration 149/150 Run name: cartpole_default Total steps: 9830400 Steps per second: 824669 Collection time: 0.038s Learning time: 0.041s Mean value loss: 0.0000 Mean surrogate loss: 0.0047 Mean entropy loss: -1.1233 Mean reward: 4.94 Mean episode length: 300.00 Mean action std: 0.08 Episode_Reward/alive: 1.0000 Episode_Reward/terminating: 0.0000 Episode_Reward/pole_pos: -0.0074 Episode_Reward/cart_vel: -0.0032 Episode_Reward/pole_vel: -0.0014 Episode_Termination/time_out: 1.0000 Episode_Termination/cart_out_of_bounds: 0.0000 -------------------------------------------------------------------------------- Iteration time: 0.08s Time elapsed: 0:00:12 ETA: 0:00:00 Training time: 12.67 seconds ``` Start with these clues when reading the output: * The final logged metrics are consistent with episodes reaching the 300-step time limit: mean episode length is 300, the timeout rate is 1.0, and the cart-out-of-bounds rate is 0.0. * The mean reward of 4.94 is close to the maximum implied by the logged terms: the policy receives the full alive reward while keeping the pole-angle, cart-velocity, and pole-velocity penalties small. * Together, these values are consistent with a policy that balances the pole for the full five-second episode rather than merely avoiding an early termination. * The optimization statistics also look stable at the end of this run. The value loss is near zero, the surrogate loss is small, and the action standard deviation has fallen to 0.08. These values describe the final PPO updates; to judge balancing, also check the task metrics and watch the policy. ### Inspect the Learning Curves RSL-RL writes the resolved configuration, event data, and checkpoints below `logs/rsl_rl/`. 1. Find your run directories and saved checkpoints: ```bash find logs/rsl_rl -maxdepth 3 -type d -print find logs/rsl_rl -name 'model_*.pt' -print ``` 2. Open the training curves in [TensorBoard](https://www.tensorflow.org/tensorboard). Start TensorBoard in the project root with the log directory as its data source: ```bash uv run tensorboard --logdir logs/rsl_rl ``` If TensorBoard is running on a remote machine, you can forward the port to your local machine. From a terminal on your local machine, forward the port listed by the `tensorboard` command over SSH: ```bash ssh -N -L PORT:localhost:PORT USER@REMOTE_HOST ``` Replace `PORT` with the port number printed by the `tensorboard` command, and replace `USER@REMOTE_HOST` with your SSH user and the hostname or IP address of the remote machine. Keep the SSH session open, then visit the link given by the `tensorboard` command in your local browser. In the next section we’ll show you how to compare runs with Weights & Biases, but first let’s look at the TensorBoard metrics. Here are some suggestions for what to look for in TensorBoard: 1. Check `Train/mean_episode_length` and `Train/mean_reward`. The mean reward should increase and the mean episode length should approach the maximum of 300 steps. 2. Check `Episode_Termination/cart_out_of_bounds` and `Episode_Termination/time_out`. The cart-out-of-bounds rate should decrease to zero, and the timeout rate should increase to one. 3. Check `Episode_Reward/cart_vel` and `Episode_Reward/pole_vel`. These values should trend toward zero, indicating that the policy is keeping the cart and pole stable. You can also use the TensorBoard logs to make your own plots later. For now, compare your curves with the reference figure below. It shows reward and episode length for both runs, with environment transitions on a logarithmic axis. ![Reference TensorBoard reward and episode-length curves for the short and default PPO runs, plotted against environment transitions on a logarithmic axis.](images/cartpole-training-results.png) ## Compare Runs With Weights & Biases If you’d like interactive charts and an easy way to compare experiments, try [Weights & Biases (W&B)](https://wandb.ai/). Isaac Lab includes support for a W&B logger, so you can record metrics and the resolved configuration without writing a plotting script. The generated project includes W&B as an optional dependency. If you have a W&B account, enable this extra and authenticate on the machine where you train. Run this command from the generated project root: ```bash uv run --extra wandb --with wandb==0.26.1 wandb login ``` Include `--extra wandb` in each W&B command below so `uv` installs the project’s optional W&B dependencies. The additional `--with wandb==0.26.1` constraint keeps the `wandb.util.generate_id` API used by this Isaac Lab logger; newer W&B releases remove that API. Follow the prompt to enter your W&B API key. If you use a team project, set `WANDB_USERNAME` to the team entity name before training; the RSL-RL logger uses this variable to choose the W&B entity. To try W&B, use the two training commands below instead of the earlier ones. If you’ve already finished those runs, you can keep using TensorBoard or start new runs with W&B. These commands won’t import your existing TensorBoard logs. ```bash uv run --extra wandb --with wandb==0.26.1 isaaclab train \ --rl_library rsl_rl \ --task IsaacLabTutorial-Balance-Cartpole \ --num_envs 64 \ --max_iterations 50 \ --seed 42 \ --run_name cartpole_short \ --logger wandb \ --log_project_name cartpole-training \ --visualizer none uv run --extra wandb --with wandb==0.26.1 isaaclab train \ --rl_library rsl_rl \ --task IsaacLabTutorial-Balance-Cartpole \ --seed 42 \ --run_name cartpole_default \ --logger wandb \ --log_project_name cartpole-training \ --visualizer none ``` Open the project link printed by W&B in your browser. Select the two runs in the `cartpole-training` project and compare `Train/mean_reward`, `Train/mean_episode_length`, and the reward and termination metrics listed above. The run names include a timestamp followed by the `--run_name` suffix. The W&B logger also writes TensorBoard event files and saves checkpoints under `logs/rsl_rl/`, so the TensorBoard and playback commands still apply. The two runs collect different amounts of experience per iteration: 1,024 transitions for the short run and 65,536 for the default run. Keep that difference in mind when comparing curves. The reference figure above instead plots both runs against environment transitions, with episode length in simulation steps. If the training machine cannot connect to W&B, prefix each training command with `WANDB_MODE=offline`. W&B prints the local run directory; upload it later from a machine with network access using: ```bash uv run --extra wandb --with wandb==0.26.1 wandb sync PATH_TO_OFFLINE_RUN ``` Replace `PATH_TO_OFFLINE_RUN` with the directory printed by W&B. ## Compare the Saved Policies Now for the most important analysis: watching what your policies have learned. The `isaaclab play` command can run a saved policy and record a video. The recording command uses Newton GL. RTX is optional and depends on compatible ovrtx plugins. ```bash CARTPOLE_CHECKPOINT=$(find logs/rsl_rl \ -path '*cartpole_short/model_49.pt' -print -quit) uv run --with 'moviepy<2' \ isaaclab play \ --rl_library rsl_rl \ --task IsaacLabTutorial-Balance-Cartpole \ --checkpoint "$CARTPOLE_CHECKPOINT" \ --num_envs 16 \ --seed 123 \ --visualizer newton \ --video --video_length 900 \ env.sim.default_visualizer_cfg.eye='[14.0,-18.0,13.0]' \ env.sim.default_visualizer_cfg.lookat='[0.0,0.0,0.8]' \ env.sim.default_visualizer_cfg.focal_length=24.0 ``` The shell variable `CARTPOLE_CHECKPOINT` holds the checkpoint path, so you can reuse the playback command by changing that path. The playback command loads the policy without further training and records 900 control steps (15 seconds). By default, the video is written to a `videos` subdirectory of the checkpoint’s parent directory. You can find the recording with: ```bash find "$(dirname "$CARTPOLE_CHECKPOINT")/videos" -type f -name '*.mp4' ``` ![Recorded inference from the 50-iteration PPO checkpoint; poles are mostly upright but carts can drift toward the track bounds.](images/cartpole-short-policy.mp4) Now locate and record the final checkpoint from the default run: ```bash CARTPOLE_CHECKPOINT=$(find logs/rsl_rl \ -path '*cartpole_default/model_149.pt' -print -quit) # This is the same command as above, but with the new checkpoint path. # You can use your shell history to repeat the command. uv run --with 'moviepy<2' \ isaaclab play \ --rl_library rsl_rl \ --task IsaacLabTutorial-Balance-Cartpole \ --checkpoint "$CARTPOLE_CHECKPOINT" \ --num_envs 16 \ --seed 123 \ --visualizer newton \ --video --video_length 900 \ env.sim.default_visualizer_cfg.eye='[14.0,-18.0,13.0]' \ env.sim.default_visualizer_cfg.lookat='[0.0,0.0,0.8]' \ env.sim.default_visualizer_cfg.focal_length=24.0 find "$(dirname "$CARTPOLE_CHECKPOINT")/videos" -type f -name '*.mp4' ``` Using the same environment count, seed, viewer, and recording length makes the two checkpoint videos directly comparable. ![Recorded inference from the default 150-iteration PPO checkpoint; sixteen cartpoles recover from randomized resets and balance.](images/cartpole-trained-policy.mp4) Compare your recordings with the [zero-action and random-action baselines](https://docs.nvidia.com/learning/physical-ai/getting-started-with-isaac-lab/2026-10.0/isaac-lab-basics/06-isaac-lab-cartpole.md#recording-a-video) from the previous lesson. In the reference recordings shown here, the short checkpoint usually keeps the poles near upright but can let carts drift toward the track boundary. The default checkpoint recovers more consistently from randomized resets and keeps both pole angle and cart position controlled. Together with the reward and episode-length curves, the recordings support learning progress for this seed. To find out how consistently the policy performs, you’d need to try additional training seeds or changes to the dynamics. ## Review ### Check Your Understanding Does one successful playback recording prove that a policy is reliable? ### Answer No. One recording shows a limited set of attempts under particular conditions. Additional complete-episode evaluations and seeds help assess consistency. ### Apply What You Learned How would you use your training metrics and checkpoint playback to judge learning progress and choose a further reliability check? Compare the trained policies with zero-action and random-action baselines, and describe what another seed or evaluation could tell you. ### Agent Tutor Prompt Copy this prompt and include your answer: > Review my answer to this question: How would you use your training metrics and checkpoint playback to judge learning progress and choose a further reliability check? Compare the trained policies with zero-action and random-action baselines, and describe what another seed or evaluation could tell you. > My answer: [paste your answer here]. > Check my reasoning about baselines, pole angle, cart drift, episode endings, training experience, and additional seeds. Ask for missing context rather than assuming it. Identify what I explained well, point out one gap or misconception, and ask a follow-up question that helps me improve my answer. Let me revise it before supplying a complete answer. ### Key Takeaways - Start with a short run, then try a longer one once training works. - Environment count changes both workload and experience collected per iteration. - Use metrics and playback together, comparing them with your baselines to understand what the policy learned. ### Resources - [Isaac Lab Reinforcement Learning](https://isaac-sim.github.io/IsaacLab/release/3.0.0/source/concepts/reinforcement_learning.html) — Documentation on supported reinforcement learning libraries and workflows for training and evaluating policies. - [RSL-RL](https://leggedrobotics.github.io/rsl_rl/) — An overview of the GPU-based reinforcement learning library used for robot policy training. - [W&B Login](https://docs.wandb.ai/models/ref/cli/wandb-login) — Authenticate the training machine with W&B. - [W&B Sync](https://docs.wandb.ai/models/ref/cli/wandb-sync) — Upload runs recorded in offline mode.