Training and Playback#

We’ve seen every piece of the pipeline. Now let’s put it together: first run a full environment loop with random actions and collect statistics, then load a policy trained with PPO and watch the difference. Finally, we’ll cover the commands that train and play back policies from a terminal.

In this lesson, we will:

  • Run a full vectorized environment loop and interpret its reward statistics.

  • Contrast random behavior with a trained policy.

  • Load and play back a trained PPO checkpoint.

  • Train your own policy with the provided RSL-RL scripts.

Part 5: Putting It All Together#

We build the environment, then step it for 1000 steps with random actions across 32 parallel environments, tracking the mean reward and how many episodes reset.

import gymnasium as gym
import torch

from franka_cube.tasks.direct.franka_cube.franka_cube_env_cfg import FrankaCubeEnvCfg

env_cfg = FrankaCubeEnvCfg()
env_cfg.scene.num_envs = 32
env = gym.make("Template-Franka-Cube-Direct-v0", cfg=env_cfg)
obs, info = env.reset()

total_rewards = []
num_resets = 0

for i in range(1000):
    with torch.inference_mode():
        actions = 2 * torch.rand(
            (env.unwrapped.num_envs, env.unwrapped.cfg.action_space),
            device=env.unwrapped.device,
        ) - 1
        obs, reward, terminated, truncated, info = env.step(actions)
        total_rewards.append(reward.mean().item())
        num_resets += (terminated | truncated).sum().item()

avg_reward = sum(total_rewards) / len(total_rewards)
print(f"Full environment loop: 1000 steps x {env.unwrapped.num_envs} envs")
print(f"  Mean reward per step: {avg_reward:.4f}")
print(f"  Total environment resets: {int(num_resets)}")

With random actions the robot flails and the cube falls off frequently, so the mean reward stays low and dominated by the always-on reaching term. A trained RL policy, by contrast, learns to coordinate seven arm actions and one coupled gripper action to reach, grasp, and lift.

Part 6: Playing a Trained Policy#

So far we’ve used only random actions. Now let’s load a policy trained with PPO (Proximal Policy Optimization) and see what a trained agent looks like. The policy is a small MLP (256 → 128 → 64) that maps the 41-dimensional observation to 8-dimensional actions. It was trained with RSL-RL on the reach-grasp-lift reward from the previous lesson.

Use a checkpoint produced by the training workflow under logs/rsl_rl/franka_cube/. Loading it involves wrapping the environment for RSL-RL and restoring the actor weights:

import importlib.metadata as metadata
import os

from rsl_rl.runners import OnPolicyRunner

from isaaclab_rl.rsl_rl import RslRlVecEnvWrapper, handle_deprecated_rsl_rl_cfg
from isaaclab_tasks.utils import get_checkpoint_path
from franka_cube.tasks.direct.franka_cube.agents.rsl_rl_ppo_cfg import PPORunnerCfg
from franka_cube.tasks.direct.franka_cube.franka_cube_env_cfg import FrankaCubeEnvCfg

env_cfg = FrankaCubeEnvCfg()
env_cfg.scene.num_envs = 32
env = gym.make("Template-Franka-Cube-Direct-v0", cfg=env_cfg)

installed_version = metadata.version("rsl-rl-lib")
agent_cfg = handle_deprecated_rsl_rl_cfg(PPORunnerCfg(), installed_version)
agent_cfg.device = str(env.unwrapped.device)

env_wrapped = RslRlVecEnvWrapper(env, clip_actions=agent_cfg.clip_actions)
runner = OnPolicyRunner(env_wrapped, agent_cfg.to_dict(), log_dir=None, device=agent_cfg.device)
log_root_path = os.path.abspath(os.path.join("logs", "rsl_rl", agent_cfg.experiment_name))
checkpoint_path = get_checkpoint_path(log_root_path, agent_cfg.load_run, agent_cfg.load_checkpoint)
runner.load(checkpoint_path)
policy = runner.get_inference_policy(device=env.unwrapped.device)

Note

The checkpoint must be produced by the current RSL-RL training script. The runner and checkpoint schema must match; legacy RSL-RL 3.x checkpoints are not loaded by this canonical RSL-RL 5.x workflow.

With the policy loaded, the playback loop feeds observations to the policy instead of sampling random actions:

obs = env_wrapped.get_observations()

for i in range(1000):
    with torch.inference_mode():
        actions = policy(obs)
        obs, _, _, _ = env_wrapped.step(actions)

print("The robot should reach for the cube, close around it, and lift it.")
print("Compare this to the random flailing from earlier!")

Watching the trained policy after the random baseline is the payoff of the whole module: the same environment, the same physics, but a policy that has learned to coordinate the arm and gripper into a clean reach-grasp-lift.

Running Training and Playback From a Shell#

The canonical, runnable path uses the provided RSL-RL scripts from a terminal. The commands differ depending on where you run them: on the Isaac Launchable, call Isaac Lab’s Python through its wrapper (isaaclab.sh -p) and stream the UI with --livestream 2; locally, use the python interpreter that has Isaac Lab installed.

Training#

First, train your own policy from scratch. The provided configuration runs for 3000 iterations and saves a checkpoint every 50 iterations. Training is compute-intensive and takes time depending on your GPU.

Train headless for maximum throughput:

cd /workspace/franka_cube
/workspace/isaaclab/isaaclab.sh -p scripts/rsl_rl/train.py \
  --task=Template-Franka-Cube-Direct-v0 \
  --num_envs=4096 \
  --viz none

cd franka_cube
python scripts/rsl_rl/train.py \
  --task=Template-Franka-Cube-Direct-v0 \
  --num_envs=4096 \
  --viz none

To watch a smaller training run:

cd /workspace/franka_cube
/workspace/isaaclab/isaaclab.sh -p scripts/rsl_rl/train.py \
  --task=Template-Franka-Cube-Direct-v0 \
  --num_envs=32 \
  --livestream 2 --viz kit

cd franka_cube
python scripts/rsl_rl/train.py \
  --task=Template-Franka-Cube-Direct-v0 \
  --num_envs=32 \
  --viz kit

Playback#

Once training finishes, play the policy back with the UI. The commands below pass the bundled checkpoint/model_399.pt explicitly, so play.py runs that final checkpoint instead of selecting a checkpoint automatically.

cd /workspace/franka_cube
/workspace/isaaclab/isaaclab.sh -p scripts/rsl_rl/play.py \
  --task=Template-Franka-Cube-Direct-v0 \
  --num_envs=16 \
  --livestream 2 --viz kit \
  --checkpoint /workspace/franka_cube/checkpoint/model_399.pt

cd franka_cube
python scripts/rsl_rl/play.py \
  --task=Template-Franka-Cube-Direct-v0 \
  --num_envs=16 \
  --viz kit \
  --checkpoint checkpoint/model_399.pt

Tip

Train headless with a large --num_envs (for example 4096) for throughput, and only attach the viewer (--viz kit, plus --livestream 2 on the Isaac Launchable) when you play a policy back. Rendering every environment during training wastes GPU cycles you want spent on learning.

The final pretrained checkpoint is bundled at franka_cube/checkpoint/model_399.pt. Checkpoints from your own training runs are written under franka_cube/logs/rsl_rl/franka_cube/. Pass --load_run or --checkpoint to select a specific policy.

Key Takeaways#

You ran a full vectorized environment loop, measured random-action performance, and then loaded a trained PPO policy that reaches, grasps, and lifts the cube, a stark contrast to random flailing. You also learned the shell commands to train and play back policies. You’ve now gone from raw Newton concepts all the way to a trained robot-learning policy running on Newton physics inside Isaac Lab.

Go Further With Reinforcement Learning#

This module focused on running Newton physics inside an Isaac Lab RL task. To build deeper intuition for reinforcement learning itself, from the Markov decision process and reward design to training and evaluating policies, work through these companion courses: