> This page is for version Release 3.0 (default).
> For other versions, use one of these documentation indexes:
> - Release 3.0 (default): https://docs.nvidia.com/isaaclab/3.0/llms.txt
> - Release 2.3: https://docs.nvidia.com/isaaclab/2.3/llms.txt
> - Release 2.2: https://docs.nvidia.com/isaaclab/2.2/llms.txt
> - Release 2.1: https://docs.nvidia.com/isaaclab/2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/isaaclab/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/isaaclab/_mcp/server.

# Training with an RL Agent

In the previous tutorials, we covered how to define an RL task environment, register
it into the `gym` registry, and interact with it using a random agent. We now move
on to the next step: training an RL agent to solve the task.

Although the `envs.ManagerBasedRLEnv` conforms to the `gymnasium.Env` interface,
it is not exactly a `gym` environment. The input and outputs of the environment are
not numpy arrays, but rather based on torch tensors with the first dimension being the
number of environment instances.

Additionally, most RL libraries expect their own variation of an environment interface.
For example, [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/) expects the environment to conform to its
[VecEnv API](https://stable-baselines3.readthedocs.io/en/master/guide/vec_envs.html#vecenv-api-vs-gym-api) which expects a list of numpy arrays instead of a single tensor. Similarly,
[RSL-RL](https://github.com/leggedrobotics/rsl_rl), [RL-Games](https://github.com/Denys88/rl_games) and [SKRL](https://skrl.readthedocs.io) expect a different interface. Since there is no one-size-fits-all
solution, we do not base the `envs.ManagerBasedRLEnv` on any particular learning library.
Instead, we implement wrappers to convert the environment into the expected interface.
These are specified in the `isaaclab_rl` module.

In this tutorial, we will use [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/) to train an RL agent to solve the
cartpole balancing task.

> **Warning**
>
> Wrapping the environment with the respective learning framework's wrapper should happen in the end,
> i.e. after all other wrappers have been applied. This is because the learning framework's wrapper
> modifies the interpretation of environment's APIs which may no longer be compatible with `gymnasium.Env`.

## The Code

For this tutorial, we use the training implementation from [Stable-Baselines3](https://stable-baselines3.readthedocs.io/en/master/) workflow in the
`isaaclab_rl.entrypoints.backends.train_sb3` module.

```python
# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause

"""Stable-Baselines3 training backend of the unified reinforcement learning entrypoint."""

from __future__ import annotations

import argparse
import contextlib
import gc
import os
import signal
import sys
import time
from datetime import datetime
from pathlib import Path

import numpy as np
from stable_baselines3 import PPO
from stable_baselines3.common.callbacks import CheckpointCallback, LogEveryNTimesteps
from stable_baselines3.common.vec_env import VecNormalize

from isaaclab.app import add_launcher_args, launch_simulation, report_activity
from isaaclab.envs import DirectMARLEnvCfg
from isaaclab.utils.seed import configure_seed

import isaaclab_tasks  # noqa: F401
from isaaclab_tasks.utils import resolve_task_config, setup_preset_cli

from ...sb3 import Sb3VecEnvWrapper, process_sb3_cfg
from ..common import (
    CHECKPOINT_SELECTORS,
    add_common_train_args,
    apply_env_overrides,
    apply_video_recording,
    close_env,
    create_isaaclab_env,
    dump_train_configs,
    enable_cameras_for_video,
    pre_launch_video_config,
    resolve_checkpoint_selector,
    resolve_seed,
    set_hydra_args,
    show_run_summary,
    startup_screen,
    wrap_sensor_capture,
    write_run_manifest,
)

# PLACEHOLDER: Extension template (do not remove this comment)
with contextlib.suppress(ImportError):
    import isaaclab_tasks_experimental  # noqa: F401

def _cleanup_pbar(*args) -> None:
    """Stop training and close the rich progress bars on Ctrl+C."""
    for obj in gc.get_objects():
        if "tqdm_rich" in type(obj).__name__:
            obj.close()
    raise KeyboardInterrupt

def _parse_args(argv: list[str]) -> argparse.Namespace:
    """Parse Stable-Baselines3 training arguments."""
    parser = argparse.ArgumentParser(description="Train an RL agent with Stable-Baselines3.")
    add_common_train_args(
        parser,
        agent_default="sb3_cfg_entry_point",
        agent_help="Name of the RL agent configuration entry point.",
        include_distributed=False,
    )
    parser.add_argument("--log_interval", type=int, default=100_000, help="Log data every n timesteps.")
    parser.add_argument("--checkpoint", type=str, default=None, help="Checkpoint path, or latest/best.")
    parser.add_argument(
        "--keep_all_info",
        action="store_true",
        default=False,
        help="Use a slower SB3 wrapper but keep all the extra training info.",
    )
    add_launcher_args(parser)
    args_cli, hydra_args = setup_preset_cli(parser, argv)
    enable_cameras_for_video(args_cli)
    set_hydra_args(hydra_args)
    return args_cli

def _resolve_checkpoint(args_cli: argparse.Namespace, log_root_path: str) -> str | None:
    """Resolve the checkpoint to resume from, or None when training starts from scratch."""
    if args_cli.checkpoint in CHECKPOINT_SELECTORS:
        return resolve_checkpoint_selector(
            log_root_path,
            args_cli.checkpoint,
            library="sb3",
            task=args_cli.task,
            checkpoint_pattern=r"model(?:_.*)?\.zip",
            preferred_checkpoint_pattern=r"model\.zip",
            metadata={"agent": args_cli.agent},
        )
    return args_cli.checkpoint

def run(argv: list[str]) -> None:
    """Train a Stable-Baselines3 agent."""
    signal.signal(signal.SIGINT, _cleanup_pbar)
    args_cli = _parse_args(argv)
    with startup_screen(args_cli, num_stages=3) as screen:
        env_cfg, agent_cfg = resolve_task_config(args_cli.task, args_cli.agent)
        pre_launch_video_config(env_cfg, args_cli)
        show_run_summary(screen, args_cli, env_cfg, library="sb3", action="train")
        screen.stage("Launching simulation")
        with launch_simulation(env_cfg, args_cli), contextlib.ExitStack() as cleanup:
            apply_env_overrides(args_cli, env_cfg)
            args_cli.seed = resolve_seed(args_cli.seed)
            if args_cli.seed is not None:
                agent_cfg["seed"] = args_cli.seed
            if args_cli.max_iterations is not None:
                agent_cfg["n_timesteps"] = args_cli.max_iterations * agent_cfg["n_steps"] * env_cfg.scene.num_envs
            env_cfg.seed = agent_cfg["seed"]

            log_root_path = os.path.abspath(os.path.join("logs", "sb3", args_cli.task))
            print(f"[INFO] Logging experiment in directory: {log_root_path}")
            run_name = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
            print(f"Exact experiment name requested from command line: {run_name}")
            log_dir = os.path.join(log_root_path, run_name)
            write_run_manifest(log_dir, library="sb3", task=args_cli.task, metadata={"agent": args_cli.agent})
            dump_train_configs(log_dir, env_cfg, agent_cfg)
            (Path(log_dir) / "command.txt").write_text(" ".join(sys.orig_argv))

            checkpoint_path = _resolve_checkpoint(args_cli, log_root_path)
            agent_cfg = process_sb3_cfg(agent_cfg, env_cfg.scene.num_envs)
            policy_arch = agent_cfg.pop("policy")
            n_timesteps = agent_cfg.pop("n_timesteps")
            norm_args = {}
            for key in ("normalize_input", "normalize_value", "clip_obs"):
                if key in agent_cfg:
                    norm_args[key] = agent_cfg.pop(key)

            env_cfg.log_dir = log_dir
            apply_video_recording(env_cfg, log_dir, args_cli)

            screen.stage("Creating environment")
            env = create_isaaclab_env(
                args_cli.task,
                env_cfg,
                args_cli,
                convert_marl_to_single_agent=isinstance(env_cfg, DirectMARLEnvCfg),
            )
            cleanup.callback(lambda: close_env(env))
            env = wrap_sensor_capture(env, log_dir, args_cli)

            screen.stage("Preparing agent")
            start_time = time.time()
            report_activity("Wrapping environment")
            env = Sb3VecEnvWrapper(env, fast_variant=not args_cli.keep_all_info)
            report_activity(None)
            if norm_args.get("normalize_input"):
                print(f"Normalizing input, {norm_args=}")
                env = VecNormalize(
                    env,
                    training=True,
                    norm_obs=norm_args["normalize_input"],
                    norm_reward=norm_args.get("normalize_value", False),
                    clip_obs=norm_args.get("clip_obs", 100.0),
                    gamma=agent_cfg["gamma"],
                    clip_reward=np.inf,
                )

            report_activity("Building policy")
            agent = PPO(policy_arch, env, verbose=1, tensorboard_log=log_dir, **agent_cfg)
            report_activity(None)
            if checkpoint_path is not None:
                agent = agent.load(checkpoint_path, env, print_system_info=True)

            # configure_seed must run after PPO construction and loading so torch determinism does not disturb
            # SB3's initialization
            if args_cli.deterministic:
                configure_seed(env_cfg.seed, torch_deterministic=True)

            callbacks = [
                CheckpointCallback(save_freq=1000, save_path=log_dir, name_prefix="model", verbose=2),
                LogEveryNTimesteps(n_steps=args_cli.log_interval),
            ]

            screen.close()
            with contextlib.suppress(KeyboardInterrupt):
                agent.learn(total_timesteps=n_timesteps, callback=callbacks, progress_bar=True, log_interval=None)

            agent.save(os.path.join(log_dir, "model"))
            print(f"Saving to:\n{os.path.join(log_dir, 'model.zip')}")
            if isinstance(env, VecNormalize):
                print("Saving normalization")
                env.save(os.path.join(log_dir, "model_vecnormalize.pkl"))
            print(f"Training time: {round(time.time() - start_time, 2)} seconds")
```

## The Code Explained

Most of the code above is boilerplate code to create logging directories, saving the parsed configurations,
and setting up different Stable-Baselines3 components. For this tutorial, the important part is creating
the environment and wrapping it with the Stable-Baselines3 wrapper.

There are three wrappers used in the code above:

1. `gymnasium.wrappers.RecordVideo`: This wrapper records a video of the environment
   and saves it to the specified directory. This is useful for visualizing the agent's behavior
   during training.
2. `wrappers.sb3.Sb3VecEnvWrapper`: This wrapper converts the environment
   into a Stable-Baselines3 compatible environment.
3. [stable_baselines3.common.vec_env.VecNormalize](https://stable-baselines3.readthedocs.io/en/master/guide/vec_envs.html#vecnormalize): This wrapper normalizes the
   environment's observations and rewards.

Each of these wrappers wrap around the previous wrapper by following `env = wrapper(env, *args, **kwargs)`
repeatedly. The final environment is then used to train the agent. For more information on how these
wrappers work, please refer to the [Wrapping environments](/source/how-to/wrap_rl_env) documentation.

## The Code Execution

We train a PPO agent from Stable-Baselines3 to solve the cartpole balancing task.

### Training the agent

There are three main ways to train the agent. Each of them has their own advantages and disadvantages.
It is up to you to decide which one you prefer based on your use case.

#### Headless execution

When no visualizer is requested, no interactive visualizer window is opened during training. This is useful
when training on a remote server or when you do not need live visual feedback, which can add some compute cost.
Rendering can still be active for sensor/camera data capture when enabled by the workflow.

#### uv (Recommended)

```bash
uv run --extra sb3 isaaclab train --rl_library sb3 --task Isaac-Cartpole --num_envs 64
```

#### isaaclab.sh / isaaclab.bat

```bash
./isaaclab.sh train --rl_library sb3 --task Isaac-Cartpole --num_envs 64
```

#### Headless execution with off-screen render

To inspect behavior from a training run without an interactive window, pass `--video`.
By default, it records the scene from a headless Kit visualizer, which requires Isaac Sim. See
[/source/how-to/record_video](/source/how-to/record_video) to choose a different visualizer or a camera-sensor output.

#### uv (Recommended)

```bash
uv run --extra sb3,video isaaclab train --rl_library sb3 --task Isaac-Cartpole --num_envs 64 --video
```

#### isaaclab.sh / isaaclab.bat

```bash
./isaaclab.sh train --rl_library sb3 --task Isaac-Cartpole --num_envs 64 --video
```

The videos are saved to the `logs/sb3/Isaac-Cartpole/<run-dir>/videos/train` directory. You can open these videos
using any video player.

For tasks with on-scene cameras, you can also save the sensor image outputs directly during training
with `--capture_env_sensors`. See [/source/how-to/capture_sensor_frames](/source/how-to/capture_sensor_frames) for the available
options and output formats.

#### Interactive execution

While the above two methods are useful for training the agent, they don't allow you to interact with the
simulation to see what is happening. In this case, run the
training command as follows:

#### uv (Recommended)

```bash
uv run --extra sb3 isaaclab train --rl_library sb3 --task Isaac-Cartpole --num_envs 64 --viz kit
```

#### isaaclab.sh / isaaclab.bat

```bash
./isaaclab.sh train --rl_library sb3 --task Isaac-Cartpole --num_envs 64 --viz kit
```

This will open the Kit visualizer window and you can see the agent training in the environment. However, this
can slow down the training process because interactive visual feedback is enabled. As a workaround, you
can switch between different render modes in the `"Isaac Lab"` window that is docked on the bottom-right
corner of the screen. To learn more about these render modes, please check the
`sim.SimulationContext.RenderMode` class.

### Viewing the logs

On a separate terminal, you can monitor the training progress by executing the following command:

#### uv (Recommended)

```bash
# execute from the root directory of the repository
uv run python -m tensorboard.main --logdir logs/sb3/Isaac-Cartpole
```

#### isaaclab.sh / isaaclab.bat

```bash
# execute from the root directory of the repository
./isaaclab.sh -p -m tensorboard.main --logdir logs/sb3/Isaac-Cartpole
```

### Playing the trained agent

Once the training is complete, you can visualize the trained agent by executing the following command:

#### uv (Recommended)

```bash
# execute from the root directory of the repository
uv run --extra sb3 isaaclab play --rl_library sb3 --task Isaac-Cartpole --num_envs 32 --viz kit
```

#### isaaclab.sh / isaaclab.bat

```bash
# execute from the root directory of the repository
./isaaclab.sh play --rl_library sb3 --task Isaac-Cartpole --num_envs 32 --viz kit
```

The above command will load the latest checkpoint from the `logs/sb3/Isaac-Cartpole`
directory. You can also specify a specific checkpoint by passing the `--checkpoint` flag.