> This page is for version Release 2.3.
> For other versions, use one of these documentation indexes:
> - Release 3.0 (default): https://docs.nvidia.com/isaaclab/3.0/llms.txt
> - Release 2.3: https://docs.nvidia.com/isaaclab/2.3/llms.txt
> - Release 2.2: https://docs.nvidia.com/isaaclab/2.2/llms.txt
> - Release 2.1: https://docs.nvidia.com/isaaclab/2.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/isaaclab/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/isaaclab/_mcp/server.

# Multi-GPU and Multi-Node Training

Isaac Lab supports multi-GPU and multi-node reinforcement learning. Currently, this feature is only
available for RL-Games, RSL-RL and skrl libraries workflows. We are working on extending this feature to
other workflows.

> **Warning**
>
> Multi-GPU and multi-node training is only supported on Linux. Windows support is not available at this time.
> This is due to limitations of the NCCL library on Windows.

## Multi-GPU Training

Isaac Lab supports the following multi-GPU training frameworks:

* [Torchrun](https://docs.pytorch.org/docs/stable/elastic/run.html) through [PyTorch distributed](https://pytorch.org/docs/stable/distributed.html)
* [JAX distributed](https://jax.readthedocs.io/en/latest/jax.distributed.html)

### Pytorch Torchrun Implementation

We are using [Pytorch Torchrun](https://docs.pytorch.org/docs/stable/elastic/run.html) to manage multi-GPU
training. Torchrun manages the distributed training by:

* **Process Management**: Launching one process per GPU, where each process is assigned to a specific GPU.
* **Script Execution**: Running the same training script (e.g., RL Games trainer) on each process.
* **Environment Instances**: Each process creates its own instance of the Isaac Lab environment.
* **Gradient Synchronization**: Aggregating gradients across all processes and broadcasting the synchronized
  gradients back to each process after each training step.

> **Tip**
>
> Check out this [3 minute youtube video from PyTorch](https://www.youtube.com/watch?v=Cvdhwx-OBBo\&list=PL_lsbAsL_o2CSuhUhJIiW0IkdT5C2wGWj\&index=2)
> to understand how Torchrun works.

The key components in this setup are:

* **Torchrun**: Handles process spawning, communication, and gradient synchronization.
* **RL Library**: The reinforcement learning library that runs the actual training algorithm.
* **Isaac Lab**: Provides the simulation environment that each process instantiates independently.

Under the hood, Torchrun uses the [DistributedDataParallel](https://docs.pytorch.org/docs/2.7/notes/ddp.html#internal-design)
module to manage the distributed training. When training with multiple GPUs using Torchrun, the following happens:

* Each GPU runs an independent process

* Each process executes the full training script

* Each process maintains its own:

  * Isaac Lab environment instance (with *n* parallel environments)
  * Policy network copy
  * Experience buffer for rollout collection

* All processes synchronize only for gradient updates

For a deeper dive into how Torchrun works, checkout
[PyTorch Docs: DistributedDataParallel - Internal Design](https://pytorch.org/docs/stable/notes/ddp.html#internal-design).

### Jax Implementation

> **Tip**
>
> JAX is only supported with the skrl library.

With JAX, we are using [skrl.utils.distributed.jax](https://skrl.readthedocs.io/en/latest/api/utils/distributed.html)
Since the ML framework doesn't automatically start multiple processes from a single program invocation,
the skrl library provides a module to start them.

![Multi-GPU training paradigm](/isaaclab/_fern-files/isaaclab.docs.buildwithfern.com/d31682a343628c4ec7cc419f3f1ef703a43fdfb3dab73d46b90243a098b63573/assets/releases/2.3/static/multi-gpu-rl/a3c-light.svg)

![Multi-GPU training paradigm](/isaaclab/_fern-files/isaaclab.docs.buildwithfern.com/5a629c3cc8a42acd831ce86952cdbdbdd61722d07c6fd76e266f6cb7deccb568/assets/releases/2.3/static/multi-gpu-rl/a3c-dark.svg)

### Running Multi-GPU Training

To train with multiple GPUs, use the following command, where `--nproc_per_node` represents the number of available GPUs:

#### rl_games

```shell
python -m torch.distributed.run --nnodes=1 --nproc_per_node=2 scripts/reinforcement_learning/rl_games/train.py --task=Isaac-Cartpole-v0 --headless --distributed
```

#### rsl_rl

```shell
python -m torch.distributed.run --nnodes=1 --nproc_per_node=2 scripts/reinforcement_learning/rsl_rl/train.py --task=Isaac-Cartpole-v0 --headless --distributed
```

#### skrl

#### PyTorch

```shell
python -m torch.distributed.run --nnodes=1 --nproc_per_node=2 scripts/reinforcement_learning/skrl/train.py --task=Isaac-Cartpole-v0 --headless --distributed
```

#### JAX

```shell
python -m skrl.utils.distributed.jax --nnodes=1 --nproc_per_node=2 scripts/reinforcement_learning/skrl/train.py --task=Isaac-Cartpole-v0 --headless --distributed --ml_framework jax
```

## Multi-Node Training

To scale up training beyond multiple GPUs on a single machine, it is also possible to train across multiple nodes.
To train across multiple nodes/machines, it is required to launch an individual process on each node.

For the master node, use the following command, where `--nproc_per_node` represents the number of available GPUs, and
`--nnodes` represents the number of nodes:

#### rl_games

```shell
python -m torch.distributed.run --nproc_per_node=2 --nnodes=2 --node_rank=0 --rdzv_id=123 --rdzv_backend=c10d --rdzv_endpoint=localhost:5555 scripts/reinforcement_learning/rl_games/train.py --task=Isaac-Cartpole-v0 --headless --distributed
```

#### rsl_rl

```shell
python -m torch.distributed.run --nproc_per_node=2 --nnodes=2 --node_rank=0 --rdzv_id=123 --rdzv_backend=c10d --rdzv_endpoint=localhost:5555 scripts/reinforcement_learning/rsl_rl/train.py --task=Isaac-Cartpole-v0 --headless --distributed
```

#### skrl

#### PyTorch

```shell
python -m torch.distributed.run --nproc_per_node=2 --nnodes=2 --node_rank=0 --rdzv_id=123 --rdzv_backend=c10d --rdzv_endpoint=localhost:5555 scripts/reinforcement_learning/skrl/train.py --task=Isaac-Cartpole-v0 --headless --distributed
```

#### JAX

```shell
python -m skrl.utils.distributed.jax --nproc_per_node=2 --nnodes=2 --node_rank=0 --coordinator_address=ip_of_master_machine:5555 scripts/reinforcement_learning/skrl/train.py --task=Isaac-Cartpole-v0 --headless --distributed --ml_framework jax
```

Note that the port (`5555`) can be replaced with any other available port.

For non-master nodes, use the following command, replacing `--node_rank` with the index of each machine:

#### rl_games

```shell
python -m torch.distributed.run --nproc_per_node=2 --nnodes=2 --node_rank=1 --rdzv_id=123 --rdzv_backend=c10d --rdzv_endpoint=ip_of_master_machine:5555 scripts/reinforcement_learning/rl_games/train.py --task=Isaac-Cartpole-v0 --headless --distributed
```

#### rsl_rl

```shell
python -m torch.distributed.run --nproc_per_node=2 --nnodes=2 --node_rank=1 --rdzv_id=123 --rdzv_backend=c10d --rdzv_endpoint=ip_of_master_machine:5555 scripts/reinforcement_learning/rsl_rl/train.py --task=Isaac-Cartpole-v0 --headless --distributed
```

#### skrl

#### PyTorch

```shell
python -m torch.distributed.run --nproc_per_node=2 --nnodes=2 --node_rank=1 --rdzv_id=123 --rdzv_backend=c10d --rdzv_endpoint=ip_of_master_machine:5555 scripts/reinforcement_learning/skrl/train.py --task=Isaac-Cartpole-v0 --headless --distributed
```

#### JAX

```shell
python -m skrl.utils.distributed.jax --nproc_per_node=2 --nnodes=2 --node_rank=1 --coordinator_address=ip_of_master_machine:5555 scripts/reinforcement_learning/skrl/train.py --task=Isaac-Cartpole-v0 --headless --distributed --ml_framework jax
```

For more details on multi-node training with PyTorch, please visit the
[PyTorch documentation](https://pytorch.org/tutorials/intermediate/ddp_series_multinode.html).
For more details on multi-node training with JAX, please visit the
[skrl documentation](https://skrl.readthedocs.io/en/latest/api/utils/distributed.html) and the
[JAX documentation](https://jax.readthedocs.io/en/latest/multi_process.html).

> **Note**
>
> As mentioned in the PyTorch documentation, "multi-node training is bottlenecked by inter-node communication
> latencies". When this latency is high, it is possible multi-node training will perform worse than running on
> a single node instance.