> This page is for version Release 2.1. > For other versions, use one of these documentation indexes: > - Release 3.0 (default): https://docs.nvidia.com/isaaclab/3.0/llms.txt > - Release 2.3: https://docs.nvidia.com/isaaclab/2.3/llms.txt > - Release 2.2: https://docs.nvidia.com/isaaclab/2.2/llms.txt > - Release 2.1: https://docs.nvidia.com/isaaclab/2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/isaaclab/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/isaaclab/_mcp/server. # Multi-GPU and Multi-Node Training Isaac Lab supports multi-GPU and multi-node reinforcement learning. Currently, this feature is only available for RL-Games, RSL-RL and skrl libraries workflows. We are working on extending this feature to other workflows. > **Warning** > > Multi-GPU and multi-node training is only supported on Linux. Windows support is not available at this time. > This is due to limitations of the NCCL library on Windows. ## Multi-GPU Training For complex reinforcement learning environments, it may be desirable to scale up training across multiple GPUs. This is possible in Isaac Lab through the use of the [PyTorch distributed](https://pytorch.org/docs/stable/distributed.html) framework or the [JAX distributed](https://jax.readthedocs.io/en/latest/jax.distributed.html) module respectively. In PyTorch, the `torch.distributed` API is used to launch multiple processes of training, where the number of processes must be equal to or less than the number of GPUs available. Each process runs on a dedicated GPU and launches its own instance of Isaac Sim and the Isaac Lab environment. Each process collects its own rollouts during the training process and has its own copy of the policy network. During training, gradients are aggregated across the processes and broadcasted back to the process at the end of the epoch. In JAX, since the ML framework doesn't automatically start multiple processes from a single program invocation, the skrl library provides a module to start them. ![Multi-GPU training paradigm](/isaaclab/_fern-files/isaaclab.docs.buildwithfern.com/d31682a343628c4ec7cc419f3f1ef703a43fdfb3dab73d46b90243a098b63573/assets/releases/2.1/static/multi-gpu-rl/a3c-light.svg) ![Multi-GPU training paradigm](/isaaclab/_fern-files/isaaclab.docs.buildwithfern.com/5a629c3cc8a42acd831ce86952cdbdbdd61722d07c6fd76e266f6cb7deccb568/assets/releases/2.1/static/multi-gpu-rl/a3c-dark.svg) To train with multiple GPUs, use the following command, where `--nproc_per_node` represents the number of available GPUs: #### rl_games ```shell python -m torch.distributed.run --nnodes=1 --nproc_per_node=2 scripts/reinforcement_learning/rl_games/train.py --task=Isaac-Cartpole-v0 --headless --distributed ``` #### rsl_rl ```shell python -m torch.distributed.run --nnodes=1 --nproc_per_node=2 scripts/reinforcement_learning/rsl_rl/train.py --task=Isaac-Cartpole-v0 --headless --distributed ``` #### skrl #### PyTorch ```shell python -m torch.distributed.run --nnodes=1 --nproc_per_node=2 scripts/reinforcement_learning/skrl/train.py --task=Isaac-Cartpole-v0 --headless --distributed ``` #### JAX ```shell python -m skrl.utils.distributed.jax --nnodes=1 --nproc_per_node=2 scripts/reinforcement_learning/skrl/train.py --task=Isaac-Cartpole-v0 --headless --distributed --ml_framework jax ``` ## Multi-Node Training To scale up training beyond multiple GPUs on a single machine, it is also possible to train across multiple nodes. To train across multiple nodes/machines, it is required to launch an individual process on each node. For the master node, use the following command, where `--nproc_per_node` represents the number of available GPUs, and `--nnodes` represents the number of nodes: #### rl_games ```shell python -m torch.distributed.run --nproc_per_node=2 --nnodes=2 --node_rank=0 --rdzv_id=123 --rdzv_backend=c10d --rdzv_endpoint=localhost:5555 scripts/reinforcement_learning/rl_games/train.py --task=Isaac-Cartpole-v0 --headless --distributed ``` #### rsl_rl ```shell python -m torch.distributed.run --nproc_per_node=2 --nnodes=2 --node_rank=0 --rdzv_id=123 --rdzv_backend=c10d --rdzv_endpoint=localhost:5555 scripts/reinforcement_learning/rsl_rl/train.py --task=Isaac-Cartpole-v0 --headless --distributed ``` #### skrl #### PyTorch ```shell python -m torch.distributed.run --nproc_per_node=2 --nnodes=2 --node_rank=0 --rdzv_id=123 --rdzv_backend=c10d --rdzv_endpoint=localhost:5555 scripts/reinforcement_learning/skrl/train.py --task=Isaac-Cartpole-v0 --headless --distributed ``` #### JAX ```shell python -m skrl.utils.distributed.jax --nproc_per_node=2 --nnodes=2 --node_rank=0 --coordinator_address=ip_of_master_machine:5555 scripts/reinforcement_learning/skrl/train.py --task=Isaac-Cartpole-v0 --headless --distributed --ml_framework jax ``` Note that the port (`5555`) can be replaced with any other available port. For non-master nodes, use the following command, replacing `--node_rank` with the index of each machine: #### rl_games ```shell python -m torch.distributed.run --nproc_per_node=2 --nnodes=2 --node_rank=1 --rdzv_id=123 --rdzv_backend=c10d --rdzv_endpoint=ip_of_master_machine:5555 scripts/reinforcement_learning/rl_games/train.py --task=Isaac-Cartpole-v0 --headless --distributed ``` #### rsl_rl ```shell python -m torch.distributed.run --nproc_per_node=2 --nnodes=2 --node_rank=1 --rdzv_id=123 --rdzv_backend=c10d --rdzv_endpoint=ip_of_master_machine:5555 scripts/reinforcement_learning/rsl_rl/train.py --task=Isaac-Cartpole-v0 --headless --distributed ``` #### skrl #### PyTorch ```shell python -m torch.distributed.run --nproc_per_node=2 --nnodes=2 --node_rank=1 --rdzv_id=123 --rdzv_backend=c10d --rdzv_endpoint=ip_of_master_machine:5555 scripts/reinforcement_learning/skrl/train.py --task=Isaac-Cartpole-v0 --headless --distributed ``` #### JAX ```shell python -m skrl.utils.distributed.jax --nproc_per_node=2 --nnodes=2 --node_rank=1 --coordinator_address=ip_of_master_machine:5555 scripts/reinforcement_learning/skrl/train.py --task=Isaac-Cartpole-v0 --headless --distributed --ml_framework jax ``` For more details on multi-node training with PyTorch, please visit the [PyTorch documentation](https://pytorch.org/tutorials/intermediate/ddp_series_multinode.html). For more details on multi-node training with JAX, please visit the [skrl documentation](https://skrl.readthedocs.io/en/latest/api/utils/distributed.html) and the [JAX documentation](https://jax.readthedocs.io/en/latest/multi_process.html). > **Note** > > As mentioned in the PyTorch documentation, "multi-node training is bottlenecked by inter-node communication > latencies". When this latency is high, it is possible multi-node training will perform worse than running on > a single node instance.