Ecosystem | Isaac Lab

Isaac Lab is an open-source framework for training and testing robot-learning systems in GPU-parallel simulation. It supports reinforcement learning, imitation learning, teleoperation, and policy post-training.

Shared scene, asset, sensor, and task APIs let you use different physics engines, renderers, and learning libraries. Newton provides Warp-based simulation and CUDA graph support; OvPhysX and OvRTX provide standalone PhysX and RTX runtimes. Choose the combination your task needs. See /source/concepts/backends_and_presets for configuration and launch commands.

Modular Multi-Backend Physics and Rendering

Isaac Lab supports two physics backends, with PhysX support available through either a Kit-based Isaac Sim PhysX implementation and a standalone (Kit-less) OvPhysX implementation.

Backend	Best fit	Benefits	Trade-offs
Newton	Lightweight training and workflows that can use multiple solver families	Warp-native GPU execution without Isaac Sim; provides MJWarp, VBD, MPM, and other solver paths	Task and feature coverage varies by solver, and changing solvers can require retuning
OvPhysX	Kit-less PhysX workflows	PhysX SDK available through a standalone runtime and can pair with kit-less OVRTX rendering	Cannot be mixed together with Isaac Sim workflows
Isaac Sim PhysX	Isaac Sim workflows that require full Kit integration	PhysX SDK through Isaac Sim and Kit	Requires the larger Isaac Sim and Kit runtime

Additionally, Isaac Lab provides multiple rendering options through the Newton Tiled Camera sensor and RTX, available as a standalone library OvRTX, as well as Isaac Sim Kit-based Isaac Sim RTX.

Renderer	Best fit	Benefits	Trade-offs
Newton Tiled Camera Sensor	Kit-less training where lightweight camera observations are sufficient at small resolutions	Warp-native raytraced tiled rendering; supports RGB, depth, normals, and semantic and instance segmentation at high throughput	Lower fidelity rendering; minimal support for complex lighting and physics-based materials
OvRTX	Kit-less rendering workflows that require high-fidelity RTX image quality	Provides higher throughput minimal mode; supports photo-real rendering and can be combined with Newton or OvPhysX without requiring Isaac Sim	Requires the optional `ovrtx` runtime; may require higher VRAM for high fidelity rendering
Isaac RTX	Isaac Sim-based workflows that require full RTX fidelity and the broadest sensor-output coverage	Provides higher throughput minimal mode and photo-real rendering integrated with PhysX, Kit, and the Isaac Sim toolchain	Requires the larger Isaac Sim and Kit runtime

Newton Tiled Camera

Lightweight Warp sensor for tiled camera observations.

OVRTX

Kit-less RTX materials, lighting, and camera outputs.

Isaac RTX

Full RTX fidelity and camera-output coverage in Isaac Sim.

See Renderer outputs at a glance for the complete output gallery.

Physics, camera sensor rendering, and interactive visualization are separate choices. For example, Newton can use the lightweight Newton Tiled Camera Sensor or OvRTX for RTX image quality without Isaac Sim. See /source/concepts/physics_backends, /source/concepts/renderers, and https://isaac-sim.github.io/IsaacLab/release/3.0.0/source/concepts/visualization.html for current support matrices and preset commands.

Newton is included in the default installation and can run without Isaac Sim. See Installation for optional backends and renderers.

Capabilities in motion

These examples span real-to-sim reconstruction, vision-policy learning, and contact-rich material simulation. Explore the packaged demos and environment catalog for more examples.

Real-to-sim scene reconstruction

A camera capture becomes a simulation-ready NuRec reconstruction of a living room.

NuRec turns real-world captures into simulation-ready scenes for policy training and evaluation. See COMPASS with NuRec .

Vision-policy distillation

An SO-101 arm uses a wrist-camera policy to place a vial in a rack.

The SO-101 tutorial trains a state teacher and distills it into a wrist-camera policy for vial placement. See the SO-101 tutorial .

Robot cable manipulation

A Rizon robot and Sharpa hand demonstrate contact-rich manipulation of an RJ45 cable.

Robot teapot manipulation

A Rizon robot and Sharpa hand demonstrate dexterous teapot manipulation with MPM fluid simulation.

Is Isaac Lab a simulator?

At its core, Isaac Lab is not a robotics simulator; it is a framework for building robot learning applications on top of a simulator. An analogous example is RoboSuite, which is built on top of MuJoCo for fixed-base manipulation. Other examples include MuJoCo Playground (built on MJX) and Isaac Gym (built on PhysX).

Isaac Lab’s shared interfaces cover the Newton, Kit-based PhysX, and OvPhysX backends without requiring environment code to import backend-specific modules directly.

The framework addresses a recurring problem with standalone task implementations: because each task reimplements the observation, reward, termination, and randomization logic from scratch, large projects accumulate significant code duplication. Isaac Lab solves this with two complementary patterns:

Manager-based environments defer every behavioral concern to typed, composable manager objects (ObservationManager, RewardManager, TerminationManager, EventManager, CurriculumManager, CommandManager, ActionManager, RecorderManager). Each manager is driven by small, reusable MDP term functions that live in isaaclab.envs.mdp. This makes it easy to mix and match terms across tasks and to test individual components in isolation.
Direct environments implement _get_observations, _get_rewards, _get_dones, and _reset_idx directly in a subclass, similar to the Isaac Gym style. They sacrifice some modularity for simplicity and are a natural starting point for rapid prototyping.

Both patterns expose a standard gymnasium Env interface with vectorized semantics, so the same environment works unmodified with any of the supported RL libraries. Configuration management uses Hydra with a preset system that allows selecting physics backends and hyperparameter sweeps from the command line.

Why should I use Isaac Lab?

Isaac Lab provides an open-sourced platform for the community to build benchmarks and robot learning systems together. Sharing a common infrastructure lets teams reuse existing components, compare results on the same tasks, and focus on the research problems that matter rather than rebuilding simulation scaffolding from scratch.

Concretely, Isaac Lab offers:

Two authoring patterns — manager-based for modular research and direct for rapid prototyping — with a shared InteractiveScene and sensor stack.
Multi-backend simulation — select Newton, Kit-based PhysX or OvPhysX from the command line when the task provides the corresponding preset. Additionally, choose from Newton rendering sensor, Kit-based RTX, or OvRTX for rendering pipelines.
Rich sensor suite — cameras, ray-casters, contact sensors, IMU, frame transformers, joint-wrench sensors, and visuo-tactile sensors.
Imitation learning tooling — isaaclab_mimic provides cuRobo-based planners and a full dataset-generation pipeline for human demonstration collection.
Teleoperation and XR — isaaclab_teleop supports OpenXR, CloudXR, gamepads, spacemouses, and Haply devices with retargeters for manipulators and humanoids.
Hydra configuration management — hierarchical configs with command-line overrides and a preset system for multi-backend environment variants.
RL library integrations — wrappers for RSL-RL, skrl, Stable Baselines 3, and RL Games ship in isaaclab_rl. Additionally, integration of RLinf provides RL post-training capabilities for fine-tuning large foundation models.
Kit-less deployment — run policies and simulations using the Newton backend without a full Isaac Sim installation. URDF and MJCF command-line conversion can also run kit-less when the standalone isaacsim-asset-isolated importer wheel is installed.

Applications built on Isaac Lab

The framework also serves as the simulation and environment layer for higher-level applications. These projects have their own releases and installation requirements:

Isaac Lab-Arena

Composes scenes, robot embodiments, and tasks into scalable benchmarks, then evaluates generalist robot policies across controlled environment variations.

NVIDIA Isaac GR00T workflows

Uses Isaac Lab environments for demonstration collection, synthetic trajectory generation, policy training, and closed-loop evaluation of generalist robot models.