Isaac Capture

Isaac Capture is the unified framework for high-fidelity egocentric and robot data collection. It provides a standardized device interface, a flexible graph-based retargeting pipeline, and works seamlessly across simulated and real-world robots.

Isaac Capture replaces the previous native XR teleop stack (isaaclab.devices.openxr) in Isaac Lab. For migration details see Migrating To 3.0.

Supported Devices

Isaac Capture supports multiple XR headsets and tracking peripherals. Each device provides different input modes, which determine which retargeters and control schemes are available.

Choose a Control Scheme

The right combination of input device and retargeters depends on your task. Use this table as a starting point, then see the detailed pipeline examples below.

Why motion controllers for manipulation? Controllers provide precise spatial control via a grip pose and a physical trigger for gripper actuation, making them ideal for pick-and-place tasks.

Why hand tracking for complex dex hands? Hand tracking captures the full 26-joint hand pose required for high-fidelity dexterous retargeting. This is essential when individual finger control matters.

How It Works

The ~isaaclab_teleop.IsaacTeleopDevice is the main integration point between Isaac Capture and Isaac Lab. It composes three collaborators:

• XrAnchorManager — creates and synchronizes an XR anchor prim in the simulation, and computes the world_T_anchor transform matrix that maps XR tracking data into the simulation coordinate frame.

• TeleopSessionLifecycle — builds the retargeting pipeline, acquires OpenXR handles from Isaac Sim’s XR bridge (or, in standalone mode, creates its own OpenXR session through the CloudXR runtime — see Run Without a Headset (Standalone I/O)), creates the TeleopSession, and steps it each frame to produce an action tensor.

• CommandHandler — lightweight callback registry for START / STOP / RESET commands. Scripts can register callbacks via ~isaaclab_teleop.IsaacTeleopDevice.add_callback, but the primary control path uses ~isaaclab_teleop.poll_control_events (see Teleop Control States (Start / Stop / Reset)).

The session uses deferred creation: if the user has not yet clicked “Start XR” in the Isaac Sim UI, the session is not created immediately. Instead, each call to advance() retries session creation until OpenXR handles become available. Once connected, advance() returns a flattened action tensor (torch.Tensor) on the configured device. It returns None when the session is not yet ready or has been torn down.

In standalone mode (Run Without a Headset (Standalone I/O)), creation is not gated on Kit XR handles — the session starts as soon as the CloudXR runtime is available.

Visualize XR Tracking

Isaac Capture can draw the raw XR tracking poses in the Isaac Lab world frame. Use this visualization to confirm that tracking data is available and aligned with the simulated robot:

• red spheres show the 26 joints of each tracked hand when the retargeting pipeline contains a HandsSource;
• RGB coordinate axes show the OpenXR aim pose of each tracked controller. The X, Y, and Z axes are red, green, and blue, respectively, and the controller’s -Z axis points forward in its natural pointing direction.

Enable the visualization when launching a teleoperation session:

uv run --extra teleop,isaacsim isaaclab teleop run \
    --task IsaacContrib-PickPlace-Locomanipulation-G1-Abs \
    --visualizer kit \
    --xr \
    --enable_debug_visualization

The --enable_debug_visualization flag is also available in scripts/tools/record_demos.py and scripts/environments/teleoperation/teleop_replay_agent.py. The option is applied when the Isaac Capture device is created and cannot be toggled during a running device session. The markers are diagnostic only and do not change the actions produced by the retargeting pipeline.

Teleop Control States (Start / Stop / Reset)

Isaac Lab supports remote teleop control commands — start, stop, and reset — sent from the XR headset to the simulation. These are used to begin and end demonstration recording, pause the robot, or reset the environment without touching the simulation host. The same commands can be driven locally without a headset via ~isaaclab_teleop.IsaacTeleopDevice.request_start / ~isaaclab_teleop.IsaacTeleopDevice.request_stop and ~isaaclab_teleop.IsaacTeleopDevice.reset (see Run Without a Headset (Standalone I/O)).

How it works

By default, every ~isaaclab_teleop.IsaacTeleopCfg enables a control message channel using the well-known UUID uuid5(NAMESPACE_DNS, "teleop_command"). The channel is created as a teleop_control_pipeline inside TeleopCore’s TeleopSession, which means:

1. A ~isaacteleop.retargeting_engine.deviceio_source_nodes.MessageChannelSource opens an OpenXR opaque data channel (XR_NV_opaque_data_channel) with the agreed-upon UUID.
2. The CloudXR JS client (or any other client) discovers the channel by UUID and sends UTF-8 JSON commands:

{"type": "teleop_command", "message": {"command": "start teleop"}}
{"type": "teleop_command", "message": {"command": "stop teleop"}}
{"type": "teleop_command", "message": {"command": "reset teleop"}}

3. A ~isaaclab_teleop.teleop_message_processor.TeleopMessageProcessor parses these payloads and produces boolean pulse signals (run_toggle, kill, reset).
4. ~isaacteleop.teleop_session_manager.DefaultTeleopStateManager consumes the boolean signals, runs its state machine (edge detection, fail-safe), and produces teleop_state (one-hot) and reset_event (bool pulse) outputs.
5. TeleopCore decodes these outputs into ExecutionEvents and injects them into every retargeter’s ComputeContext, so stateful retargeters can react to state changes (e.g. reinitializing cross-step state on reset).

Polling control events in your script

Use ~isaaclab_teleop.poll_control_events to read the latest control state each frame:

from isaaclab_teleop import poll_control_events
with IsaacTeleopDevice(cfg) as device:
    running = False
    while sim_app.is_running():
        action = device.advance()
        ctrl = poll_control_events(device)
        if ctrl.is_active is not None:
            running = ctrl.is_active      # True after "start", False after "stop"
        if ctrl.should_reset:
            env.reset()                    # "reset" command received this frame
        if action is not None and running:
            env.step(action.repeat(num_envs, 1))
        else:
            env.sim.render()

~isaaclab_teleop.ControlEvents has two fields:

• is_active — True after a “start” command, False after “stop”, None when no command has been received yet (callers should leave their own flag unchanged).
• should_reset — True for exactly one frame after a “reset” command.

Disabling the control channel

If you do not need headset-driven start/stop/reset (e.g. keyboard-only workflows), set control_channel_uuid=None in your config:

IsaacTeleopCfg(
    pipeline_builder=_build_my_pipeline,
    control_channel_uuid=None,   # no opaque data channel created
)

Using a custom channel UUID

To use a different channel UUID (e.g. for a separate control protocol), pass any 16-byte bytes value:

import uuid
MY_UUID = uuid.uuid5(uuid.NAMESPACE_DNS, "my_custom_control").bytes
IsaacTeleopCfg(
    pipeline_builder=_build_my_pipeline,
    control_channel_uuid=MY_UUID,
)

The CloudXR JS client must be updated to discover this UUID when sending commands.

Run Without a Headset (Standalone I/O)

By default the teleop scripts drive Isaac Capture through Isaac Sim’s Kit XR bridge, which renders the scene to a connected XR headset. When you only need Isaac Capture as an input/output transport — for example a joint-space leader arm streaming encoder angles, or any non-XR device — you can run it standalone, with no Kit XR rendering.

The teleop_se3_agent.py and record_demos.py scripts select the mode with the --xr flag:

• With --xr — the full Kit XR path (headset rendering, anchor, hand / controller tracking).
• Without --xr — Isaac Capture creates and owns its own OpenXR session through the CloudXR runtime (create_isaac_teleop_device(..., use_kit_xr_bridge=False)); teleop input/output works headless, with no Kit XR rendering.

Starting teleop without a headset

Without a headset there is no client to send the start command, so the scripts drive the state machine locally:

• On startup they call ~isaaclab_teleop.IsaacTeleopDevice.request_start, which transitions the teleop state machine to RUNNING (see Teleop Control States (Start / Stop / Reset)) — so a headless session begins running immediately, with no headset UI.
• When a Kit viewport window is present, they also bind keys for interactive control:

Both paths flow through the same state machine, so ~isaaclab_teleop.IsaacTeleopDevice.request_start / ~isaaclab_teleop.IsaacTeleopDevice.request_stop and remote headset commands are interchangeable, and ~isaaclab_teleop.poll_control_events stays authoritative.

Example: SO-101 leader-arm joint teleoperation

IsaacContrib-Stack-Cube-SO101-Joint-Teleop-v0 mirrors the joint angles streamed by a physical SO-101 leader arm directly onto the simulated follower — no XR headset, inverse kinematics, or anchor. Its pipeline is JointStateSource -> JointStateRetargeter (mode="joint") -> TensorReorderer.

For the end-to-end walkthrough — building the plugin, hardware setup, calibration, launching, and recording a dataset — see Data Collection in Sim in the Isaac Capture documentation. The prerequisites below are the minimum needed to get this example running.

Prerequisites

• SO-101 hardware: A physical SO-101 leader arm connected to the workstation over USB.

• The so101_leader plugin, built from Isaac Capture source: the leader’s joint state is streamed by a standalone C++ plugin that you run alongside the sim.

Install the build prerequisites first — a missing clang-format-14 is the most common cause of a failed build, because the format check is enforced by default on Linux:

sudo apt-get update
sudo apt-get install -y build-essential cmake libx11-dev clang-format-14 ccache patchelf

Then clone, check out the release branch matching the isaacteleop version Isaac Lab is pinned to (isaacteleop~=1.4.145 in the teleop extra of the root pyproject.toml, so the plugin’s wire format matches the isaacteleop package Isaac Lab installs), configure, build, and install. To target a specific Python version, pass -DISAAC_TELEOP_PYTHON_VERSION=3.12 (or 3.11, 3.13) on the configure line; each version needs its own build directory if building multiple at once:

git clone https://github.com/NVIDIA/IsaacCapture.git
cd IsaacCapture
git checkout release/1.4.x
cmake -B build                       # configure (default: Python 3.11)
cmake --build build --parallel       # build
cmake --install build                # install into ./install

The plugin is installed to <IsaacCapture>/install/plugins/so101_leader/so101_leader_plugin. Every later command in this section runs from the Isaac Capture checkout root; substitute your own path for /path/to/IsaacCapture. Verify the build by running the plugin with no arguments — that selects the synthetic backend, so it starts without any hardware attached:

cd /path/to/IsaacCapture
./install/plugins/so101_leader/so101_leader_plugin

See Build from Source for the full prerequisite list and all build options, and the build-troubleshooting table in Data Collection in Sim for common failures (including the clang-format not found but ENABLE_CLANG_FORMAT_CHECK is ON CMake error).

• Calibration: The SO-101 arm must be calibrated before use. The plugin talks to the FEETECH servos directly — it has no lerobot or FEETECH SDK dependency — so calibrate with its own calibrate subcommand, which needs no OpenXR runtime.

  First, identify the serial port. The easiest way is uvx, which runs lerobot-find-port in a temporary environment with no installation required:

uvx --from "lerobot[hardware]" lerobot-find-port

Alternatively, plug in the USB cable and immediately run:

sudo dmesg | grep tty | tail -1

Because tail -1 shows only the most recent kernel message, the output unambiguously names the just-connected device, e.g. [12345.6] usb ... ttyACM0. Use that path (/dev/ttyACM0, /dev/ttyACM1, etc.) in the commands below.

On Linux, serial devices are owned by the dialout group, so the plugin fails with Permission denied unless your user can access the port. Add your user to the group once, then log out and back in for it to take effect:

sudo usermod -aG dialout $USER

Alternatively, grant access for the current session only (this resets when the arm is unplugged):

sudo chmod 666 /dev/ttyACM0

Then run the calibration:

cd /path/to/IsaacCapture
./install/plugins/so101_leader/so101_leader_plugin calibrate /dev/ttyACM0 so101_leader.calib

It runs two interactive steps (hold the arm at mid-range, then sweep every joint through its full range) and writes the calibration file, which the plugin reads at startup to map raw encoder values to joint angles. An existing LeRobot calibration .json can be passed instead. Running without calibration produces incorrect joint mappings and the follower arm will not track the leader.

Run the simulation

The SO-101 workflow supports two monitoring modes — choose based on whether you have an XR headset available:

Without a headset (local viewport only)

Omit --xr. The sim runs in standalone mode and the teleop state machine starts automatically on launch — no headset connection is needed (see Run Without a Headset (Standalone I/O)). Pass --visualizer kit to watch the follower in the local Kit viewport; headless is the default.

uv run --extra teleop,isaacsim isaaclab teleop run \
    --task IsaacContrib-Stack-Cube-SO101-Joint-Teleop-v0 \
    --num_envs 1 \
    --visualizer kit \
    physics=isaacsim_physx

With a headset (immersive XR view)

Add --xr to stream the simulation to a Quest, Pico, or Apple Vision Pro headset while the SO-101 leader arm still drives the follower joints. The retargeting pipeline is unchanged; --xr only controls whether the scene is rendered to the headset. Follow the connection steps in Connect an XR Device to pair the headset after launching.

uv run --extra teleop,isaacsim isaaclab teleop run \
    --task IsaacContrib-Stack-Cube-SO101-Joint-Teleop-v0 \
    --num_envs 1 \
    --visualizer kit --xr \
    physics=isaacsim_physx

Start the plugin

Isaac Lab does not spawn the plugin for you. Launch the sim first — it brings up the CloudXR runtime the plugin connects through — then, in a separate terminal, source the environment file the runtime writes on startup (this points the OpenXR loader at CloudXR) and start the plugin on the leader’s serial port (replace /dev/ttyACM0 with your device — see the port-identification note in the Calibration step above):

cd /path/to/IsaacCapture
source ~/.cloudxr/run/cloudxr.env
./install/plugins/so101_leader/so101_leader_plugin /dev/ttyACM0 so101_leader so101_leader.calib

Arguments are positional: [device_path] [collection_id] [calibration_file]. The collection_id must stay so101_leader — that is the tensor collection this task’s JointStateSource subscribes to.

Running the plugin with no arguments at all streams a synthetic trajectory on the default so101_leader collection, which is a good way to confirm the sim side is wired up before connecting hardware. Because the arguments are positional, there is no way to skip only the device path while still passing the later two:

./install/plugins/so101_leader/so101_leader_plugin   # synthetic, no hardware needed

See the SO-101 plugin README and Data Collection in Sim for all configuration options.

Start teleoperation

Without --xr: the script calls ~isaaclab_teleop.IsaacTeleopDevice.request_start automatically on startup, so the follower begins mirroring the leader immediately with no additional input required.

With --xr: a start command can be sent from the connected headset UI (Quest / Pico: the CloudXR.js Connect button; Apple Vision Pro: the Play button in the Isaac XR Teleop Sample Client). The SO-101 then begins driving the follower as soon as the headset sends the start command.

Regardless of mode, when the Kit viewport is open you can also control the session with keyboard shortcuts:

Move the physical SO-101 leader arm and the simulated follower will mirror its joint angles in real time. To record demonstrations from this task, run scripts/tools/record_demos.py with the same --task and the plugin running in its second terminal — see Data Collection in Sim for the full recording workflow and runtime troubleshooting.

Retargeting Framework

Isaac Capture uses a graph-based retargeting pipeline. Data flows from source nodes through retargeters and is combined into a single action tensor.

Source Nodes

• HandsSource — provides hand tracking data (left/right, 26 joints each).
• ControllersSource — provides motion controller data (grip pose, trigger, thumbstick, etc.).

Available Retargeters

Retargeters are provided by the isaacteleop package from the Isaac Capture repository. The retargeters listed below are those used by the built-in Isaac Lab environments. Isaac Capture may offer additional retargeters not listed here — refer to the Isaac Capture repository for the full set.

Maps hand or controller tracking to end-effector pose. Se3AbsRetargeter outputs a 7D absolute pose (position + quaternion). Se3RelRetargeter outputs a 6D delta. Configurable rotation offsets (roll, pitch, yaw in degrees).

Outputs a single float (-1.0 closed, 1.0 open). Uses controller trigger (priority) or thumb-index pinch distance from hand tracking.

Retargets full hand tracking (26 joints) to robot-specific hand joint angles using the dex-retargeting library. Requires a robot hand URDF and a YAML configuration file.

Maps VR controller buttons (trigger, squeeze) to G1 TriHand joints (7 DOF per hand). Simple mapping: trigger controls the index finger, squeeze controls the middle finger, and both together control the thumb.

Maps controller thumbsticks to a 4D locomotion command: [vel_x, vel_y, rot_vel_z, hip_height].

Utility that flattens and reorders outputs from multiple retargeters into a single 1D action tensor. The output_order must match the action space expected by the environment.

The built-in Isaac Lab environments use these retargeters as follows:

Teleoperation Environment Reference

The tables below list every built-in Isaac Lab environment that supports teleoperation, organized by input method. For closed-loop policy evaluation, the play script automatically applies each environment config’s play_mode overrides; pass --train_env_cfg to play the training configuration as-is.

Isaac Capture (XR Headset) Environments

These environments use the Isaac Capture XR pipeline with motion controllers or hand tracking.

Keyboard and SpaceMouse Environments

The device button layouts below apply to all environments in this section. Per-environment differences (gripper enabled/disabled, sensitivity) are noted in the environment table that follows.

Keyboard

SpaceMouse

Gamepad (Reach environments only)

Leader-Arm Environments

These environments are driven by a physical leader arm that streams joint angles over the Isaac Capture tensor transport; the angles are mirrored directly onto the simulated follower with no XR headset or inverse kinematics. See Example: SO-101 leader-arm joint teleoperation above for the run command and pipeline.

Switch Between Controllers and Hand Tracking

The retargeting pipeline determines whether an environment uses motion controllers or hand tracking. Switching input modes requires changing the pipeline_builder function in your environment config. No other environment-level changes are needed as long as the action space (TensorReorderer output order) stays the same.

Controller to hand tracking

The key changes are:

1. Create a HandsSource and apply the world-to-anchor transform to it (instead of ControllersSource).
2. Point the Se3RetargeterConfig.input_device at the appropriate HandsSource key.
3. Set use_wrist_rotation=True and use_wrist_position=True so that the SE3 retargeter reads from the hand wrist joint rather than the controller grip pose.
4. The GripperRetargeter already supports both inputs — it uses the controller trigger when connected to a ControllersSource or thumb-index pinch when connected to a HandsSource.

Here is the Franka stack environment’s controller-based pipeline alongside a hand-tracking variant for comparison.

Original (controller-based):

# SE3: tracks right controller grip pose
se3_cfg = Se3RetargeterConfig(
    input_device=ControllersSource.RIGHT,
    use_wrist_rotation=False,
    use_wrist_position=False,
    target_offset_roll=90.0,
)
se3 = Se3AbsRetargeter(se3_cfg, name="ee_pose")
connected_se3 = se3.connect({
    ControllersSource.RIGHT: transformed_controllers.output(
        ControllersSource.RIGHT
    ),
})

Modified (hand-tracking-based):

se3_cfg = Se3RetargeterConfig(
    input_device=HandsSource.RIGHT,
    use_wrist_rotation=True,
    use_wrist_position=True,
    target_offset_roll=0.0,
)
se3 = Se3AbsRetargeter(se3_cfg, name="ee_pose")
transformed_hands = hands.transformed(transform_input.output(ValueInput.VALUE))
connected_se3 = se3.connect({
    HandsSource.RIGHT: transformed_hands.output(HandsSource.RIGHT),
})

The GripperRetargeter needs no changes — it accepts both controller and hand inputs and uses whichever source is connected.

Hand tracking to controller

Reverse the steps above: set input_device to a ControllersSource key, transform the controllers instead of the hands, and set use_wrist_rotation=False and use_wrist_position=False. Adjust target_offset_roll/pitch/yaw to account for the controller grip frame orientation (typically 90 degrees roll for Franka-style grippers).

Build a Retargeting Pipeline

A pipeline builder is a callable that constructs the retargeting graph and returns an OutputCombiner with a single "action" key. Here is a complete example for a Franka manipulator (from stack_ik_abs_env_cfg.py):

def _build_franka_stack_pipeline():
    from isaacteleop.retargeting_engine.deviceio_source_nodes import ControllersSource, HandsSource
    from isaacteleop.retargeting_engine.interface import OutputCombiner, ValueInput
    from isaacteleop.retargeters import (
        GripperRetargeter, GripperRetargeterConfig,
        Se3AbsRetargeter, Se3RetargeterConfig,
        TensorReorderer,
    )
    from isaacteleop.retargeting_engine.tensor_types import TransformMatrix
    # 1. Create input sources
    controllers = ControllersSource(name="controllers")
    hands = HandsSource(name="hands")
    # 2. Apply coordinate-frame transform (world_T_anchor provided by IsaacTeleopDevice)
    transform_input = ValueInput("world_T_anchor", TransformMatrix())
    transformed_controllers = controllers.transformed(
        transform_input.output(ValueInput.VALUE)
    )
    # 3. Create and connect retargeters
    se3_cfg = Se3RetargeterConfig(
        input_device=ControllersSource.RIGHT,
        target_offset_roll=90.0,
    )
    se3 = Se3AbsRetargeter(se3_cfg, name="ee_pose")
    connected_se3 = se3.connect({
        ControllersSource.RIGHT: transformed_controllers.output(ControllersSource.RIGHT),
    })
    gripper_cfg = GripperRetargeterConfig(hand_side="right")
    gripper = GripperRetargeter(gripper_cfg, name="gripper")
    connected_gripper = gripper.connect({
        ControllersSource.RIGHT: transformed_controllers.output(ControllersSource.RIGHT),
        HandsSource.RIGHT: hands.output(HandsSource.RIGHT),
    })
    # 4. Flatten into a single action tensor with TensorReorderer
    ee_elements = ["pos_x", "pos_y", "pos_z", "quat_x", "quat_y", "quat_z", "quat_w"]
    reorderer = TensorReorderer(
        input_config={
            "ee_pose": ee_elements,
            "gripper_command": ["gripper_value"],
        },
        output_order=ee_elements + ["gripper_value"],
        name="action_reorderer",
        input_types={"ee_pose": "array", "gripper_command": "scalar"},
    )
    connected_reorderer = reorderer.connect({
        "ee_pose": connected_se3.output("ee_pose"),
        "gripper_command": connected_gripper.output("gripper_command"),
    })
    # 5. Return OutputCombiner with "action" key
    return OutputCombiner({"action": connected_reorderer.output("output")})

Configure Your Environment

Register the pipeline in your environment configuration using ~isaaclab_teleop.IsaacTeleopCfg:

from isaaclab_teleop import IsaacTeleopCfg, XrCfg
@configclass
class MyTeleopEnvCfg(ManagerBasedRLEnvCfg):
    xr: XrCfg = XrCfg(anchor_pos=(0.5, 0.0, 0.5))
    def __post_init__(self):
        super().__post_init__()
        self.isaac_teleop = IsaacTeleopCfg(
            pipeline_builder=_build_my_pipeline,
            sim_device=self.sim.device,
            xr_cfg=self.xr,
        )

Key IsaacTeleopCfg fields:

• pipeline_builder — callable that returns an OutputCombiner with an "action" output.
• retargeters_to_tune — optional callable returning retargeters to expose in the live tuning UI.
• xr_cfg — ~isaaclab_teleop.XrCfg for anchor configuration (see below).
• xr_camera_feeds — ordered selection and per-feed panel settings for existing task cameras. The list is empty by default, so tasks opt in to PiP explicitly.
• xr_camera_feed_layout — viewer reference, fixed world pose, and manual, horizontal, vertical, or grid panel packing.
• plugins — list of Isaac Capture plugin configurations (e.g. Manus).
• sim_device — torch device string (default "cuda:0").
• retargeting_execution — Isaac Capture retargeting execution settings. Defaults to RetargetingExecutionConfig(mode="pipelined") with DeadlinePacingConfig(safety_margin_s=0.025) so retargeting can run on the Isaac Capture worker instead of blocking the simulation loop. The 25 ms safety margin staggers Isaac Capture’s Python work behind Isaac Lab’s step Python, giving native work such as rendering time to overlap instead of having both Python stacks contend for the GIL at the start of the step.

CloudXR Environment Profiles

Isaac Lab ships three .env profiles that configure the CloudXR runtime — two for XR devices and one for headless standalone use. These are bundled inside the isaaclab_teleop package and can be referenced via constants:

All three profiles set NV_CXR_ENABLE_PUSH_DEVICES=0, which is correct for headset optical hand tracking (the most common setup). For external push-device peripherals such as Manus gloves, set this to 1 in a custom profile (see below). They also set NV_ENABLE_POSE_WAIT=0 so CloudXR does not throttle the application when frame times spike. This favors lower latency over CloudXR’s pose-wait smoothing.

~isaaclab_teleop.CLOUDXR_STANDALONE_ENV is the default when running without --xr (see Run Without a Headset (Standalone I/O)). It pins an emulated quest3 device profile so a CloudXR runtime with no client still advertises an OpenXR system, working around XR_ERROR_FORM_FACTOR_UNAVAILABLE.

Override at launch time

The --cloudxr_env flag on teleop_se3_agent.py and record_demos.py selects which .env profile to use. When unset it defaults to cloudxrjs (Quest/Pico) with --xr and standalone without --xr. Use the avp shorthand for Apple Vision Pro, standalone for the headless profile, or pass a full file path for a custom profile:

# Use the AVP profile
uv run --extra teleop,isaacsim isaaclab teleop run \
    --task IsaacContrib-PickPlace-GR1T2-WaistEnabled-Abs \
    --visualizer kit --xr \
    --cloudxr_env avp

Create a custom profile

Copy a shipped profile and edit it:

# Start from the Quest/Pico profile
cp $(uv run --extra teleop,isaacsim python -c \
    "from isaaclab_teleop import CLOUDXR_JS_ENV; print(CLOUDXR_JS_ENV)") ~/my-cloudxr.env

Edit ~/my-cloudxr.env to change any values (e.g. NV_CXR_ENABLE_PUSH_DEVICES=1 for Manus gloves), then pass it via --cloudxr_env ~/my-cloudxr.env.

Disable auto-launch

If you prefer to run the CloudXR runtime manually in a separate terminal (python -m isaacteleop.cloudxr), you can disable auto-launch in several ways:

• CLI flag: --no-auto_launch_cloudxr on the teleop script.
• Disable CloudXR entirely: --cloudxr_env none.
• Environment variable: ISAACLAB_CXR_SKIP_AUTOLAUNCH=1 overrides the CLI flag at runtime.

# Disable via CLI flag
uv run --extra teleop,isaacsim isaaclab teleop run \
    --task IsaacContrib-PickPlace-Locomanipulation-G1-Abs \
    --visualizer kit --xr \
    --no-auto_launch_cloudxr
# Or disable via environment variable
ISAACLAB_CXR_SKIP_AUTOLAUNCH=1 uv run --extra teleop,isaacsim isaaclab teleop run \
    --task IsaacContrib-PickPlace-Locomanipulation-G1-Abs \
    --visualizer kit --xr

Accept the CloudXR license non-interactively

The NVIDIA CloudXR license is separate from the Omniverse one. The first time the runtime starts it asks for it on stdin:

NVIDIA CloudXR EULA must be accepted to run. View: <license URL>
Accept NVIDIA CloudXR EULA? [y/N]:

There is no terminal to answer on in a headless, container or CI run, so the launch fails with RuntimeError: CloudXR EULA was not accepted; cannot start the runtime. Set ISAACLAB_CXR_ACCEPT_EULA=1 to accept it up front, the same way OMNI_KIT_ACCEPT_EULA works for the Omniverse license:

ISAACLAB_CXR_ACCEPT_EULA=1 ./isaaclab.sh -p scripts/environments/teleoperation/teleop_se3_agent.py \
    --task IsaacContrib-PickPlace-Locomanipulation-G1-Abs \
    --xr

y, yes and 1 accept it, case-insensitively and ignoring surrounding whitespace — the same spellings OMNI_KIT_ACCEPT_EULA takes; leaving the variable unset, or setting any other value, keeps the interactive prompt. Acceptance is recorded in ~/.cloudxr/run/eula_accepted, so once the license has been accepted — interactively or through this variable — later runs no longer prompt. The variable applies to every script that launches the runtime, including the process-scoped launcher in teleop_replay_agent.py.

Configure the XR Anchor

The ~isaaclab_teleop.XrCfg controls how the simulation is positioned and oriented in the XR device’s view.

anchor_pos / anchor_rot Static anchor placement. The simulation point at these coordinates appears at the XR device’s local origin (floor level). Set to a point on the floor beneath the robot to position it in front of the user.

anchor_prim_path Attach the anchor to a USD prim for dynamic positioning. Use this for locomotion tasks where the robot moves and the XR camera should follow.

anchor_rotation_mode Controls how anchor rotation behaves:

fixed_anchor_height When True (default), keeps the anchor height at its initial value. Prevents vertical bobbing during locomotion.

near_plane Closest render distance for the XR device (default 0.15 m).

XR Camera Feedback

An ordered list of ~isaaclab_teleop.XrCameraFeedCfg objects selects existing task scene cameras. The manager publishes each new RGBA frame after rendering, while ~isaaclab_teleop.XrCameraFeedLayoutCfg places the panels manually or in horizontal, vertical, and grid layouts. The following registered tasks enable PiP by default:

• IsaacContrib-PickPlace-GR1T2-Abs
• IsaacContrib-NutPour-GR1T2-Pink-IK-Abs
• IsaacContrib-ExhaustPipe-GR1T2-Pink-IK-Abs
• IsaacContrib-PickPlace-Locomanipulation-G1-Abs
• IsaacContrib-PickPlace-FixedBaseUpperBodyIK-G1-Abs

Both G1 tasks present the calibrated head camera in a head_locked panel that follows headset position and orientation. Locomanipulation retains the camera as a recorded policy observation; fixed-base G1 uses it only for PiP and keeps its policy observation schema unchanged.

To run fixed-base G1 upper-body teleoperation with its default head-locked PiP:

uv run --extra teleop,isaacsim isaaclab teleop run \
    --task IsaacContrib-PickPlace-FixedBaseUpperBodyIK-G1-Abs \
    --num_envs 1 --xr --device cpu

This task uses motion controllers for the arms and TriHand fingers while the robot root remains fixed. The panel follows the headset, but its image comes from the robot’s head camera, not the headset view. --device cpu selects CPU simulation; camera rendering and CloudXR encoding still use the GPU.

teleop_se3_agent.py and record_demos.py show every enabled feed when an Isaac Capture-enabled environment runs with --xr. PiP is absent unless the task explicitly selects an existing CameraCfg through xr_camera_feeds. Except for fixed-base G1, the selected robot_pov_cam is also a policy image observation, so the normal demonstration recorder stores the same view shown to the operator. Each camera is parented to a physical robot body link, so the camera view follows robot motion:

Both G1 tasks enable isaaclab_teleop.XrCameraFeedLayoutCfg.use_scene_partition to avoid this recursion. Before environment construction, preparation temporarily disables per-environment partitioning on selected Isaac RTX camera renderers. Binding sets /rtx/scenePartitioning/showAllPartitionsByDefault=False after renderer initialization and before creating panels. Geometry must start as shared, unpartitioned background; clearing its partition after camera initialization can hide it from XR. Other scene cameras must also set enable_scene_partitioning=False.

The SceneUI adapter assigns the XR camera and /ui to isaaclab_teleop_xr_camera_pip and refreshes inheritance when SceneUI children appear. Robot cameras see shared background without UI; XR sees both. XR/UI overrides use the session layer. Prior camera, renderer, and partition settings are restored after their final owner closes, unless changed externally. Camera configuration is restored after the last prepared session closes; the global setting’s bind-time value is restored after the last bound session closes. Prepared sessions must close even if environment construction fails; the teleoperation and recording scripts handle this cleanup. Panels remain hidden while tracking or isolation is unavailable and reappear after recovery. Other tasks retain their existing behavior because use_scene_partition defaults to False.

uv run --extra teleop,isaacsim isaaclab teleop run \
    --task IsaacContrib-PickPlace-GR1T2-Abs \
    --xr --device cpu

XR camera PiP currently supports exactly one environment. When a task has enabled PiP feeds, startup rejects --num_envs values other than 1; Isaac Capture XR behavior without PiP is unchanged. For either G1 task, set env.isaac_teleop.xr_camera_feeds=[] to disable PiP and retain the camera sensor and any configured camera observations.

The reference feeds request render-product-local DLSS Ray Reconstruction and quality execution mode through ~isaaclab_teleop.XrCameraFeedCfg. The private PiP adapter authors those two attributes only on the selected camera render product. Ray Reconstruction also requires the process-global responsive-denoising setting, which the teleoperation and recording scripts enable before environment construction when a selected feed requests Ray Reconstruction.

Camera selection

Tasks declare their default selection through IsaacTeleopCfg.xr_camera_feeds:

from isaaclab_teleop import IsaacTeleopCfg, XrCameraFeedCfg
self.isaac_teleop = IsaacTeleopCfg(
    pipeline_builder=_build_my_pipeline,
    xr_camera_feeds=[
        XrCameraFeedCfg(
            camera_name="left_wrist_camera",
            enable_dlss_ray_reconstruction=True,
            dlss_exec_mode="quality",
        ),
        XrCameraFeedCfg(camera_name="overview_camera", enabled=False),
    ],
)

For a recorded training view, define the named CameraCfg in the task scene and a matching mdp.image_rgb term in observations.policy. The normal recorder then stores that observation, while the PiP declaration above only selects it for presentation. Enabled task-declared entries control camera selection, panel count, display order, and automatic-layout order without mutating the reusable task configuration.

Placement

The default isaaclab_teleop.XrCameraFeedLayoutCfg.placement is "viewer_start". The presenter waits for the first valid headset pose, captures its eye position and upright yaw, then places the layout 0.8 m ahead at eye height. That pose remains fixed in the world while the user moves. Panels hide when the XR display disconnects and capture a new starting pose after reconnect; resetting the environment does not recenter them.

Three placement references are available:

from isaaclab_teleop import XrCameraFeedLayoutCfg
# Default: fixed in the world at the user's starting eye pose.
self.isaac_teleop.xr_camera_feed_layout = XrCameraFeedLayoutCfg()
# Follow the headset and pack feeds left to right.
self.isaac_teleop.xr_camera_feed_layout = XrCameraFeedLayoutCfg(
    placement="head_locked",
    mode="horizontal",
    distance_m=0.8,
)
# Place a grid in an Isaac Lab Z-up world. This example assumes an eye at
# (0, 0, 1.6) looking along world +Y.
self.isaac_teleop.xr_camera_feed_layout = XrCameraFeedLayoutCfg(
    placement="world",
    mode="grid",
    world_position_m=(0.0, 0.8, 1.6),
    world_orientation_xyzw=(0.7071067812, 0.0, 0.0, 0.7071067812),
    max_columns=2,
)

manual mode preserves each feed’s offset_m and distance_m. horizontal, vertical, and grid modes pack enabled feeds around center_offset_m using panel_gap_m; grid mode also honors max_columns. Offsets are measured in the selected layout plane. With viewer_start and head_locked, distance_m places that plane in front of the viewer reference. In manual mode, feeds with identical offsets overlap; select an automatic mode or assign distinct feed offsets when showing multiple panels.

With world, the explicit pose is measured in the Isaac Lab USD stage world in meters, not OpenXR physical space. Isaac Lab stages are Z-up. world_orientation_xyzw maps panel-local coordinates into that world: local +X is image right, local +Y is image up, and local +Z points from the readable face toward the viewer. Feed and automatic-layout offsets are applied in the panel’s local XY plane.

Disable cameras and PiP

Set isaaclab_teleop.XrCameraFeedCfg.enabled to False to suppress one feed, or set xr_camera_feeds=[] to suppress all PiP. The list is empty by default. These choices do not remove camera sensors independently owned by the task.

--disable_external_cameras is the master camera-rendering switch for teleop_se3_agent.py and record_demos.py. It strips camera sensors and suppresses every configured PiP feed.

Kit Scene UI presentation

PiP presentation uses Kit Scene UI and SpatialSource placement. Kit imports are deferred until an enabled feed is requested. If the Scene UI modules cannot be imported, the scripts log a warning and continue without PiP; task-owned cameras and recording observations remain unchanged. This keeps the camera selection configuration usable when a future kitless entry point no longer provides Scene UI. Render-product tuning is also best-effort: backends without a compatible render product keep the Camera-buffer path, and schema-authoring failures warn while PiP continues. Other configuration, camera-buffer, and panel-initialization errors still fail during startup.

Haptic Feedback

Isaac Capture can render haptic feedback on the operator’s device from a sim-side signal, closing the loop on grasp feel during teleoperation and demonstration recording. Two backends ship today:

• Controller vibration — a motion controller rumbles in proportion to the total contact force on the corresponding robot hand.
• Haptic glove — each finger of a haptic glove vibrates in proportion to how tightly that finger grips the manipulated object.

Feedback is an output path that mirrors the input retargeting pipeline in reverse. Each step, a ~isaaclab_teleop.HapticFeedbackDriver reads a per-hand signal vector from the scene and pushes it to the ~isaaclab_teleop.IsaacTeleopDevice, which renders it through an Isaac Capture HapticSink. The signal-to-device mapping runs inside the retargeting graph, so no device-specific code lives in your environment.

The design keeps three concerns pluggable, so the same seam serves both devices:

• Signal source — how the per-hand vector is read from the environment.
• Output port — the ~isaaclab_teleop.HapticFeedbackReceiver protocol, whose payload is a vector (one scalar for a rumble motor, one value per finger for a glove).
• Device backend — selected by the concrete ~isaaclab_teleop.HapticFeedbackCfg subclass, which wires the retargeter and device behind the HapticSink.

To enable feedback, add per-hand ~isaaclab.sensors.ContactSensor s to the scene and attach a ~isaaclab_teleop.HapticFeedbackCfg subclass as a sibling of isaac_teleop. The teleop scripts (teleop_se3_agent.py and record_demos.py) detect the haptic_feedback attribute automatically — no extra flags are needed. If the active device cannot render haptics (e.g. keyboard) or the env has no haptic_feedback, the feature is silently skipped.

Controller vibration

Use ~isaaclab_teleop.ControllerHapticFeedbackCfg. It sums each hand’s contact force to a single amplitude and maps it via amplitude = clamp(gain * (force - deadband), 0, saturation) (fields gain [1/N], deadband [N], saturation, plus frequency_hz / duration_s for the pulse).

from isaaclab.sensors import ContactSensorCfg
from isaaclab_teleop import ControllerHapticFeedbackCfg
@configclass
class MySceneCfg(InteractiveSceneCfg):
    left_hand_contact = ContactSensorCfg(
        prim_path="{ENV_REGEX_NS}/Robot/left_hand_.*_link", update_period=0.0, history_length=3
    )
    right_hand_contact = ContactSensorCfg(
        prim_path="{ENV_REGEX_NS}/Robot/right_hand_.*_link", update_period=0.0, history_length=3
    )
# in the env cfg __post_init__:
self.scene.robot.spawn.activate_contact_sensors = True
self.haptic_feedback = ControllerHapticFeedbackCfg(
    left_sensor_name="left_hand_contact",
    right_sensor_name="right_hand_contact",
)

Enabled on the two G1 loco-manipulation teleop environments (IsaacContrib-PickPlace-Locomanipulation-G1-Abs, IsaacContrib-PickPlace-FixedBaseUpperBodyIK-G1-Abs). Controller vibration is delivered over OpenXR (xrApplyHapticFeedback), so no CloudXR .env profile change is required.

Haptic glove (per-finger grip)

Use ~isaaclab_teleop.GloveHapticFeedbackCfg. Each hand’s contact sensor is filtered against the grasped object (filter_prim_paths_expr), so normal_force_matrix_w reports each finger’s force on that object and nothing else. The driver orders the per-finger magnitudes Thumb..Pinky (matched from the sensor’s body names via finger_order) into a FingerPowerVector rendered by a cross-process glove device on collection_id.

from isaaclab.sensors import ContactSensorCfg
from isaaclab_teleop import GloveHapticFeedbackCfg
@configclass
class MySceneCfg(InteractiveSceneCfg):
    object = RigidObjectCfg(prim_path="{ENV_REGEX_NS}/Object", ...)
    left_hand_contact = ContactSensorCfg(
        prim_path="{ENV_REGEX_NS}/Robot/.*L_(thumb_distal|index_intermediate|"
        "middle_intermediate|ring_intermediate|pinky_intermediate)_link",
        filter_prim_paths_expr=["{ENV_REGEX_NS}/Object"],
        update_period=0.0, history_length=3,
    )
    right_hand_contact = ContactSensorCfg(
        prim_path="{ENV_REGEX_NS}/Robot/.*R_(...)_link",
        filter_prim_paths_expr=["{ENV_REGEX_NS}/Object"],
        update_period=0.0, history_length=3,
    )
# in the env cfg __post_init__:
self.haptic_feedback = GloveHapticFeedbackCfg(
    left_sensor_name="left_hand_contact",
    right_sensor_name="right_hand_contact",
)

Enabled on the two GR1T2 pick-place teleop environments (IsaacContrib-PickPlace-GR1T2-Abs, IsaacContrib-PickPlace-GR1T2-WaistEnabled-Abs).

Record Demonstrations for Imitation Learning

Isaac Capture integrates with Isaac Lab’s record_demos.py script for recording teleoperated demonstrations, exposed as the isaaclab teleop record command.

When your environment configuration has an isaac_teleop attribute, the script automatically uses create_isaac_teleop_device() — no --teleop_device flag is needed:

uv run --extra teleop,isaacsim isaaclab teleop record \
    --task IsaacContrib-PickPlace-Locomanipulation-G1-Abs \
    --visualizer kit \
    --xr

The run, record, and replay workflows accept task configuration selectors using Hydra-style key=value syntax (with no leading --). For example, record the Franka reach task with the PhysX and differential IK presets as follows:

uv run --extra teleop,isaacsim isaaclab teleop record \
    --task Isaac-Reach-Franka \
    --visualizer kit \
    --dataset_file ./datasets/dataset.hdf5 \
    --num_demos 1 \
    --teleop_device keyboard \
    physics=isaacsim_physx presets=diffik

When replaying a preset-configured dataset, pass the same selectors again. The HDF5 metadata stores the registered task ID, but not the command-line selector values:

uv run --extra teleop,isaacsim isaaclab teleop replay \
    --task Isaac-Reach-Franka \
    --visualizer kit \
    --dataset_file ./datasets/dataset.hdf5 \
    physics=isaacsim_physx presets=diffik

Some environments use the legacy teleop_devices configuration instead of isaac_teleop (e.g. the Galbot RmpFlow relative-mode tasks). For these, pass --teleop_device to select the input device:

uv run --extra teleop,isaacsim isaaclab teleop record \
    --task IsaacContrib-Stack-Cube-Galbot-Left-Arm-Gripper-RmpFlow \
    --visualizer kit \
    --teleop_device keyboard

The workflow is:

1. Configure your environment with IsaacTeleopCfg (see Configure Your Environment) or teleop_devices for legacy devices (keyboard, spacemouse).
2. Run record_demos.py with the task name and any physics=, renderer=, or presets= selectors required by the task.
3. For XR tasks: start AR, connect your XR device, and teleoperate. For legacy tasks: use the configured input device directly.
4. Demonstrations are recorded to HDF5 files.
5. Use the recorded data with Isaac Lab Mimic or other imitation learning frameworks.

For the broader imitation learning pipeline (replay, augmentation, policy training), see Synthetic Data Generation and Imitation Learning with Isaac Lab Mimic.

Add a New Robot

To add teleoperation support for a new robot in Isaac Lab:

1. Choose a control scheme. Refer to the Choose a Control Scheme table to determine which retargeters match your robot’s capabilities.

2. Build the pipeline. If existing retargeters are sufficient (e.g. Se3AbsRetargeter + GripperRetargeter for a new manipulator), write a pipeline builder function following the pattern in Build a Retargeting Pipeline. Configure the TensorReorderer output order to match your environment’s action space.

3. For dexterous hands: create a robot hand URDF and YAML config for DexHandRetargeter. Ensure fingertip links are positioned at the actual fingertips, not mid-finger.

4. For a custom retargeter: see Add a New Retargeter below.

5. Configure the XR anchor for your robot (static for manipulation, dynamic for locomotion). See Configure the XR Anchor.

6. Register in env config via IsaacTeleopCfg (see Configure Your Environment).

Add a New Retargeter

If the built-in retargeters do not cover your use case, you can implement a custom one in the Isaac Capture repository:

1. Inherit from BaseRetargeter and implement input_spec(), output_spec(), and compute().
2. Optionally add a ParameterState for parameters that should be live-tunable via the retargeter tuning UI.
3. Connect to existing source nodes (HandsSource, ControllersSource) or create a new IDeviceIOSource subclass for custom input devices.

See the Isaac Capture repository and Contributing Guide for details.

Add a New Device

There are two levels of device integration:

Isaac Capture plugin (C++ level) For new hardware that requires a custom driver or SDK. Plugins push data via OpenXR tensor collections. Existing plugins include Manus gloves, Haptikos exoskeletons, OAK-D camera, controller synthetic hands, and foot pedals. After creating the plugin, update the retargeting pipeline config to consume data from the new plugin’s source node.

See the Plugins directory for examples.

Pipeline configuration only For devices already supported by Isaac Capture (or whose data is available as hand / controller tracking). Simply update your pipeline_builder to use the appropriate source nodes and retargeters for the device’s data format.

Optimize XR Performance

Ensure the simulation render time step roughly matches the XR device’s display rate and can be sustained in real time. Quest 3 and Pico 4 Ultra typically run at 90 Hz, so we recommend a simulation dt of 90 Hz with a render_interval of 2 (rendering at 45 Hz):

@configclass
class XrTeleopEnvCfg(ManagerBasedRLEnvCfg):
    def __post_init__(self):
        self.sim.dt = 1.0 / 90        # physics steps at 90 Hz
        self.sim.render_interval = 2  # one render per 2 physics steps -> 45 Hz

sim.render_interval is the number of physics simulation steps that occur between renders. Increasing it reduces rendering frequency (and GPU cost) without changing physics behavior — useful when physics can keep up but rendering cannot.

The choice of sim.dt is a trade-off between stability and performance: a smaller dt (e.g. 1.0 / 120) integrates contacts more accurately and is more stable for stiff contact-rich tasks, but each step costs more wall-clock time and lowers achievable frame rate. A larger dt (e.g. 1.0 / 60) is cheaper but can introduce contact jitter or instabilities. Pick the largest dt your task tolerates.

The RTX - Minimal renderer trades image fidelity for substantially lower per-frame GPU cost. It is the recommended choice when the simulation cannot sustain the XR device’s display rate in real time — for example on lower-spec GPUs, in scenes with many lights or complex materials, or when you have already configured sim.dt and sim.render_interval and still see dropped frames.

To enable it, click the renderer dropdown at the top-left of the Isaac Lab viewport and select RTX - Minimal:

For best results, open Render Settings from the top-right of the Isaac Lab UI, switch to the Minimal tab, and set Minimal Shading Mode to Diffuse/Glossy/Emission:

import isaaclab.sim as sim_utils
from isaaclab.assets import AssetBaseCfg
light = AssetBaseCfg(
    prim_path="/World/light",
    spawn=sim_utils.DistantLightCfg(color=(0.75, 0.75, 0.75), intensity=3000.0),
)

import isaaclab.sim as sim_utils
from isaaclab.assets import AssetBaseCfg
light = AssetBaseCfg(
    prim_path="/World/light",
    spawn=sim_utils.DistantLightCfg(color=(0.75, 0.75, 0.75), intensity=3000.0),
    init_state=AssetBaseCfg.InitialStateCfg(
        rot=(0.0, 0.0, 0.0, 1.0),  # quaternion (x, y, z, w); adjust to reduce shadow overlap
    ),
)

The XR render resolution multiplier scales the size of the render buffers that are then upscaled to the headset’s recommended display resolution. Lowering it trades image sharpness for substantially lower per-frame GPU cost, which can help sustain real-time frame rates on lower-spec GPUs or in heavy scenes.

In the Isaac Lab UI, open the XR tab on the right-side panel, expand Advanced Settings -> Render Resolution, and drag the Resolution Multiplier slider:

A value around 0.8 is usually a good starting point: noticeable GPU savings with minimal perceptible quality loss. Reduce further only if you still cannot hit the headset’s display rate.

The teleop scripts (teleop_se3_agent.py, record_demos.py, and teleop_replay_agent.py) render external camera sensors by default. Camera render products add significant GPU cost and contend with the XR view, so if your task does not need camera observations during teleoperation, disable them with --disable_external_cameras:

uv run --extra teleop,isaacsim isaaclab teleop run \
    --task IsaacContrib-PickPlace-Locomanipulation-G1-Abs \
    --visualizer kit --xr \
    --disable_external_cameras

The flag strips the environment’s camera sensors (equivalent to calling ~isaaclab_teleop.remove_camera_configs on the env config) and selects a lighter Kit experience. For teleop_se3_agent.py it is also the master PiP gate: task-default and task-configured feeds are ignored and no camera panel is created. Omit it to keep cameras enabled (the default) — required when recording camera observations, showing XR camera feedback, or when you want teleop_replay_agent.py to mirror the production render load.

Passing --visualizer kit opens the local Kit viewport, which renders a window on the host in addition to the XR view. On a server or cloud instance — or whenever you do not need the local window — run headless so only the XR view is rendered, freeing GPU for the headset:

uv run --extra teleop,isaacsim isaaclab teleop run \
    --task IsaacContrib-PickPlace-Locomanipulation-G1-Abs \
    --viz none --xr

In headless XR the OpenXR/AR session starts automatically — there is no local viewport to click Start XR, so Isaac Lab begins streaming as soon as a CloudXR client connects.

Isaac Capture can run retargeting either synchronously on the application thread or asynchronously through a pipelined worker. This is controlled by RetargetingExecutionConfig.

In synchronous mode, retargeting runs inline with the simulation step. This can be the best choice for lightweight retargeting or retargeting implemented mostly in Python, since a background Python worker can still contend with the application thread through the GIL.

In pipelined mode, Isaac Capture submits retargeting work to a background worker and the application uses the most recent completed result. This is useful when retargeting has enough native work to overlap with simulation or rendering, or when the retargeting cost is large enough that running it inline would directly extend the frame.

retargeting_execution=RetargetingExecutionConfig(
    mode="pipelined",
    pacing=DeadlinePacingConfig(safety_margin_s=0.025),
)

DeadlinePacingConfig intentionally delays the background retargeting work until closer to when the next result is needed, instead of starting it immediately when the request is submitted. This helps avoid competing with the Python work Isaac Lab performs at the beginning of the frame, and tends to line the retargeting work up with rendering or other native work where overlap is more useful.

The safety_margin_s value controls how early retargeting starts before the predicted deadline. A larger margin starts retargeting earlier, which gives heavier or more variable retargeting work more time to finish before the next frame consumes the result. The trade-off is that the input sample may be slightly older, and Python-heavy retargeting may introduce more GIL contention.

If retargeting is mostly Python and lightweight, consider mode="sync". If retargeting performs substantial native work or has occasional long spikes, use mode="pipelined" and increase safety_margin_s so the work starts earlier.

The CloudXR Runtime frame pacer attempts to keep the client experience smooth. If the application has repeated frame-time spikes, the pacer may settle at a lower stable frame rate instead of oscillating between rates. This can make a connected client appear slower even when Isaac Lab profiling does not show a proportional simulation-side regression.

The shipped CloudXR profiles set NV_ENABLE_POSE_WAIT=0 to mitigate this case, favoring lower latency over pose-wait smoothing. If you use a custom .env file, copy that setting into the custom profile, then point teleop_se3_agent.py or record_demos.py at it with --cloudxr_env. See CloudXR Environment Profiles for the profile override workflow.

Known Issues

• OpenXR runtime error messages (XR_ERROR_...) are hidden by default

  Isaac Lab sets OXR_NO_PRINTING=true for teleoperation sessions. Set OXR_NO_PRINTING=false before launching to see them when debugging an XR connection.

• XR_ERROR_VALIDATION_FAILURE: xrWaitFrame(frameState->type == 0) when stopping AR Mode

  Can be safely ignored. Caused by a race condition in the exit handler.

• XR_ERROR_INSTANCE_LOST in xrPollEvent

  Occurs if the CloudXR runtime exits before Isaac Lab. Restart the runtime to resume.

• [omni.usd] TF_PYTHON_EXCEPTION when starting/stopping AR Mode

  Can be safely ignored. Caused by a race condition in the enter/exit handler.

• Invalid version string in _ParseVersionString

  Caused by shader assets authored with older USD versions. Typically safe to ignore.

• XR device connects but no video is displayed (viewport responds to tracking)

  The GPU index may differ between host and container. Set NV_GPU_INDEX to 0, 1, or 2 in the runtime to match the host GPU.

API Reference

See the isaaclab_teleop for full class and function documentation:

• ~isaaclab_teleop.IsaacTeleopCfg

• ~isaaclab_teleop.IsaacTeleopDevice

• ~isaaclab_teleop.create_isaac_teleop_device

• ~isaaclab_teleop.ControlEvents

• ~isaaclab_teleop.SupportsControlEvents

• ~isaaclab_teleop.poll_control_events

• ~isaaclab_teleop.TELEOP_CONTROL_CHANNEL_UUID

• ~isaaclab_teleop.HapticFeedbackCfg

• ~isaaclab_teleop.ControllerHapticFeedbackCfg

• ~isaaclab_teleop.GloveHapticFeedbackCfg

• ~isaaclab_teleop.HapticFeedbackReceiver

• ~isaaclab_teleop.HapticFeedbackDriver

• ~isaaclab_teleop.create_haptic_feedback_driver

• ~isaaclab_teleop.XrCameraFeedCfg

• ~isaaclab_teleop.XrCameraFeedLayoutCfg

• ~isaaclab_teleop.XrCfg

• ~isaaclab_teleop.XrAnchorRotationMode

  References