SkillGen for Automated Demonstration Generation

SkillGen is an advanced demonstration generation system that enhances Isaac Lab Mimic by integrating motion planning. It generates high-quality, adaptive, collision-free robot demonstrations by combining human-provided subtask segments with automated motion planning.

What is SkillGen?

SkillGen addresses key limitations in traditional demonstration generation:

• Motion Quality: Uses cuRobo’s GPU-accelerated motion planner to generate smooth, collision-free trajectories
• Validity: Generates kinematically feasible plans between skill segments
• Diversity: Generates varied demonstrations through configurable sampling and planning parameters
• Adaptability: Generates demonstrations that can be adapted to new object placements and scene configurations during data generation

The system works by taking manually annotated human demonstrations, extracting localized subtask skills (see Subtasks in SkillGen_), and using cuRobo to plan feasible motions between these skill segments while respecting robot kinematics and collision constraints.

Prerequisites

Before using SkillGen, you must understand:

1. Teleoperation: How to control robots and record demonstrations using keyboard, SpaceMouse, or hand tracking
2. Isaac Lab Mimic: The complete workflow including data collection, annotation, generation, and policy training

Installation

SkillGen requires Isaac Lab, Isaac Sim, and cuRobo. Follow these steps in your Isaac Lab conda environment.

Step 1: Install and verify Isaac Sim and Isaac Lab

Follow the official Isaac Sim and Isaac Lab installation guide here.

Step 2: Install cuRobo

cuRobo provides the motion planning capabilities for SkillGen. This installation is tested to work with Isaac Lab’s PyTorch and CUDA requirements:

# One line installation of cuRobo (formatted for readability)
conda install -c nvidia cuda-toolkit=12.8 -y && \
export CUDA_HOME="$CONDA_PREFIX" && \
export PATH="$CUDA_HOME/bin:$PATH" && \
export LD_LIBRARY_PATH="$CUDA_HOME/lib:$LD_LIBRARY_PATH" && \
export TORCH_CUDA_ARCH_LIST="8.0+PTX" && \
pip install -e "git+https://github.com/NVlabs/curobo.git@ebb71702f3f70e767f40fd8e050674af0288abe8#egg=nvidia-curobo" --no-build-isolation

Step 3: Install Rerun

For trajectory visualization during development:

pip install rerun-sdk==0.23

Step 4: Verify Installation

Test that cuRobo works with Isaac Lab:

# This should run without import errors
python -c "import curobo; print('cuRobo installed successfully')"

conda config --env --set channel_priority strict
conda config --env --add channels conda-forge
conda install -y -c conda-forge "libstdcxx-ng>=12" "libgcc-ng>=12"

Download the SkillGen Dataset

We provide a pre-annotated dataset to help you get started quickly with SkillGen.

Dataset Contents

The dataset contains:

• Human demonstrations of Franka arm cube stacking
• Manually annotated subtask boundaries for each demonstration
• Compatible with both basic cube stacking and adaptive bin cube stacking tasks

Download and Setup

1. Download the pre-annotated dataset by clicking here.

2. Prepare the datasets directory and move the downloaded file:

# Make sure you are in the root directory of your Isaac Lab workspace
cd /path/to/your/IsaacLab
# Create the datasets directory if it does not exist
mkdir -p datasets
# Move the downloaded dataset into the datasets directory
mv /path/to/annotated_dataset_skillgen.hdf5 datasets/annotated_dataset_skillgen.hdf5

./isaaclab.sh -p scripts/tools/record_demos.py \
--task Isaac-Stack-Cube-Franka-IK-Rel-Skillgen-v0 \
--teleop_device spacemouse \
--dataset_file ./datasets/dataset_skillgen.hdf5 \
--num_demos 10

./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/annotate_demos.py \
--device cpu \
--task Isaac-Stack-Cube-Franka-IK-Rel-Skillgen-v0 \
--input_file ./datasets/dataset_skillgen.hdf5 \
--output_file ./datasets/annotated_dataset_skillgen.hdf5 \
--annotate_subtask_start_signals

Understanding Dataset Annotation

SkillGen requires datasets with annotated subtask start and termination boundaries. Auto-annotation is not supported.

Subtasks in SkillGen

Technical definition: A subtask is a contiguous demo segment that achieves a manipulation objective, defined via SubTaskConfig:

• object_ref: the object (or None) used as the spatial reference for this subtask
• subtask_term_signal: the binary termination signal name (transitions 0 to 1 when the subtask completes)
• subtask_start_signal: the binary start signal name (transitions 0 to 1 when the subtask begins; required for SkillGen)

The subtask localization process performs:

• detection of signal transition points (0 to 1) to identify subtask boundaries [t_start, t_end];
• extraction of the subtask segment between boundaries;
• computation of end-effector trajectories and key poses in an object- or task-relative frame (using object_ref if provided);

This converts absolute, scene-specific motions into object-relative skill segments that can be adapted to new object placements and scene configurations during data generation.

Manual Annotation Workflow

Contrary to the Isaac Lab Mimic workflow, SkillGen requires manual annotation of subtask start and termination boundaries. For example, for grasping a cube, the start signal is right before the gripper closes and the termination signal is right after the object is grasped. You can adjust the start and termination signals to fit your subtask definition.

Data Generation with SkillGen

SkillGen transforms annotated demonstrations into diverse, high-quality datasets using motion planning.

How SkillGen Works

The SkillGen pipeline uses your annotated dataset and the environment’s Mimic API to synthesize new demonstrations:

1. Subtask boundary use: Reads per-subtask start and termination indices from the annotated dataset
2. Goal sampling: Samples target poses per subtask according to task constraints and datagen config
3. Trajectory planning: Plans collision-free motions between subtask segments using cuRobo (when --use_skillgen)
4. Trajectory stitching: Stitches skill segments and planned trajectories into complete demonstrations.
5. Success evaluation: Validates task success terms; only successful trials are written to the output dataset

Usage Parameters

Key parameters for SkillGen data generation:

• --use_skillgen: Enables SkillGen planner (required)
• --generation_num_trials: Number of demonstrations to generate
• --num_envs: Parallel environments (tune based on GPU memory)
• --device: Computation device (cpu/cuda). Use cpu for stable physics
• --headless: Disable visualization for faster generation

Task 1: Basic Cube Stacking

Generate demonstrations for the standard Isaac Lab Mimic cube stacking task. In this task, the Franka robot must:

1. Pick up the red cube and place it on the blue cube
2. Pick up the green cube and place it on the red cube
3. Final stack order: blue (bottom), red (middle), green (top).

Small-Scale Generation

Start with a small dataset to verify everything works:

./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/generate_dataset.py \
--device cpu \
--num_envs 1 \
--generation_num_trials 10 \
--input_file ./datasets/annotated_dataset_skillgen.hdf5 \
--output_file ./datasets/generated_dataset_small_skillgen_cube_stack.hdf5 \
--task Isaac-Stack-Cube-Franka-IK-Rel-Skillgen-v0 \
--use_skillgen

Full-Scale Generation

Once satisfied with small-scale results, generate a full training dataset:

./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/generate_dataset.py \
--device cpu \
--headless \
--num_envs 1 \
--generation_num_trials 1000 \
--input_file ./datasets/annotated_dataset_skillgen.hdf5 \
--output_file ./datasets/generated_dataset_skillgen_cube_stack.hdf5 \
--task Isaac-Stack-Cube-Franka-IK-Rel-Skillgen-v0 \
--use_skillgen

Task 2: Adaptive Cube Stacking in a Bin

SkillGen can also be used to generate datasets for adaptive tasks. In this example, we generate a dataset for adaptive cube stacking in a narrow bin. The bin is placed at a fixed position and orientation in the workspace and a blue cube is placed at the center of the bin. The robot must generate successful demonstrations for stacking the red and green cubes on the blue cube without colliding with the bin.

Small-Scale Generation

Test the adaptive stacking setup:

./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/generate_dataset.py \
--device cpu \
--num_envs 1 \
--generation_num_trials 10 \
--input_file ./datasets/annotated_dataset_skillgen.hdf5 \
--output_file ./datasets/generated_dataset_small_skillgen_bin_cube_stack.hdf5 \
--task Isaac-Stack-Cube-Bin-Franka-IK-Rel-Mimic-v0 \
--use_skillgen

Full-Scale Generation

Generate the complete adaptive stacking dataset:

./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/generate_dataset.py \
--device cpu \
--headless \
--num_envs 1 \
--generation_num_trials 1000 \
--input_file ./datasets/annotated_dataset_skillgen.hdf5 \
--output_file ./datasets/generated_dataset_skillgen_bin_cube_stack.hdf5 \
--task Isaac-Stack-Cube-Bin-Franka-IK-Rel-Mimic-v0 \
--use_skillgen

Learning Policies from SkillGen Data

Similar to the Isaac Lab Mimic workflow, you can train imitation learning policies using the generated SkillGen datasets with Robomimic.

Basic Cube Stacking Policy

Train a state-based policy for the basic cube stacking task:

./isaaclab.sh -p scripts/imitation_learning/robomimic/train.py \
--task Isaac-Stack-Cube-Franka-IK-Rel-Skillgen-v0 \
--algo bc \
--dataset ./datasets/generated_dataset_skillgen_cube_stack.hdf5

Adaptive Bin Cube Stacking Policy

Train a policy for the more complex adaptive bin stacking:

./isaaclab.sh -p scripts/imitation_learning/robomimic/train.py \
--task Isaac-Stack-Cube-Bin-Franka-IK-Rel-Mimic-v0 \
--algo bc \
--dataset ./datasets/generated_dataset_skillgen_bin_cube_stack.hdf5

Evaluating Trained Policies

Test your trained policies:

# Basic cube stacking evaluation
./isaaclab.sh -p scripts/imitation_learning/robomimic/play.py \
--device cpu \
--task Isaac-Stack-Cube-Franka-IK-Rel-Skillgen-v0 \
--num_rollouts 50 \
--checkpoint /path/to/model_checkpoint.pth

# Adaptive bin cube stacking evaluation
./isaaclab.sh -p scripts/imitation_learning/robomimic/play.py \
--device cpu \
--task Isaac-Stack-Cube-Bin-Franka-IK-Rel-Mimic-v0 \
--num_rollouts 50 \
--checkpoint /path/to/model_checkpoint.pth

cuRobo Interface Features

This section summarizes the cuRobo planner interface and features. The SkillGen pipeline uses the cuRobo planner to generate collision-free motions between subtask segments. However, the user can use cuRobo as a standalone motion planner for your own tasks. The user can also implement their own motion planner by subclassing the base motion planner and implementing the same API.

Base Motion Planner (Extensible)

• Location: isaaclab_mimic/motion_planners/base_motion_planner.py
• Purpose: Uniform interface for all motion planners used by SkillGen
• Extensibility: New planners can be added by subclassing and implementing the same API; SkillGen consumes the API without code changes

cuRobo Planner (GPU, collision-aware)

• Location: isaaclab_mimic/motion_planners/curobo

• Multi-phase planning:

  • Retreat → Contact → Approach phases per subtask
  • Configurable collision filtering in contact phases
  • For SkillGen, retreat and approach phases are collision-free. The transit phase is collision-checked.

• World synchronization:

  • Updates robot state, attached objects, and collision spheres from the Isaac Lab scene each trial
  • Dynamic attach/detach of objects during grasp/place

• Collision representation:

  • Contact-aware sphere sets with per-phase enables/filters

• Outputs:

  • Time-parameterized, collision-checked trajectories for stitching

• Tests:

  • source/isaaclab_mimic/test/test_curobo_planner_cube_stack.py
  • source/isaaclab_mimic/test/test_curobo_planner_franka.py
  • source/isaaclab_mimic/test/test_generate_dataset_skillgen.py

Cube stack planner test.

Franka planner test.

These tests can also serve as a reference for how to use cuRobo as a standalone motion planner.

Generation Pipeline Integration

When --use_skillgen is enabled in generate_dataset.py, the following pipeline is executed:

1. Randomize subtask boundaries: Randomize per-demo start and termination indices for each subtask using task-configured offset ranges.

2. Build per-subtask trajectories: For each end-effector and subtask:

   • Select a source demonstration segment (strategy-driven; respects coordination/sequential constraints)
   • Transform the segment to the current scene (object-relative or coordination delta; optional first-pose interpolation)
   • Wrap the transformed segment into a waypoint trajectory

3. Transition between subtasks:

   • Plan a collision-aware transition with cuRobo to the subtask’s first waypoint (world sync, optional attach/detach), execute the planned waypoints, then resume the subtask trajectory

4. Execute with constraints:

   • Execute waypoints step-by-step across end-effectors while enforcing subtask constraints (sequential, coordination with synchronous steps); optionally update planner visualization if enabled

5. Record and export:

   • Accumulate states/observations/actions, set the episode success flag, and export the episode (the outer pipeline filters/consumes successes)

Visualization and Debugging

Users can visualize the planned trajectories and debug for collisions using Rerun-based plan visualizer. This can be enabled by setting visualize_plan = True in the cuRobo planner configuration. Note that rerun needs to be installed to visualize the planned trajectories. Refer to Step 3 in Installation for installation instructions.