Concepts Overview#

Overview#

This section covers the core concepts of the GR00T reference workflow.

This reference workflow packages supported models, libraries, frameworks, and a task into a single reproducible workflow for the Unitree G1 humanoid robot.

What Is the GR00T Reference Workflow#

The GR00T reference workflow is a structured workflow for Unitree G1 humanoid robot policy learning, spanning from simulation to real hardware.

It targets engineers who need to understand:

each stage of the pipeline
how those stages connect for a specific manipulation task
how to perform the workflow in practice

The workflow also serves as an end-to-end blueprint, not a fixed product stack that every team must adopt unchanged. Its modular design lets robotics teams use the full platform or integrate selected capabilities into existing development pipelines, helping them scale humanoid development without rebuilding the same infrastructure for each robot or task.

Robotics teams can start from validated development workflows, then adapt them for their own tasks, sensors, and humanoid configurations.

Diagram showing the architecture of the GR00T reference workflow. — The architecture diagram illustrates how each component in the reference workflow connects, from setup to deployment.#

What End-to-End Means#

Here is a high-level overview of what this workflow covers:

In this course, end-to-end refers to all major stages required to build a humanoid manipulation policy for a task, starting in simulation and ending on the real robot.

Environment setup establishes the software stack, task assets, robot embodiment, sensors, and workspace within a simulation environment.
Teleoperation and data collection records demonstrations in simulation or on the real robot.
Data conversion transforms demonstrations into LeRobot-format datasets.
GR00T Vision-Language-Action (VLA) post-training fine-tunes a GR00T policy for the apple pick-and-place task from collected demonstrations.
Evaluation in simulation measures the policy in Isaac Lab-Arena before hardware deployment.
Deployment on G1 packages and runs the post-trained VLA policy on the physical robot.

Conceptually, the reference workflow follows a sim-first to deployment flow for a given task. Each stage still needs clear validation.

Warning

A weak dataset, incorrect action mapping, unstable controller, or unrepeatable workspace can break the pipeline even if the surrounding tools are connected correctly.

How Humanoid Robots Learn Tasks#

In this workflow, humanoid robots learn manipulation tasks by combining demonstrations, perception, robot state, control, and policy learning.

A human operator demonstrates the desired behavior through teleoperation. The system records what the robot sees, where its body is, and which actions move the task forward. GR00T then fine-tunes a vision-language-action (VLA) policy on those demonstrations.

Humanoid manipulation adds another challenge because the robot must move its arms and hands while staying balanced. A standing humanoid must solve manipulation and stability at the same time, unlike a robotic manipulator arm with a fixed base.

GR00T Whole-Body Control (WBC) supports multiple controller options. AGILE is currently the recommended default for the GR00T end-to-end workflow.

In this workflow, AGILE is the baseline whole-body controller for standing and balance only, so the operator or learned policy can focus on upper-body task execution.

The Task: Apple Pick-and-Place#

In the target task for this course, the Unitree G1 stands at a table, picks up an apple, and places it on a plate.

The Unitree G1 standing at a table with an apple prop.

Why This Task#

The apple pick-and-place task exercises the full workflow without requiring complex multi-step reasoning. The task requires:

Stable standing and balance during reaching
Visual identification of the apple and plate
Coordinated arm and hand motion for grasping
Whole-body adjustment during the transfer
Release on the target

This task is intentionally bounded. It is complex enough to exercise perception, teleoperation, whole-body control, policy training, evaluation, and deployment, but simple enough to debug stage by stage.

With the big picture in place, the following sections examine each stage of the workflow in detail.

Environment Setup#

Everything in Its Place#

First, configure the robot and establish a workspace in both simulation and the real world.

Before collecting data, training a model, or evaluating a policy, the workflow requires a consistent task definition and a repeatable execution environment.

Simulation Setup#

In simulation, environment setup involves:

Configuring Isaac Lab-Arena
Defining the evaluation task
Registering the robot embodiment
Adding task objects
Setting up observation and action spaces

What Is Isaac Lab-Arena#

Isaac Lab-Arena is an open-source framework built on NVIDIA Isaac Lab for rapidly creating, managing, and evaluating large-scale embodied AI tasks with modular components, rich task variation, and scalable benchmarking tools.

This lets you define tasks from reusable pieces instead of creating each environment from scratch. It also supports GPU-accelerated evaluation across many simulated rollouts, task variations, objects, and environments.

Bringing in New Assets#

Bringing in new assets means adding task objects, robot configurations, scene elements, and observation sources in a way that the rest of the pipeline can consume. The asset setup must support the simulation task and the data schema used later for training.

The goal is not to make simulation and reality identical. The goal is to make each path explicit, inspectable, and repeatable so you can debug failures at the right layer.

Real-Robot Setup#

For the real robot, environment setup involves:

Validating workstation hardware
Confirming robot connectivity
Checking camera placement
Controlling lighting
Setting workspace boundaries
Placing objects in the work area

A small physical change can alter the data distribution, so the workspace needs to stay repeatable across collection and evaluation.

Data Collection#

High-quality demonstration data is necessary for training robust robot policies. This course uses teleoperation to collect demonstrations.

Teleoperation#

Teleoperation is the most common way to collect this task data. A human expert performs the task while the system records demonstrations for imitation learning.

Humanoids require more sophisticated teleoperation systems to capture the full range of motion required for complex tasks.

Humanoid tasks usually require upper-body control so that the operator can demonstrate reaching, grasping, and placement behavior.

In this workflow, teleoperation is scoped to the device and control paths described in the instructions, such as a PICO headset with inverse kinematics (IK) and AGILE for standing and balance.

Because human and robot limbs differ in proportion, joint limits, and reachable configurations, teleoperation needs a step called retargeting. A retargeting layer maps human motion into commands the robot can physically execute.

Tools for Collecting Data#

This workflow uses three primary tools:

Isaac Teleop for teleoperation and recording
Isaac Lab-Arena for simulation-side task execution
AGILE as the recommended default GR00T WBC controller for standing and balance

Isaac Teleop#

NVIDIA Isaac Teleop is a teleoperation platform for high-fidelity data collection.

It provides a standardized device interface for XR headsets and peripherals, a retargeting pipeline, and a data schema for MCAP recording.

In simulation, Isaac Teleop capabilities integrate with Isaac Lab-Arena and can use CloudXR to stream the simulated environment to the demonstrator’s XR headset in real time. This enables immersive teleoperation from a first-person view.

On the real robot, Isaac Teleop records live robot and sensor streams for downstream conversion.

A key benefit is consistency across simulation and real-robot collection. Isaac Teleop and Isaac Lab-Arena use shared concepts for device interfaces, retargeting, and recording schemas, which reduces the amount of rework needed when moving from simulated teleoperation to physical robot data collection.

Using AGILE With Teleoperation#

Teleoperating a humanoid robot like the G1 is difficult because the operator needs to control arms, hands, torso, legs, and balance at the same time.

AGILE enables the robot to stand on its own, allowing the operator to focus on fewer axes of motion during teleoperation.

AGILE is the baseline whole-body control component for standing and balance in the G1 workflow.

When integrated with the teleoperation stack, AGILE manages standing and balance while the operator manages upper-body manipulation through XR teleoperation.

In this course, the key architectural idea is decoupled whole-body control. AGILE provides the standing and balance baseline, while upper-body degrees of freedom remain available for inverse kinematics or vision-language-action manipulation policies.

Data Conversion#

For both simulation and real-world collection, convert the data to a correctly mapped format. If observations, robot state, or actions are mapped incorrectly, training quality drops or completely fails, even when the files appear structurally valid.

Different formats serve different priorities. At teleoperation time, the system requires fast recording, data recovery, and efficient replay.

LeRobot dataset format is a standardized, machine-learning-optimized format designed specifically for training robot learning models, while MCAP serves as a general-purpose recording format for heterogeneous robotics data.

The key difference lies in their intended use cases. MCAP excels at data capture, while LeRobot excels at training and sharing models.

LeRobot Standardization#

This workflow standardizes on the Hugging Face LeRobot dataset schema because it stores episodes as observations, robot state, and actions in a structure that aligns with GR00T training (vision-language-action).

Simulation and real-world collection use different recording formats:

Source	Recording Format	Conversion Path
Simulation with Isaac Lab-Arena	HDF5	HDF5 to LeRobot
Real robot with Isaac Teleop	MCAP	MCAP to LeRobot

For Simulation: HDF5 to LeRobot Conversion#

For simulation data, Isaac Lab-Arena records demonstrations as HDF5 files.

These files contain joint states, end-effector poses, RGB images, and action labels. A LeRobot converter transforms these HDF5 recordings into the LeRobot dataset format by mapping simulation-specific fields to the standardized schema.

For the Real-Robot: MCAP to LeRobot Conversion#

For real-world data collection, Isaac Teleop records demonstrations in MCAP (“em-cap”), an open-source container format designed for multimodal robotics logging. MCAP files serialize sensor streams, robot commands, and camera feeds with timestamps and optional compression.

The system captures real robot demonstrations as live sensor streams across multiple ROS 2 topics, including camera feeds, joint states, wrist force-torque readings, and teleoperation commands. These streams may arrive asynchronously and at different frequencies. A ROS 2 bag records that live stream to disk, and modern ROS 2 systems can store those bag files in MCAP format.

The MCAP-to-LeRobot converter extracts, synchronizes, and restructures this data into the LeRobot schema.

After conversion, LeRobot-format datasets can also be pushed to the Hugging Face Hub for sharing, reuse, and direct access from training workflows.

Post-Training a Policy#

After collecting and converting demonstration data into the LeRobot dataset format, the next step is to post-train a policy.

Post-training starts from a pretrained GR00T 1.7 foundation model and adapts it to the target task and robot embodiment through imitation learning.

VLA Inputs and Outputs#

GR00T 1.7 is a vision-language-action model. It takes visual information, language instructions, and robot state as inputs, then outputs actions for the robot to execute.

Category	Role in Training
Vision inputs	RGB images from one or more cameras mounted on or around the robot, such as wrist cameras or head cameras
Language inputs	A natural language instruction, such as `move the apple to the plate`
Robot state inputs	The current proprioceptive state of the robot, including joint positions, joint velocities, and end-effector pose
Action outputs	A sequence of joint position targets or action chunks for the controller to execute

GR00T uses action chunking, which means the model predicts a short horizon of future actions at once instead of a single next action. This can improve temporal consistency and smoothness during execution.

The LeRobot dataset schema is organized around the same structure:

observation.images
observation.state
action

That alignment is why format unification during data conversion matters.

Set Up a New Embodiment#

Because GR00T 1.7 was pretrained on specific robot hardware, adapting it to a new embodiment requires describing the robot’s physical configuration and data layout. This ability to work across new embodiments is one of GR00T’s key features.

This involves:

Defining the action space by specifying which joints are controlled and what ranges are valid.
Configuring the observation space by specifying camera count, camera resolution, and how robot state fields map to the model’s expected inputs.
Running embodiment fine-tuning by post-training on collected demonstrations so the model adapts to the robot’s morphology, dynamics, and task context.

The GR00T training scripts use configuration files to declare the embodiment’s sensor and actuator layout.

Hyperparameters to Tune#

Hyperparameters are settings that control how training is performed. For GR00T post-training, the key hyperparameters include:

Hyperparameter	Why It Matters
Learning rate	Controls how aggressively the model updates during training. Too high can overwrite the pretrained prior. Too low can prevent adaptation.
Batch size	Controls how many demonstration steps are processed per training update. Larger batches can stabilize gradients but require more GPU memory.
Number of training steps	Controls training duration. Too many steps with limited data can cause overfitting.
Action chunk size	Controls how many future actions the model predicts at once. Larger chunks can improve smoothness but reduce reactivity.
Dataset mixing ratio	Controls sampling frequency when using more than one dataset source.

The resulting artifact is a post-trained GR00T policy checkpoint.

Preparing Trained Models for Real-Robot Deployment#

In simulation, Isaac Lab-Arena can evaluate a trained GR00T checkpoint directly. The real-robot path requires an additional export step.

A training checkpoint is built for training flexibility, not real-time robot execution. It preserves training-time framework behavior that helps during fine-tuning but adds unnecessary overhead during deployment. On the G1, the policy must run as part of a low-latency inference pipeline that handles:

Preprocessing robot observations
Running GR00T inference
Postprocessing predicted actions for the robot controller

LEAPP, or Lightweight Export Annotations for Policy Pipelines, packages that pipeline for deployment. It traces the annotated GR00T policy pipeline, exports the model stages, and produces a deployment descriptor that tells the robot software stack how data flows through preprocessing, inference, and postprocessing.

Note

This export step applies only to the real-robot path. Simulation evaluation uses the trained checkpoint directly in Isaac Lab-Arena.

The detailed export commands, validation steps, and output checks appear later in the real-robot workflow.

Inference and Evaluation#

Training produces a policy, but a trained policy does not guarantee strong task performance. The next step is to measure how well the policy performs in simulation and on the real robot. Learning and performing well are different milestones.

Determine policy success by running inference to generate actions during test trials, then using evaluation to measure task performance.

Inference is the process of running a trained policy in an environment. The policy receives live observations and produces actions. In this workflow the model is a VLA, so at each step it perceives the scene and decides what action to take next. Model weights are frozen at this point, so no additional learning happens.

Evaluation is the process of combining inference with measurement. Run the policy against many trials and track its success in objective terms.

Start by evaluating in simulation where failure is cheap and iteration is fast. Automation is straightforward because simulation exposes the full state of the world. After validating in simulation, evaluate on the real robot as the hardware validation point.

Simulation: Isaac Lab-Arena#

Simulation evaluation acts as a validation gate before deploying on the real robot. Isaac Lab-Arena provides a controlled environment with parallel rollouts, instant resets, task randomization, and safe failure.

The same task environment used for data collection also supports evaluation. Isaac Lab-Arena spawns the robot, sets up the scene, and feeds simulated observations to the policy.

A low task success rate in simulation is a strong signal that the policy is not ready for hardware testing.

Because simulation trials can be repeated quickly, you can use evaluation results to identify failure modes, refine the training setup, add demonstrations, and re-evaluate.

Evaluation is also a decision point. Use the results to determine whether the policy meets the target success rate, whether failures cluster around a specific task phase, and whether the failures are safe enough to continue testing. If the policy misses the apple, drops the object, places off target, or destabilizes the robot, return to data collection, dataset validation, or training configuration before moving forward.

Real-Robot: Running on Edge#

After validating the policy in simulation, move to hardware evaluation on the real robot.

On the real robot, inference runs on edge compute. In this course, that means deploying the LEAPP-exported policy pipeline on Jetson Thor and connecting it to live robot sensor and actuator streams through the robot software stack.

At each timestep, the robot’s cameras and joint encoders feed observations into the pipeline. The policy predicts the next action chunk, and the system sends commands to the low-level controller for execution.

The policy inference must run at the frequency required to control the robot in real time, on hardware with limited memory and compute compared to a training machine.

Real-robot evaluation measures whether the policy completes the apple pick-and-place task while the G1 remains stable and within operational limits. It is the hardware validation point for the real-robot path.

For this course, a successful real-robot evaluation should show that the policy can:

Identify the apple and target plate
Reach and grasp the apple
Move the apple to the plate
Release the apple on the target
Maintain stable standing behavior
Complete the task without a safety abort

Real-robot evaluation is the ultimate test of the end-to-end workflow.

Success here means the post-trained policy can execute the apple pick-and-place task on the physical G1 under the tested workspace, hardware, and safety conditions.

Conclusion#

This module covered how the major pieces of the GR00T reference workflow fit together.

Isaac Lab-Arena supports simulation task setup and evaluation.
Isaac Teleop supports demonstration collection.
AGILE, one controller option provided by GR00T WBC, provides the baseline controller for standing and balance during teleoperation and policy execution.
LeRobot conversion prepares data for training.
GR00T fine-tuning adapts a VLA policy to the task and embodiment.
LEAPP prepares the real-robot deployment pipeline.

Now that you have this mental model, verify prerequisites and set up the environment so you can start the hands-on modules.

What’s Next#

Proceed to Agent Skills to explore the optional agent skills for troubleshooting, tutoring, and working through the rest of this course.