GR00T N and whole-body control - Simulation
Load compatible static-apple ONNX models produced by LEAPP export into Isaac OS, connect to Isaac Sim, and watch the G1 lift an apple and release it onto a plate. Finish by resetting the scene for another attempt.
As of Isaac OS v0.11, only the bundled simulation model is supported. Other GR00T exports, including your own LEAPP exports, are not supported yet.
Before you start
Check your host
Complete Check your host in Getting started. For this simulation path, use a Linux x86_64 workstation that also meets the Isaac Sim requirements. Run the simulator and controller on the same workstation, with a desktop display to observe the task and network access to download scene assets.
Prepare your source checkout
Follow Fetch the reference source code if you do not already have an Isaac OS checkout. Complete the sample app run to check the shared build-and-run setup; it should exit after 100 commands.
Open two terminals at the Isaac OS repository root and run the following steps there. Isaac Sim setup covers display and cache configuration.
Use the bundled ONNX models
The build downloads the bundled static-apple models: five ONNX models and their external weight files, with a 40-step decoded action horizon. Do not export a new policy for this tutorial. Isaac OS loads ONNX using its own model YAML; it does not ingest LEAPP metadata. The model handoff contract lists the interfaces and joint mappings.
1. Build the controller
In terminal 2:
Wait for the build to finish successfully. The first build obtains the container, dependencies, WBC policy and cached GR00T N models. At launch, the bundled models you copy next replace the cached GR00T N model paths; WBC continues using the downloaded policy.
2. Copy the bundled models
In terminal 2, copy the bundled models and their model YAML:
Run this after the build, using the same cache mounts. Copying inside the container resolves the cached model symlinks.
Keep external-data files beside their ONNX models, with their original names.
Keep these files outside the build-managed data/ directories.
In deployment/gr00t-sim/model.yaml, replace the directory prefix
data/gr00t_n1_7/g1_static_apple_sim/ in all five gr00t_n1_7.models[].path
entries with deployment/gr00t-sim/policy/. The paths should now be:
Leave the bindings unchanged. All paths resolve from
the repository root, which is mounted inside the application container.
--model-config-path below selects deployment/gr00t-sim/model.yaml.
3. Start the apple scene
In terminal 1:
Accept the Isaac Sim EULA when prompted. The first launch can take several minutes while assets load and shaders compile. Keep the initial robot and object poses. Leave terminal 1 running. The four physics substeps give this scene its 50 Hz control rate. No separate policy server or Isaac Lab process is needed.
4. Run the policy
In terminal 2:
The first launch builds TensorRT engines for the GR00T N models and WBC, which can take several minutes. Wait for inference initialization and the UCX connection. The simulator then starts stepping. Startup hands the upper body to GR00T N automatically; no keyboard or gamepad enables are required in this exercise. Leave locomotion at zero.
📄 Readiness messages
The controller reports accelerated inference is ready. and
Connected to the UCX peer at 127.0.0.1:50008.. Terminal 1 reports
UCX client connected - starting simulation..
⚠️ The controller does not start or connect
For missing model files, check the five paths and the ONNX external-data files
under deployment/gr00t-sim/policy/. For binding or shape errors, compare the
export with the simulation model contract before retrying.
For a connection failure, check that terminal 1 is waiting on port 50008 and
that no other controller is connected. On a host with multiple interfaces, see
UCX connection troubleshooting.
If startup reports an unsupported CUDA Green Context allocation, append
--green-context-wbc-sms=0 to the controller command to disable partitioning.
This still requires a CUDA GPU and can change performance.
5. Observe pickup and placement
Watch the Isaac Sim viewport. A successful attempt has two observable outcomes: the G1 lifts the apple clear of the table, then releases it onto the plate. Inference readiness or a scene reset does not establish task success. The policy continues after placement and may move the apple again.
⚠️ The robot does not complete the task
Verify the simulation model configuration, the original apple scene, and
--substeps 4. Keep the robot and objects at their initial poses and leave
locomotion at zero. Inspect both terminals for camera, inference or simulator
errors. Use a fresh episode for the next attempt.
6. Stop and repeat
Press Ctrl+C in terminal 2 and wait for the controller to exit. Isaac Sim resets the scene. Repeat step 4 for another attempt; the controller can start while Isaac Sim resets. Restart the controller for every episode; do not reset the scene underneath a running controller. When finished, stop the controller first, then press Ctrl+C in terminal 1 to stop Isaac Sim.
❓ How can I compare attempts?
Record the Isaac OS commit (git rev-parse HEAD), model identity, configuration,
and pickup and placement outcomes. Keep the observation duration consistent in
simulation time. Record any later movement off the plate separately; it does not
undo an earlier successful placement.
For a separate deployment path requiring a physical-G1 export and hardware, see Real robot.