Unitree G1 GR00T N1.7 and WBC
This application combines Agile WBC locomotion with GR00T N1.7 arm and hand control on Holoscan SDK 5.
For a guided deployment of compatible ONNX models exported by the reference workflow, start with GR00T N and whole-body control: Simulation or Real robot. This reference describes the application, its cached models and configuration.
Isaac Sim uses a different model and timing profile from mock and real modes. See configuration and application modes. For locomotion only, use G1 WBC.
Architecture
Real, Isaac Sim, and mock modes share the WBC and GR00T N control pipeline:
AgileWbcOpgenerates locomotion commands from joint state and IMU.Gr00tN17FusedOppredicts arm and hand actions from camera images and joint state.- On each WBC command,
Gr00tActionHandlerOpselects the GR00T N action for the camera frame’s age and forwards the WBC command unchanged. JointCommandCombinerOpcombines 15 WBC leg/waist joints with 28 GR00T N arm/hand joints.- In real and sim,
StartUpOpfollows the combiner and receives measured joint state and optional keyboard/gamepad enable signals.
Control graph
Sensor adapters, transport internals, validation taps, and optional display are omitted. The handler’s two command outputs share one connector.
GR00T N runs five TensorRT engines inside Gr00tN17FusedOp:
GPU execution
The fused operator reuses
fixed-address buffers and submits work to the CUDA stream supplied by
ExecutionContext. Inputs are float32 joint positions and contiguous uint8 images.
Camera frame dimensions must stay constant for the run.
After one eager warmup, execution uses these capture boundaries:
- Outside capture: input copies and fresh noise generation.
- Captured: CUDA image preparation (crop, drop extra channels, cast to float), MatX joint gather and action scatter, all five TensorRT engines, and output copies into pinned host buffers.
- After replay: the CPU waits for completion, constructs the outgoing action message, and emits it.
❓ Why combine preprocessing, inference, and postprocessing?
The image kernel combines cropping, channel selection, and byte-to-float conversion in one pass, avoiding intermediate images. MatX gathers joint positions into model order and scatters predicted actions into joint-major outputs.
Capturing the chain reduces repeated CPU submission that can contend with the 50 Hz WBC loop. Stream ordering connects the GPU stages without intermediate CPU waits. Replay preserves the individual kernels; GPU partitioning separately controls SM allocation.
Stable buffers accept fresh values without rebuilding the graph. Captured shapes and image-kernel launch parameters require fixed camera dimensions. Pinned, fixed-address output buffers allow asynchronous device-to-host transfers to be captured with the computation.
Combining stages also shares their scheduling and queue boundaries, so backpressure and fault handling apply across the combined work.
See Optimize an Application for general guidance and the benchmark for measured results.
Mode differences
The WBC, action handler, combiner, and command output share a control partition, together with the mode’s state source/adapter and startup operator where present. GR00T N and image input run separately. Sim transport and optional visualization also run separately.
The UCX interface subgraph encapsulates simulator transport, sensor pairing, and clock advancement. The green-context helper configures WBC GPU isolation. Mock sources and command validation live in mock/mock_operators.cc.
Run
Run commands from the repository root. The Isaac OS CLI builds and runs in the
project’s HSDK5 container. CUDA Green Context is enabled by default and requires GPU
and driver support for SM partitioning. Use --green-context-wbc-sms=0 to disable
partitioning. A CUDA-capable GPU is still required. The first build downloads the WBC policy and
GR00T N model resources. TensorRT engines are cached on first use.
In all three modes, Ctrl+C requests shutdown and waits for operator cleanup and CUDA work. Press it again to force termination if cleanup hangs. A signal during configuration cancels the run before the graph starts.
Choose one mode and launch one controller. In real and sim, add --gamepad to enable
keyboard/gamepad input; a keyboard connected to the app host is sufficient. Complete
device setup before launching with that flag, then follow the shared
controls. Without --gamepad, WBC uses the configured static locomotion
velocity (zero by default).
GR00T N controls the arms and hands with or without --gamepad.
Mock
Isaac Sim
Real robot
Mock
Run until Ctrl+C:
A finite run validates that combined commands contain the expected WBC and GR00T N joint values. Allow enough messages for GR00T N to produce usable actions:
Model export contract
The runtime consumes five ONNX models and their external weight files, configured through Isaac OS YAML. LEAPP is the upstream export tool that produces these artifacts; the runtime does not directly ingest a LEAPP bundle or its metadata YAML.
--model-config-path takes the Isaac OS model YAML with the five ONNX paths and
their tensor bindings. Joint mappings and action timing come from the application
YAML. Retain LEAPP metadata for compatibility checks; it is not a runtime input.
A training checkpoint must be exported before use. Preserve every ONNX
external-data file beside its model.
The deployment tutorials show how to bind
compatible exports without overwriting the build-managed cached models.
Both profiles use a 640 × 480 RGB ego-view image, 43 robot joints and five ONNX stages: video preprocessing, state preprocessing, backbone, action head and action decoding. WBC supplies 15 leg/waist joints; GR00T N supplies 28 arm/hand joints. The combiner does not use predicted navigation or base-height commands to drive locomotion; WBC uses the configured static velocity or teleop input.
Compatibility includes tensor names, shapes and dtypes, image preprocessing,
normalization, state and action joint order, position/effort semantics, and action
timing. State and action hand orders differ: compare state_output_spec and
input_spec separately. Export metadata documents these semantics but is not
imported automatically. Matching filenames, joint counts or horizons is insufficient.
Differences may require model binding or application configuration changes.
Custom operators are needed when the existing operators cannot express the
export’s preprocessing, inference or action semantics. These adaptations are
outside the compatible-export tutorials.
The cached simulation export is a concrete compatibility baseline. A newly trained reference-workflow checkpoint is not automatically interchangeable with it, and Isaac Lab-Arena evaluation does not establish behavior in the Isaac OS apple scene. Model loading and mock execution do not validate joint semantics or task success.
Configuration
Real and mock
Both launchers use config/control.yaml for the application and
the G1 GR00T N model configuration
via --model-config-path.
GR00T N, WBC, action handling, the combiner, and green-context settings are shared. Real mode also reads the camera, DDS, startup, teleop, and thread-pinning sections.
The startup routine is selected by start_up.config_file; waypoint and dwell
durations use start_up.control_dt. Teleop accepts symbolic evdev names such as
ABS_X and BTN_TL, as well as numeric codes.
To use a copy of the application YAML, pass its path after the launcher’s --:
Paths inside the YAML are resolved from the repository root. --override arguments are not
supported by these binaries; edit the YAML instead.
For real and mock modes, control is configured at 50 Hz and images at 30 Hz. An image-driven gate selects the first frame in each fixed 0.2-second capture-time interval (nominally 5 Hz), including the initial interval. Missed intervals are skipped. Actual throughput depends on processing time.
Joint state is selected at or before image capture; the last selected state is reused when no eligible sample is available. Inference starts once an eligible state is available. The input queues retain one image and four states with drop-oldest overflow. Image and state timestamps must use a common clock.
Real and mock modes use the 16-step G1 apple-to-plate model and combine 15 leg/waist joints from WBC with 28 arm/hand joints from GR00T N.
Action selection runs on each WBC command. The configured spacing between predicted actions is approximately 1/30 second, and the combiner’s GR00T N startup delay is 1 second. These are separate from the 0.2-second inference period and the real robot’s startup gate.
Isaac Sim
The sim sections of control.yaml configure GR00T N, action
handling, the startup blend, the combiner and HoloViz. The default model configuration is
the apple simulation model,
selected with --model-config-path. Model resources are under
data/gr00t_n1_7/g1_static_apple_sim/. Do not substitute the mock/hardware model
when evaluating the apple scene.
The simulation model outputs commands for 43 motors.
Sim uses the same Green Context defaults and overrides as real and mock, described in GPU and CPU placement.
For a different endpoint, start the simulator with --port 50019 and run:
Use the remote simulator hostname when running across hosts. Edit a copy of the application YAML
for timing or command changes; profiles.sim.* override blocks are not this app’s interface.
To change simulated stiffness or damping, edit the drive gains in
g1_static_apple_ucx.usda for the
upper body, or its base scene for the lower body.
Keyboard and gamepad
These controls apply to both G1 WBC and GR00T N + WBC, in real and sim modes.
Isaac Sim keyboard/gamepad teleop runs on x86_64.
Pass --gamepad to enable both keyboard and gamepad readers; devices, speeds, and mappings
come from the top-level gamepad section of the app’s YAML. Without the flag, WBC uses
configured static velocities. Mock mode does not accept teleop input.
Velocity controls, gamepad arming, and the deadman trigger behave the same in real and sim.
In GR00T N + WBC, O / R1 toggles upper-body hold in both real and sim. Real holds its
final startup waypoint; sim holds the measured pose captured at the initial handoff.
Startup enables do not affect WBC-only sim. Teleop changes locomotion velocity
while GR00T N controls the arms and hands.
Device setup
Input comes from Linux evdev devices on the machine running the app. Typing in an SSH
terminal does not send teleop input. Connect the keyboard or controller to the app host.
Select gamepad.gamepad_device and gamepad.keyboard_device using an evdev path,
a case-insensitive device-name substring, or an empty string for automatic discovery.
List device names and paths with:
The selected /dev/input/event* nodes must be readable inside the app container.
Built-in mappings support DualSense controllers with the playstation or
hid-generic driver. Connect the controller before launch when using a built-in
mapping. Other controllers need a custom mapping in the gamepad section:
Choose the evdev codes for the actual controller. A custom mapping must specify all
three axes, the deadman axis, and the lower-body button. The upper-body button is
optional; keyboard O remains available in real and sim when it is omitted.
Controls
In real mode and GR00T N + WBC sim with --gamepad, press P then O on the keyboard,
or L1 then R1 on a PS5 controller, for startup enables. WBC-only sim does not need
startup enables. The mappings below describe each button’s effect.
P enables lower-body startup once per run; pressing it again does not disable it.
After releasing the upper-body hold, pressing O / R1 again gradually returns the upper
body to its startup pose while WBC continues balancing. It does not freeze the arms at
their current position. Press again to resume controller commands.
For the apple-to-plate task, leave locomotion at zero and arrange the apple and plate for the selected model. Sim supplies the apple scene; real mode requires the physical task setup. GR00T N controls grasping, lifting, and placement. The policy continues running after placement and may move the apple again.
The built-in PS5 DualSense mappings use the following physical controls with either
supported driver. A custom gamepad_mapping overrides these defaults.
Releasing movement keys returns keyboard velocity to zero. Disarming or disconnecting the gamepad returns velocity control to the keyboard. It does not reset the startup enable latch. A custom mapping uses the same arming and deadman behavior as the built-in controls above.
The default per-axis linear speed is 0.5 m/s, and turn speed is 0.5 rad/s. Keyboard
Space affects keyboard velocity input; it does not stop GR00T N arm commands or
an armed gamepad’s velocity input.
⚠️ No teleop response
- Confirm the app was launched with
--gamepadand the input device is connected to the app host. - Check device discovery messages in the app log and that the selected event nodes are readable inside the container. If discovery selected the wrong device, set its name or path explicitly.
- If the keyboard appears ignored, disarm the gamepad with
L1. An armed gamepad owns velocity input even withL2released. - For stick input, arm with
L1and holdL2beyond its threshold. In real mode, the startup routine and lower-body enable must also complete before locomotion commands take effect.
GPU and CPU placement
By default, CUDA Green Context reserves the device’s minimum SM allocation (the device quantum) for WBC; other GPU work shares the remaining SMs.
Override the WBC SM count with the command-line option --green-context-wbc-sms=<N>
or the green_context.wbc_sms parameter in the application YAML. The command-line
option takes precedence over YAML.
Setting the WBC SM count to zero disables Green Context. Use
--green-context-wbc-sms=0 or set it in YAML:
A positive count must satisfy the device’s minimum size and alignment requirements and leave at least one quantum for other GPU work. The app exits with an error if the GPU or driver cannot support the requested allocation.
Real mode’s operator_pinning.enable defaults to false. When enabled, wbc_core
and gr00t_core select CPU cores, and wbc_priority and gr00t_priority set
SCHED_FIFO priorities. The selected cores and scheduling privileges must be
available in the container.
Mock accepts --benchmark-controller-core, --benchmark-background-core,
--benchmark-gr00t-core, and --benchmark-wbc-core; supply all four together.
--benchmark-affinity-only applies those affinities without real-time scheduling.
Simulation timing and action playback
The UCX interface subgraph owns transport and sensor pairing; sim/main.cc wires it to the control and inference operators. The handler forwards every WBC command, including before GR00T N produces its first action.
- Physics runs at 200 Hz;
--substeps 4gives 50 Hz control. - Dedicated UCX time messages advance the graph’s
SyntheticClock. State and image messages retain their upstream simulation capture timestamps. - WBC uses exact timestamp alignment between joint-state and IMU messages, with a zero-nanosecond alignment window. GR00T N also pairs state and image by exact capture timestamp, selecting the first available pair in each new 0.7-second simulation-time interval after the initial interval. Its state input retains four samples while waiting for camera delivery. Missed intervals are skipped.
- GR00T N predicts 40 actions. On each WBC command,
Gr00tActionHandlerOpselects the action corresponding to the camera frame’s age, using the simulation model’s 0.02-second action interval. It interpolates between actions and blends chunk handovers over three action steps. The selected overlay is emitted before its matching WBC command reaches the combiner. This accounts for inference latency rather than starting each chunk at its arrival time. - The combiner takes lower-body/waist commands from WBC and configured upper-body/hand commands from GR00T N. Before a valid overlay arrives, it forwards the WBC command marked as degraded. Startup preserves lower-body commands and holds the upper body until a non-degraded command and any required enables arrive. The sim profile sets the additional overlay delay to zero.
- Connection is deferred until all operators finish startup, allowing model initialization before the simulator begins stepping. This is not a fixed sleep or a guarantee that the first GR00T N action is already available.
- Simulation-time scheduling does not make inference instantaneous: late results and missing inputs can still affect behavior.
Troubleshooting
For connection failures, readiness checks and repeated attempts, see the simulation tutorial. For physical model and device checks, see the real-robot tutorial.
- A non-monotonic clock error can follow a simulator reset while the controller remains alive. Restart the controller for a new episode.
RUNTIME_OPERATOR_CALLBACK_FAILEDidentifies a callback failure; inspect both controller and simulator logs for the failing operator or simulation step.- TensorRT may warn about INT64 bindings between GR00T N models on first build.
The fused operator allocates those buffers; no action is needed when engine
creation succeeds and the app reports
accelerated inference is ready.
Benchmark capture
Real and mock modes emit native Holoscan operator-centric logging markers for Nsight Systems. Capture the mock pipeline on AGX Thor from the repository root:
Use Profile an Application to export SQLite with
--include-blobs=true and run data_flow_tracking. It reports source compute start
to sink compute end; select an exact port path and warmup window for comparisons.
The app no longer accepts --metrics-path. Real and mock modes do not expose --holoviz.
See Benchmark for results, longer captures, and the exact WBC joint-state-to-command and GR00T N image-to-command paths.
❓ How are signals handled during profiling?
Real and mock handle SIGTERM from timed Nsight captures through the same shutdown path as Ctrl+C. Profiler threads may already exist at startup; no special signal handling flag is needed.