Add a robot embodiment
This guide provides pointers to support a new robot embodiment on top of the Isaac OS stack, including identifying the adequate components for integration with simulators or robot hardware. The intent is to connect robot state and commands to an Isaac OS application without first deploying a model.
The steps focus on joint I/O. Sensor I/O instructions will come in a future release.
✨ Agent prompt: Add robot joint I/O
Overview
To support a new robot embodiment, its hardware or simulator needs to connect to a Holoscan application. Operators are the graph components in Holoscan applications that exchange typed messages through ports; see Isaac OS concepts. This guide will focus on configuring or implementing the following operators for the robot:
- I/O operators: receive robot state and send commands through a robot SDK or UCX.
- Test controller: consume state and generate commands. The intent of this guide is to limit that bringup to an operator or a small subgraph without inference.
- Inspection operator: log or display state to check the integration.
For the typed messages exchanged in this graph, we recommend to use
isaac::JointStateCpuT and isaac::JointCommandCpuT from the
CPU joint schemas:
For state-only inspection, you can connect the state input to inspection and leave out the controller and command output.
❓ Do I need to use these joint message types?
While recommended, these schemas are opinionated defaults for ordered CPU joint arrays, provided for convenience and compatibility with existing Isaac OS operators and UCX interfaces. Your graph can use its own types and preserve vendor conventions. Reused components still require their declared port types.
To add a custom graph message in your project:
- Define its fields in a FlatBuffers
.fbsschema. The SO-101 state schema adds sequence and fault fields to joint readings. - Generate the payload and schema traits with
holoscan_add_flatbuffer_schema, following the SO-101 message build configuration. Set the serialized-size bound for your message, expose the generated headers, and link the message library to your operators. - Include the generated schema-traits header and use its
...Tpayload type on connectedholoscan::Input<T>andholoscan::Output<T>ports. Add conversion where a component expects a different type.
With stock Isaac Sim UCX nodes, keep the existing wire messages and convert to your custom type in the application. Changing the wire format also requires a compatible peer and UCX message traits and codecs. The UCX reference lists the wire messages and graph payloads.
Identify your robot and gather resources
Identify the following before implementing I/O:
- The manufacturer, exact model and variant, including any hands, grippers, and sensors you need.
- The deployment mode of interest, such as real hardware, vendor simulator, or Isaac Sim; see application modes for the differences.
Check for reusable operators and applications, then gather what is missing:
Check each resource’s model variant, versions, license, and interfaces. If an existing ROS 2 driver provides the state, commands, and timing you need, you can bridge its messages into your graph rather than writing new SDK I/O operators. For a direct connection, use the vendor SDK or documented protocol.
If you cannot find a compatible vendor simulator, explore Isaac Sim. For missing SDK access or undocumented control behavior, ask the vendor before attempting hardware control.
Set up robot I/O
Set up the robot or simulator interface and the Holoscan I/O operators that connect to it.
Real Robot
Vendor simulator
Isaac Sim
Real Robot
Inspect the vendor’s connection, state-read, and command examples before running them. Establish the API and lifecycle requirements before implementing the application’s I/O.
1. Identify the SDK and transport.
Check the SDK’s supported firmware versions, runtime dependencies, and required device or network permissions. Gather the connection parameters from the vendor’s example: device path, baud rate, and motor IDs for a serial bus, or interface, address, domain, and channel names for a network protocol, as applicable.
❓ Which SDK language should I use?
Prefer C++ for high-rate I/O when both language bindings provide the features you need. A blocking Python SDK call that holds the GIL can prevent other Python operators from running. Check whether SDK calls release it, and measure state and command timing at your required control rate.
📄 For example
The SO-101 driver configuration
takes a serial port and calibration file containing motor IDs.
The G1 configuration
provides network_interface, dds_domain_id, and the configured motor count.
2. Identify the state-read path.
Find the polling function or subscription callback and its message definition. Record:
- Joint fields, addressing, units, direction, calibration zeros, and limits, plus any required sensor frames and timestamps, for mapping and conversion.
- Return codes and validity or fault flags that distinguish usable data, no new sample, and communication failure.
- A sequence, device tick, timestamp, or callback event that distinguishes new samples from cached values. Use it to detect stale state.
📄 For example
The SO-101 driver
uses read_present_positions_and_velocities() and reports read
failure through its return value. The
G1 state subscriber
receives LowState on rt/lowstate, extracts joints and IMU, and counts callbacks
to wait for a new sample before emitting state.
Before running a reference application to inspect state, check its startup: SO-101 driver initialization homes the arm, and G1’s command operator releases the onboard control mode before receiving commands.
3. Identify the command-write path.
Find the write function or publisher and establish its command contract before sending motion:
📄 For example
The SO-101 driver
sends desired joint positions through send_action(); servo gains come from
driver configuration. The
G1 command publisher
fills LowCmd.motor_cmd with position, velocity, effort, and kp/kd, enables the
motors, sets the mode fields, calculates the checksum, and writes to rt/lowcmd.
4. Define startup, ownership, and stopping.
Find the vendor’s commissioning procedure, control-mode API, and watchdog or fault documentation. Establish:
- The startup order: open the transport, obtain usable state, initialize or enable motors, and take control in the order the vendor requires. Identify any startup movement and which joints lose their existing controller.
- How to confirm takeover and handle a failed or ambiguous handoff.
- The required response to normal shutdown, startup failure, stale state, rejected commands, and communication loss. The response can differ by event.
Common responses to check in that documentation are:
Use the vendor’s prescribed response for each event. If the documentation does not establish these behaviors, resolve them with the vendor before hardware tests.
A stop, hold, or handoff request may not reach the robot after communication loss. Check the device’s documented watchdog response to know what happens when commands stop arriving.
5. Bridge the application to the robot interface.
Reuse matching operators if available. Otherwise, implement Holoscan operators around the SDK or protocol calls identified above. Split the bridge’s responsibilities between the operators and any shared driver/session:
- In the operators, expose state output and command input ports. Emit readings from the state API and send received commands through the command API.
- Implement startup and shutdown in a shared driver/session or the operators’
start()andstop()methods. - In the driver or I/O operators, reject unusable state and invalid commands, and handle each fault with the response identified in step 4.
- Keep stopping with the connection lifecycle. Check hold, disable, or handoff results before stopping callbacks and closing the connection. Report failures rather than treating disconnection as a successful stop.
❓ When should the driver be separate from the operator?
Separate a driver/session when multiple operators share a bus, connection, or control lifecycle, or when you need to test device communication without a graph. The SO-101 driver and operators share one serial connection. The G1 low-level operators call the SDK directly from independent publishers and subscribers. Separate state and command operators are optional; a coupled interface can expose both ports.
Connect the application graph
Connect the I/O from your selected mode to a test controller.
Choose a test controller
Look for vendor commissioning or basic-motion examples. Reuse their callable command-generation logic or adapt it into an operator, then send commands through your output operator. Your controller must choose the command values sent through the SDK’s command API. Reuse the controller across modes when state and command semantics and the control mode match. Once I/O is validated, you can replace the test controller with model-based control compatible with your robot.
For an arm that supports position control, a measured-pose hold or small joint ramp or sine motion can be a first test. Choose starting joint positions from validated state and set bounds, speeds, and gains for that robot.
📄 Example: SO-101 command generation without inference
The SO-101 hardware graph
uses SinusMotionOp to generate desired joint positions and SimpleMotionPlannerOp to
limit each step relative to measured state. Both consume SO-101’s state type;
the sine generator uses one amplitude for all joints.
The SO-101 sim generator
consumes JointStateCpuT and directly emits JointCommandCpuT, with sine phase
driven by simulation timestamps.
For a humanoid, choose the controller for the check you need:
If takeover affects the whole body, provide valid commands for every affected joint, including those not under test. Zero-filled or omitted fields are not a general way to leave those joints under onboard control. Isaac OS does not provide a generic humanoid commissioning application.
For a humanoid, use the vendor’s commissioning procedure, including any required mechanical support. A gantry alone does not establish safe control. Position hold is not balance. Stay with non-actuating checks if ownership and stopping behavior are unknown.
❓ Can I reuse the G1 controller for another humanoid?
The G1 WBC application uses a G1-specific policy and configuration. Changing joint order alone does not adapt that balancing policy to a different robot.
Its real graph places
StartUpOp
between WBC and command output. The operator provides startup ramps, holds, and
enable gating, but runs on incoming controller commands. Removing WBC does not
turn this graph into a standalone commissioning controller.
Map and convert state and commands
You now have robot I/O and a controller, but their message types, joint order, or units may differ. Without mapping, a command can target the wrong joint or angle. Connect them in both directions:
- State: convert I/O output into the controller’s expected input.
- Commands: convert controller output into the format accepted by I/O.
Validate received state and commands, then implement the required conversions:
- Prefer keeping SDK-specific conversion in the driver or I/O operators so the controller can be reused across modes. Doing that conversion inside a custom controller is also possible, but makes it depend on the robot SDK’s data format.
- A separate adapter operator is an option when conversion is needed and you want
to leave existing operators unchanged. For example, it can receive and emit
JointStateCpuTwhile reordering joints, without introducing a new message type. - With stock UCX I/O, conversion can also run in the Isaac Sim scene.
- If types and conventions match, no adapter is needed.
📄 Examples: joint mapping and conversion
Suppose a backend reports [elbow, shoulder] in degrees while the
graph expects [shoulder, elbow] in radians, with matching zeros and directions:
Implementation examples:
- SO-101 reach state adapter: validates joint arrays, populates the graph’s state type, and preserves capture metadata.
- SO-101 hardware driver: keeps calibration and unit conversion in the driver, with graph I/O in the operators.
- G1 command publisher: maps graph command fields into SDK motor slots, including enable fields and checksums.
Assemble an application
Compose the state input, controller, and command output in your application.
Use the SO-101 application composition as a wiring example, replacing its I/O and controller assumptions with your mapping. For using Isaac OS components in your own repository, see Integrate in your project.
Example: extend the Isaac OS reference repository
Following the SO-101 application:
- Create
applications/<your_app>/with a graph entrypoint, configuration, andCMakeLists.txt. - Register the application and its dependencies with
add_isaac_applicationinapplications/CMakeLists.txt. Follow the selected mode’s CMake file for its executable and linked components. Register new shared libraries withadd_isaac_commonincommon/CMakeLists.txtand new operators withadd_isaac_operatorinoperators/CMakeLists.txt. - Add
metadata.jsonwith the application name and run command for each implemented mode, including a mock if useful for checking wiring without hardware. Check discovery with./isaac_os modes <your_app>, then follow Build and run.
Add sensor I/O
Coming soon: sensor setup, conversion, and validation instructions will be added in a future release.
Available now: the UCX operators support IMU and image messages.
Pair joint state and sensor data
Before connecting joint state and other sensor streams, check the receiving
operator’s input timing contract (contract() in a C++ operator):
- If it requires identical capture timestamps, provide related samples from the same simulation step or acquisition event and preserve their common timestamp.
- If it permits different capture times, pair samples within its allowed time window and respect its stale-input limits.
Convert timestamps to the same clock domain before comparing them.
📄 Example: G1 joint state and IMU pairing
The G1 hardware subscriber
extracts joint state and IMU from one SDK message and emits both with the same
graph timestamp. That timestamp is not a measured device capture time.
The G1 simulation adapter
receives separate messages. Both it and the
G1 WBC controller
use OnAligned with a zero window, requiring identical capture timestamps.
Check the integration
- Check conversion without motion. Before connecting command output to hardware, test both directions with distinct per-joint values in a different order. Verify units, sign/zero conversion, limits, and rejection of malformed input.
- Inspect received state before motion. After connecting the backend and application, use a documented non-actuating path where available. Check joint identity, plausible values, and fresh samples. Repeated SDK reads may return the same sample.
- Test bounded commands. Follow the robot’s commissioning procedure. For an arm, a small movement can check joint identity, direction, scale, and tracking. Select amplitudes, centers, speeds, and gains for that robot; do not copy reference defaults.
- Check stopping and failures. Verify the expected response to stale state, invalid commands, failed communication, and shutdown, beginning with mocks or simulation and using vendor procedures for physical tests.
When the integration works, reduce latency and jitter with Optimize your host and Optimize an Application.
⚠️ Joint state never arrives or stops updating in Isaac Sim
If state never arrives, check the UCX publisher’s inputs:timeoutMs. The SO-101
scene reports that 0 causes an immediate send timeout and uses 4294967295 to
wait indefinitely.
If state arrives once and stops, an indefinite command-reception wait may block the next simulation tick. Set a finite receive timeout or send commands each tick. Also check other waits in the execution chain, including clock and sensor publishers. The SO-101 reach scene layer bounds the command wait and disconnects clock and camera publication.
❓ Which clock should check state freshness?
Compare timestamps from the same clock domain. For wall-time age checks, use compatible receive timestamps or convert the source timestamps. A simulator-driven clock can stop when messages stop. Ensure a freshness monitor still runs without new input: a periodic trigger alone does not override a missing-input policy that skips execution. Queue depth does not enforce freshness or replace a device watchdog.
⚠️ A command moves the wrong joint or in the wrong direction
Stop motion and recheck the graph-to-device mapping. Compare joint names or motor slots, array order, positive axes, calibration zeros, and units. An unnamed state array cannot validate its own joint identity. Repeat conversion tests before resuming a bounded movement under the supported commissioning procedure.