Skip to navigation

Add a robot embodiment

View as Markdown

This guide provides pointers to support a new robot embodiment on top of the Isaac OS stack, including identifying the adequate components for integration with simulators or robot hardware. The intent is to connect robot state and commands to an Isaac OS application without first deploying a model.

The steps focus on joint I/O. Sensor I/O instructions will come in a future release.

✨ Agent prompt: Add robot joint I/O

Overview

To support a new robot embodiment, its hardware or simulator needs to connect to a Holoscan application. Operators are the graph components in Holoscan applications that exchange typed messages through ports; see Isaac OS concepts. This guide will focus on configuring or implementing the following operators for the robot:

  • I/O operators: receive robot state and send commands through a robot SDK or UCX.
  • Test controller: consume state and generate commands. The intent of this guide is to limit that bringup to an operator or a small subgraph without inference.
  • Inspection operator: log or display state to check the integration.

For the typed messages exchanged in this graph, we recommend to use isaac::JointStateCpuT and isaac::JointCommandCpuT from the CPU joint schemas:

JointStateCpuT JointCommandCpuT JointStateCpuT State input operator Test controlleroperator or subgraph Command output operator Inspection operator

For state-only inspection, you can connect the state input to inspection and leave out the controller and command output.

While recommended, these schemas are opinionated defaults for ordered CPU joint arrays, provided for convenience and compatibility with existing Isaac OS operators and UCX interfaces. Your graph can use its own types and preserve vendor conventions. Reused components still require their declared port types.

To add a custom graph message in your project:

  1. Define its fields in a FlatBuffers .fbs schema. The SO-101 state schema adds sequence and fault fields to joint readings.
  2. Generate the payload and schema traits with holoscan_add_flatbuffer_schema, following the SO-101 message build configuration. Set the serialized-size bound for your message, expose the generated headers, and link the message library to your operators.
  3. Include the generated schema-traits header and use its ...T payload type on connected holoscan::Input<T> and holoscan::Output<T> ports. Add conversion where a component expects a different type.

With stock Isaac Sim UCX nodes, keep the existing wire messages and convert to your custom type in the application. Changing the wire format also requires a compatible peer and UCX message traits and codecs. The UCX reference lists the wire messages and graph payloads.

Identify your robot and gather resources

Identify the following before implementing I/O:

  1. The manufacturer, exact model and variant, including any hands, grippers, and sensors you need.
  2. The deployment mode of interest, such as real hardware, vendor simulator, or Isaac Sim; see application modes for the differences.

Check for reusable operators and applications, then gather what is missing:

ModeWhat to find
Isaac SimA matching asset in the Isaac Sim 6.2.0.1 catalog, a vendor or maintained community USD, or a URDF with its meshes. An asset describes the robot’s geometry and articulation.
Vendor simulatorVendor documentation or repositories for the exact model, simulator setup, and exposed state/command APIs
Real robotVendor SDK or documented device protocol, state/command examples, connection setup, and commissioning instructions

Check each resource’s model variant, versions, license, and interfaces. If an existing ROS 2 driver provides the state, commands, and timing you need, you can bridge its messages into your graph rather than writing new SDK I/O operators. For a direct connection, use the vendor SDK or documented protocol.

If you cannot find a compatible vendor simulator, explore Isaac Sim. For missing SDK access or undocumented control behavior, ask the vendor before attempting hardware control.

Set up robot I/O

Set up the robot or simulator interface and the Holoscan I/O operators that connect to it.

Real Robot

Inspect the vendor’s connection, state-read, and command examples before running them. Establish the API and lifecycle requirements before implementing the application’s I/O.

1. Identify the SDK and transport.

Check the SDK’s supported firmware versions, runtime dependencies, and required device or network permissions. Gather the connection parameters from the vendor’s example: device path, baud rate, and motor IDs for a serial bus, or interface, address, domain, and channel names for a network protocol, as applicable.

Prefer C++ for high-rate I/O when both language bindings provide the features you need. A blocking Python SDK call that holds the GIL can prevent other Python operators from running. Check whether SDK calls release it, and measure state and command timing at your required control rate.

The SO-101 driver configuration takes a serial port and calibration file containing motor IDs. The G1 configuration provides network_interface, dds_domain_id, and the configured motor count.

2. Identify the state-read path.

Find the polling function or subscription callback and its message definition. Record:

  • Joint fields, addressing, units, direction, calibration zeros, and limits, plus any required sensor frames and timestamps, for mapping and conversion.
  • Return codes and validity or fault flags that distinguish usable data, no new sample, and communication failure.
  • A sequence, device tick, timestamp, or callback event that distinguishes new samples from cached values. Use it to detect stale state.

The SO-101 driver uses read_present_positions_and_velocities() and reports read failure through its return value. The G1 state subscriber receives LowState on rt/lowstate, extracts joints and IMU, and counts callbacks to wait for a new sample before emitting state.

Before running a reference application to inspect state, check its startup: SO-101 driver initialization homes the arm, and G1’s command operator releases the onboard control mode before receiving commands.

3. Identify the command-write path.

Find the write function or publisher and establish its command contract before sending motion:

IdentifyUse in the adapter
Motor addressing and control modeSelect the SDK slots or IDs and position, velocity, effort, or hybrid mode
Applied fieldsTranslate the supported position, velocity, effort, and gain fields; identify fields the backend ignores
Required protocol fieldsPopulate enable flags, mode values, sequence numbers, or checksums required by the vendor
Update rate and watchdogSchedule writes at the required cadence and define what happens if commands stop

The SO-101 driver sends desired joint positions through send_action(); servo gains come from driver configuration. The G1 command publisher fills LowCmd.motor_cmd with position, velocity, effort, and kp/kd, enables the motors, sets the mode fields, calculates the checksum, and writes to rt/lowcmd.

4. Define startup, ownership, and stopping.

Find the vendor’s commissioning procedure, control-mode API, and watchdog or fault documentation. Establish:

  • The startup order: open the transport, obtain usable state, initialize or enable motors, and take control in the order the vendor requires. Identify any startup movement and which joints lose their existing controller.
  • How to confirm takeover and handle a failed or ambiguous handoff.
  • The required response to normal shutdown, startup failure, stale state, rejected commands, and communication loss. The response can differ by event.

Common responses to check in that documentation are:

ResponseWhat to find
Hold positionSupported hold API or command format, required feedback and gains, and whether commands must keep arriving
Disable motorsTorque-off or motor-disable API, confirmation status, and any mechanical support required before disabling
Return controlController handoff API, the receiving controller’s required state, and how to confirm it has taken control

Use the vendor’s prescribed response for each event. If the documentation does not establish these behaviors, resolve them with the vendor before hardware tests.

A stop, hold, or handoff request may not reach the robot after communication loss. Check the device’s documented watchdog response to know what happens when commands stop arriving.

5. Bridge the application to the robot interface.

Reuse matching operators if available. Otherwise, implement Holoscan operators around the SDK or protocol calls identified above. Split the bridge’s responsibilities between the operators and any shared driver/session:

  • In the operators, expose state output and command input ports. Emit readings from the state API and send received commands through the command API.
  • Implement startup and shutdown in a shared driver/session or the operators’ start() and stop() methods.
  • In the driver or I/O operators, reject unusable state and invalid commands, and handle each fault with the response identified in step 4.
  • Keep stopping with the connection lifecycle. Check hold, disable, or handoff results before stopping callbacks and closing the connection. Report failures rather than treating disconnection as a successful stop.

Separate a driver/session when multiple operators share a bus, connection, or control lifecycle, or when you need to test device communication without a graph. The SO-101 driver and operators share one serial connection. The G1 low-level operators call the SDK directly from independent publishers and subscribers. Separate state and command operators are optional; a coupled interface can expose both ports.


Connect the application graph

Connect the I/O from your selected mode to a test controller.

Choose a test controller

Look for vendor commissioning or basic-motion examples. Reuse their callable command-generation logic or adapt it into an operator, then send commands through your output operator. Your controller must choose the command values sent through the SDK’s command API. Reuse the controller across modes when state and command semantics and the control mode match. Once I/O is validated, you can replace the test controller with model-based control compatible with your robot.

For an arm that supports position control, a measured-pose hold or small joint ramp or sine motion can be a first test. Choose starting joint positions from validated state and set bounds, speeds, and gains for that robot.

The SO-101 hardware graph uses SinusMotionOp to generate desired joint positions and SimpleMotionPlannerOp to limit each step relative to measured state. Both consume SO-101’s state type; the sine generator uses one amplitude for all joints.

The SO-101 sim generator consumes JointStateCpuT and directly emits JointCommandCpuT, with sine phase driven by simulation timestamps.

For a humanoid, choose the controller for the check you need:

CheckController requirement
Inspect stateUse a documented non-actuating read path; no command generator is needed
Joint I/O on mechanical supportAdapt the vendor’s supported hold and selected-joint test routine, using its prescribed commands, gains, control modes, and cadence
Balance or locomotionUse a balancing controller compatible with the exact embodiment and its sensors

If takeover affects the whole body, provide valid commands for every affected joint, including those not under test. Zero-filled or omitted fields are not a general way to leave those joints under onboard control. Isaac OS does not provide a generic humanoid commissioning application.

For a humanoid, use the vendor’s commissioning procedure, including any required mechanical support. A gantry alone does not establish safe control. Position hold is not balance. Stay with non-actuating checks if ownership and stopping behavior are unknown.

The G1 WBC application uses a G1-specific policy and configuration. Changing joint order alone does not adapt that balancing policy to a different robot.

Its real graph places StartUpOp between WBC and command output. The operator provides startup ramps, holds, and enable gating, but runs on incoming controller commands. Removing WBC does not turn this graph into a standalone commissioning controller.

Map and convert state and commands

You now have robot I/O and a controller, but their message types, joint order, or units may differ. Without mapping, a command can target the wrong joint or angle. Connect them in both directions:

  • State: convert I/O output into the controller’s expected input.
  • Commands: convert controller output into the format accepted by I/O.

Validate received state and commands, then implement the required conversions:

ConvertImplementation
Joint identityMap graph joints to simulator joints or SDK motor slots in both directions; handle reserved or separately controlled joints explicitly
ValuesConvert units and direction. Apply calibrated zero offsets to positions, not velocities. Check converted commands against device limits
Command fieldsPopulate the fields and enable/mode values required by the selected control mode
TimestampsPreserve capture timestamps and their clock domain; explicitly use receive time when capture time is unavailable
Sensor conventions, if usedConvert coordinate frames and quaternion conventions to match the consumer
  • Prefer keeping SDK-specific conversion in the driver or I/O operators so the controller can be reused across modes. Doing that conversion inside a custom controller is also possible, but makes it depend on the robot SDK’s data format.
  • A separate adapter operator is an option when conversion is needed and you want to leave existing operators unchanged. For example, it can receive and emit JointStateCpuT while reordering joints, without introducing a new message type.
  • With stock UCX I/O, conversion can also run in the Isaac Sim scene.
  • If types and conventions match, no adapter is needed.

Suppose a backend reports [elbow, shoulder] in degrees while the graph expects [shoulder, elbow] in radians, with matching zeros and directions:

DirectionInputConverted output
State to graph[90, 30] degrees[π/6, π/2] radians
Position command to backend[π/4, π/3] radians[60, 45] degrees

Implementation examples:

  • SO-101 reach state adapter: validates joint arrays, populates the graph’s state type, and preserves capture metadata.
  • SO-101 hardware driver: keeps calibration and unit conversion in the driver, with graph I/O in the operators.
  • G1 command publisher: maps graph command fields into SDK motor slots, including enable fields and checksums.

Assemble an application

Compose the state input, controller, and command output in your application.

Use the SO-101 application composition as a wiring example, replacing its I/O and controller assumptions with your mapping. For using Isaac OS components in your own repository, see Integrate in your project.

Following the SO-101 application:

  • Create applications/<your_app>/ with a graph entrypoint, configuration, and CMakeLists.txt.
  • Register the application and its dependencies with add_isaac_application in applications/CMakeLists.txt. Follow the selected mode’s CMake file for its executable and linked components. Register new shared libraries with add_isaac_common in common/CMakeLists.txt and new operators with add_isaac_operator in operators/CMakeLists.txt.
  • Add metadata.json with the application name and run command for each implemented mode, including a mock if useful for checking wiring without hardware. Check discovery with ./isaac_os modes <your_app>, then follow Build and run.

Add sensor I/O

Coming soon: sensor setup, conversion, and validation instructions will be added in a future release.

Available now: the UCX operators support IMU and image messages.

Pair joint state and sensor data

Before connecting joint state and other sensor streams, check the receiving operator’s input timing contract (contract() in a C++ operator):

  • If it requires identical capture timestamps, provide related samples from the same simulation step or acquisition event and preserve their common timestamp.
  • If it permits different capture times, pair samples within its allowed time window and respect its stale-input limits.

Convert timestamps to the same clock domain before comparing them.

The G1 hardware subscriber extracts joint state and IMU from one SDK message and emits both with the same graph timestamp. That timestamp is not a measured device capture time. The G1 simulation adapter receives separate messages. Both it and the G1 WBC controller use OnAligned with a zero window, requiring identical capture timestamps.

Check the integration

  1. Check conversion without motion. Before connecting command output to hardware, test both directions with distinct per-joint values in a different order. Verify units, sign/zero conversion, limits, and rejection of malformed input.
  2. Inspect received state before motion. After connecting the backend and application, use a documented non-actuating path where available. Check joint identity, plausible values, and fresh samples. Repeated SDK reads may return the same sample.
  3. Test bounded commands. Follow the robot’s commissioning procedure. For an arm, a small movement can check joint identity, direction, scale, and tracking. Select amplitudes, centers, speeds, and gains for that robot; do not copy reference defaults.
  4. Check stopping and failures. Verify the expected response to stale state, invalid commands, failed communication, and shutdown, beginning with mocks or simulation and using vendor procedures for physical tests.

When the integration works, reduce latency and jitter with Optimize your host and Optimize an Application.

If state never arrives, check the UCX publisher’s inputs:timeoutMs. The SO-101 scene reports that 0 causes an immediate send timeout and uses 4294967295 to wait indefinitely.

If state arrives once and stops, an indefinite command-reception wait may block the next simulation tick. Set a finite receive timeout or send commands each tick. Also check other waits in the execution chain, including clock and sensor publishers. The SO-101 reach scene layer bounds the command wait and disconnects clock and camera publication.

Compare timestamps from the same clock domain. For wall-time age checks, use compatible receive timestamps or convert the source timestamps. A simulator-driven clock can stop when messages stop. Ensure a freshness monitor still runs without new input: a periodic trigger alone does not override a missing-input policy that skips execution. Queue depth does not enforce freshness or replace a device watchdog.

Stop motion and recheck the graph-to-device mapping. Compare joint names or motor slots, array order, positive axes, calibration zeros, and units. An unnamed state array cannot validate its own joint identity. Repeat conversion tests before resuming a bounded movement under the supported commissioning procedure.