RobotBuilder

Isaac Sim includes a special UE4 map to demonstrate the Robot Builder plugin. The RobotBuilder map can spawn all the robot models distributed with Isaac Sim. It is provided in the following location:

IsaacSimProject/Content/Maps/RobotBuilder/RobotBuilder.umap

The map directly loads URDF robot models distributed with Isaac Sim into the Unreal Engine during runtime. No import step is required (or desired) to properly synchronize with external tools. The URDF files are located in:

IsaacSimProject/Content/URDF

Use the following command to start the map:

./Engine/Binaries/Linux/UE4Editor IsaacSimProject RobotBuilder -vulkan

The robots are arranged in a grid as shown below:

../../_images/RB_Robots_1.png

The map may take some time to start up, especially the first time. When the model importer sees a URDF model for the first time, it computes internal representations for all the meshes associated with the model. This operation takes a while, so the results are cached and subsequent launches are much faster.

This map provides an excellent opportunity to experiment with features, evaluate new configurations, and debug.

Robot Selection in RobotBuilder

To control which robots get spawned in the RobotBuilder map, edit the level Blueprints (BP) using the UE4 editor by clicking the Open Level Blueprint item in the Blueprints menu. You can easily modify this blueprint to show all the robots, a single robot, or any subset of the robots. In the level blueprint, you can make or break links from the Sequence node to the Create Robot From URDF nodes. If there is a link, that robot appears in the map.

../../_images/RB_RobotSelection.png

Isaac Sim allows you to spawn and configure robots in two ways. First, you can add a robot to any UE4 map using our custom blueprint nodes. Second, you can add robots to a map using JSON configuration files in conjunction with the Isaac SDK. Both options are documented below.

The folowing is a list of robot URDFs supported and tested by Isaac Sim:

Mobile bases

Manipulators

New URDF Setup

As noted previously, in the Content directory of the IsaacSimProject folder is a subfolder called URDF. This is the root path of all URDF imports. Each subfolder within the URDF folder contains the source data for each robot (or project). To setup a new URDF model, copy your URDF folder to a unique subdirectory of IsaacSimProject/Content/URDF. Ensure all project paths are relative to the new project subdirectory by editing or viewing the URDF as text.

Note

if xacro is the source of the URDF file, generate URDF before adding it to Isaac Sim.

Configuring Robots Using Blueprints

You can configure and spawn a robot using a set of Blueprint nodes, as shown below:

../../_images/RB_RobotConfig.png

The Create Robot From URDF node allows you to set some basic properties of the robot:

  • Model Type specifies the type of the robot model to be loaded. The choices are URDF (default) or Composite. Composite models can be defined using a JSON file, described in more detail in the section Creating Composite Robots.
  • File Name is the path of the model file to load, relative to the URDF root directory IsaacSimProject/Content/URDF.
  • Robot Name is the name assigned to the robot actor, modified with the prefix Robot_ in editor sessions.
  • Location and Rotation define the root pose for the actor.
  • Spawn Params define additional configuration options for the robot and its parts, discussed below.

The Make SpawnURDF node allows you to customize a robot instance.

  • URDF Scale is the scaling factor applied to convert URDF and associated mesh coordinates to UE4 coordinates. It should be 100 (m to cm) for all the robots distributed with IsaacSim.

  • Anchor Base allows you to affix the base of the robot at the spawn position. This is useful for robots that are not meant to be mobile, like manipulator arms. It is also useful for debugging, because you can keep the robot from drifting or falling while testing joints and other functionality.

  • Anchor Base Timeout Release the robot after the specified duration if the user sets a positive timer duration. (The default is negative.)

  • Collision Mode determines how the robot parts collide with other entities in the world and each other.

    • All specifies that robot parts collide with all entities in the world and also each other.
    • All Except Self specifies that robot parts collide with all other entities in the world, but not each other.
    • Static Only specifies that the robot parts collide only with static world geometry, but not other dynamic actors or each other.
    • None specifies that no collision is set on the Robot.
  • Physics Type specifies the type of physics constructs used to instantiate the robot.

    • D6 specifies that the robot links are created as standard rigid bodies and the joints are created as standard 6-DOF joints with degrees of freedom limited according to the joint type.
    • Articulated specifies that the robot link and joints are created as a single reduced

    coordinate articulation, which is a new feature in the physics engine.

Note

D6 is the older version of PhysX and is not fully supported and tested in Isaac Sim. Articulated is the better solution, especially for manipulators.

  • Use URDF Inertia Tensors determines whether inertia tensors, mass, and center of mass are loaded from the URDF or automatically computed by the physics engine.

  • Mass Scale is a multiplier used for the masses of the robot links.

  • Default Linear Damping and Default Angular Damping allow you to specify default damping coefficients for all the robot links. These can be customized by individual link configurations, as described below.

  • Solver Position Iterations and Solver Velocity Iterations specify the solver iteration counts used for the robot links. Higher position iteration counts generally result in less rubbery behavior at the cost of increased computational complexity. Higher velocity iteration counts can improve collision response and help prevent “explosions”, but may negatively impact joint performance in kinematic compounds.

  • Double Sided Geometry should be set if the triangle vertex winding order is variable or unknown. It helps to ensure that graphics meshes are rendered properly. The default is false.

  • Merge Fixed Joints combines multiple fixed joints in the hierarchy into a single joint.

  • Ignore Visual Substitutions disables any visual substitutions applied to the robot. This is described in the Mesh Substitutions section.

  • Enable Joint Monkey is a debugging feature. Checking this box spawns a controller that makes all the joints of the robot animate through their full ranges, one joint at a time. This is best used with the robot base anchored somewhere above ground, such that no collisions with the surroundings occur.

  • Use Importer Convex Decomposition Use the convex collision generated by the importer library. Defaulted as False.

  • Use Advanced Collision This converts a convex hull into multiple hulls (if possible). A great solution when multiple convex collision shapes are merged into one source collision asset and the number of vertices is low.

  • Use RBC When enabled, use Robot Builder Configuration JSON files. The system looks for a file that shares the same name as the URDF with the extension of “rbc”, For example, carter.urdf and carter.rbc. The RBC file overrides and appends values to the URDF file. You can tune the RBC file and the results are shared with all robots spawned from the associated URDF file.

  • Default Drive Config lets you specify the default joint drive configuration for all the joints. Robot joints have configurable drives that control their behavior. There are two types of drivable joints: prismatic and revolute. Prismatic joints have one linear degree of freedom and revolute joints have one angular degree of freedom. Also, some joints have limits and others do not. For example, an elbow joint of a manipulator arm might be allowed to rotate only in some limited range, but a wheel of a mobile base is allowed to rotate without limits. The type of joint and the joint limits are obtained from the URDF. Note that the URDF importer eliminates any fixed joints present in the robot model definition. It welds together groups of rigid bodies connected by fixed joints. This improves performance and stability of the physics simulation.

    ../../_images/RB_DefaultDriveConfig.png
    • Mode defines the behavior of the joint drive.
      • Idle means that the drive is not active. The bodies connected by the joint are free to move, subject only to the joint’s degree of freedom and joint limits, if any. The drive target value and the other remaining parameters are ignored in this mode.
      • Position Drive means that the physics engine attempts to enforce a fixed joint position specified by the drive target value. For revolute joints, the target position is an angle in radians and should be within the joint’s limit range, if any. For prismatic joints, the target position is in centimeters.
      • Velocity Drive means that the physics engine attempts to enforce a fixed joint velocity specified by the drive target value. For revolute joints, the target velocity is in radians/second. For prismatic joints, the target position is in centimeters/second.
    • Drive Target defines either the position or the velocity target value, depending on the drive mode.
    • Position Drive Stiffness and Position Drive Damping apply only when the drive is in position control mode. Stiffness determines how rigid the joint is and damping can be used to suppress oscillation. These values are used as multipliers for the internal default stiffness and damping parameters.
    • Motor Strength Maximum force that the motor will apply, units are N.
    • Velocity Drive Strength applies only when the drive is in velocity control mode. This value is used as a multiplier for the internal default strength parameter.
  • Custom Drive Configs let you override the default drive configuration for individual joints. It is a dictionary indexed by the joint name.

    ../../_images/RB_CustomDriveConfigs.png
  • Custom Link Configs let you specify custom parameters for the robot links. It is a dictionary indexed by the joint name.

    ../../_images/RB_CustomLinkConfigs.png
    • Linear Damping and Angular Damping determine the strength of drag forces used to slow down the link’s motion.
    • Is Wheel is a flag that tags this link as a wheel. IsaacSim uses special wheel collider shapes to accurately model wheel collisions. If this flag is set, a wheel collider shape will be used of this link.

Configuring Robots Using JSON

In addition to blueprints, robot configurations can also be defined in JSON. The primary use case for JSON configs is when setting up robots from Isaac SDK. A complete JSON representation might look like this:

{
  "name":"carter_1",
  "reference":"URDF'segway_description/urdf/segway.urdf'",
  "pose":[1, 0, 0, 0, -11.01, 63.47, 0.92],
  "spawn_config":{
    "urdf_scale":100,
    "anchor_base":false,
    "collision_mode":"all_except_self",
    "physics_type":"articulated",
    "use_urdf_inertia_tensors":true,
    "mass_scale":1,
    "default_linear_damping":0.01,
    "default_angular_damping":0.0,
    "joint_projection":true,
    "solver_position_iterations":64,
    "solver_velocity_iterations":8,
    "double_sided_geometry":true,
    "enable_joint_monkey":false,
    "character_overlap":true,
    "default_drive_config":{
      "mode":"idle",
      "drive_target":0,
      "position_drive_stiffness":0,
      "position_drive_damping":0,
      "velocity_drive_strength":0
    },
    "custom_drive_configs":{
      "left_wheel":{
        "mode":"velocity_drive",
        "drive_target":0,
        "postion_drive_stiffness":0.0,
        "position_drive_damping":0.0,
        "motor_strength":10000,
        "velocity_drive_strength":10.0
      },
      "right_wheel":{
        "mode":"velocity_drive",
        "drive_target":0,
        "postion_drive_stiffness":0.0,
        "position_drive_damping":0.0,
        "motor_strength":10000,
        "velocity_drive_strength":10.0
      }
    },
    "custom_link_configs":{
      "left_wheel_link":{
        "linear_damping":0.01,
        "angular_damping":0.0,
        "is_wheel":true
      },
      "right_wheel_link":{
        "linear_damping":0.01,
        "angular_damping":0.0,
        "is_wheel":true
      },
      "rear_caster_wheel_link":{
        "linear_damping":0.01,
        "angular_damping":0.0,
        "is_wheel":true
      },
      "rear_caster_swivel_link":{
        "linear_damping":0.01,
        "angular_damping":10,
        "is_wheel":false
      }
    }
  }
}

This is quite lengthy, but it is included here to show the JSON names of all the parameters. In practice, you only need to specify the parameters that need non-default values. The above definition could be reduced to this:

{
  "name":"carter_1",
  "reference":"URDF'segway_description/urdf/segway.urdf'",
  "pose":[1, 0, 0, 0, -11.01, 63.47, 0.92],
  "spawn_config":{
    "character_overlap":true,
    "default_drive_config":{
      "mode":"idle"
    },
    "custom_drive_configs":{
      "left_wheel":{
        "mode":"velocity_drive",
        "drive_target":0,
        "motor_strength":10000,
        "velocity_drive_strength":10.0
      },
      "right_wheel":{
        "mode":"velocity_drive",
        "drive_target":0,
        "motor_strength":10000,
        "velocity_drive_strength":10.0
      }
    },
    "custom_link_configs":{
      "left_wheel_link":{
        "is_wheel":true
      },
      "right_wheel_link":{
        "is_wheel":true
      },
      "rear_caster_wheel_link":{
        "is_wheel":true
      },
      "rear_caster_swivel_link":{
        "is_wheel":false
      }
    }
  }
}

Spawn Configuration JSON Properties (``”spawn_config”``).

Parameter JSON name Default JSON values
URDF Scale urdf_scale 100.0
Anchor base anchor_base false
Advanced collision advanced_collision false
Anchor base timeout position_drive_damping -1.0
Collision mode collision_mode all_except_self
Physics type physics_type articulated
Use URDF inertia tensors use_urdf_inertia_tensors true
Mass scale mass_scale 1.0f
Default linear damping default_linear_damping 0.0
Default angular damping default_angular_damping 0.0
Joint projection joint_projection true
Solver position iterations solver_position_iterations 64
Solver velocity iterations solver_velocity_iterations 1
Double sided geometry double_sided_geometry false
Merge fixed joints merge_fixed_joints true
Ignore visual substitutions ignore_visual_substitutions false
Enable joint monkey enable_joint_monkey true
Adjust pawn location adjust_spawn_location true
Use importer convex collision use_importer_convex_collision false
Use Advanced collision use_advanced_collision false
Character Overlap character_overlap false

Custom Drive Config JSON Properties (custom_drive_configs)

Parameter JSON name Default JSON values
Joint drive mode joint_drive_mode position_drive
Drive target drive_target 0.0
Position drive stiffness position_drive_stiffness 1000.0
Position drive damping position_drive_damping 1.0
Motor strength motor_strength 100000000.0
Velocity drive strength velocity_drive_strength 0.0

Custom Link Config JSON Properties (``”custom_link_configs`).

Parameter JSON name Default JSON values
Linear damping joint_drive_mode 0.01
Angular damping angular_damping 0.0
Is wheel is_wheel false
Advanced collision advanced_collision false

Enumeration JSON Values.

Parameter JSON name JSON values
Collision mode collision_mode

all

all_except_self (default)

static_only

Physics type physics_type

articulated (default)

d6

Joint drive mode mode

idle

position_drive (default)

velocity_drive

Creating Composite Robots

You can combine multiple robots into a single robot model by creating a composite robot definition using JSON. This allows you to mount one or more manipulator arms onto a mobile base or create manipulator arms with different end effector attachments. This could be done by creating a custom URDF for your composite robot, but this can be tedious in some cases. If you have two URDFs that you would like to combine by hand, you can copy one into the other, add a new joint to connect the root link of one robot to a link of the other other robot. Avoid name collisions by editing the link and joint definitions to make sure all the names are unique. This gets worse if you want to attach two submodels of the same type, because you must rename all the links and joints to avoid clashes. The JSON composite model approach is designed to gracefully handle such situations.

The following is an example of a composite robot definition that adds two Franka arms onto a Segway RMP440 mobile base:

{
  "name": "SegwayWithArms",
   "sources": [
      {
        "id": "segway",
        "path": "segway_description/urdf/segway440LE.urdf",
        "format": "urdf"
      },
      {
        "id": "franka",
        "path": "franka_description/robots/panda_arm.urdf",
        "format": "urdf"
      }
    ],
    "composition": [
      {
        "id": "base",
        "source": "segway"
      },
      {
        "id": "arm1",
        "source": "franka",
        "parent": "base",
        "parent_body": "inertial_collision_link",
        "local_pose": {
          "xyz": "0.3 0 0.265",
          "rpy": "0 0 0"
        }
      },
      {
        "id": "arm2",
        "source": "franka",
        "parent": "base",
        "parent_body": "inertial_collision_link",
        "local_pose": {
          "xyz": "-0.3 0 0.265",
          "rpy": "0 0 3.14159265359"
        }
      }
    ]
  }

The robot looks like this:

../../_images/RB_CompositeRobot.png

You can test it out using the map at Content/Maps/RobotBuilder/RobotComposition.

In the JSON script, the basic idea is to list the source URDFs to be used in sources. Then in the composition section you can refer to them by name to create new attachments. The first item in composition is implicitly the root (base) of the robot. Once the composite model is created, you can refer to the individual joints and links using names that concatenate the submodel ID with the original joint name, e.g.:

base:front_left_wheel
base:front_right_wheel
arm1:panda_joint3
arm2:panda_joint3

To load a composite robot from blueprint, you must change the Model Type to Composite in the Create Robot from URDF node and set the File Name to the path of the JSON file.

Spawning composite models is also supported by the simulation bridge. See the IsaacSim instructions for details.

Robot Configuration Files

The Robot Configuration File (RBF) file is a JSON file used override the defaults when creating a robot from a URDF. It contains all of the properties listed in spawn_config and overrides the values that are set in Blueprints when the SpawnConfig option bUseRBC is true.

The goal of the RBC file is to share a common UE4 configuration across all instances of the URDF without modifying the source and duplicating Blueprint configuration data. It also provies a mechanism to fix an error or tweak a robot configurations and have the values propagate to all robots.

Most of the URDF projects contain a Robot Configuration File. They contain valid settings for the robot to work properly within UE4. If your project fails in some way or explodes, you can reference the RBC file for clues to how to resolve your issues.

Custom Meshes & Materials

The RobotBuilder plugin imports unmodified URDF geometry and collision at runtime. The import acquires the visual representation from the referenced mesh files and generates it procedurally during load time to support the simulators workflow. The materials applied to the visuals are also defined in the URDF. Needless to say, the visual quality of the models is not as high as what you can create in the Unreal Engine.

Custom Meshes

To fill this gap in visual quality, the RobotBuilder plugin supports mesh and material substitution. This gives the designer the ability to replace the visual component without changing the behavior or structure of the robot. This is accomplished by attaching a visual component as a child of the physics component. This component is known as the link as defined in the URDF and in Isaac Sim, it is called the RobotLink.

../../_images/RB_CustomMeshes.png

The image above (of the RobotMeshes.umap test map) illustrates the difference between the default procedural mesh visuals and the substituted components. The customized meshes and materials on the right in the image above can be of the best quality in Unreal. This includes shaders, special effects, and other processing not possible in the URDF format.

In general, the quality of the meshes built for the Unreal Engine are faster and use less resources than the visuals imported from the URDF. It is also possible to take advantage of features such as RTX and Ambient Occlusion.

The mesh substitution is a UStaticMeshComponent created by an artist and imported into the engine via the Unreal Engine editor. The substitution is automated by following a specific structure of the URDF project. If a StaticMesh uasset is found in the proper location, it is loaded in place of the mesh from the URDF.

For example, take a look at the franka_description project folder which consists of a Content/Meshes and a Content/Materials folder.

The mesh names are derived from the links in the URDF. panda_link0.uasset is a static mesh and the artist created it based on the following information stored in panda_hand.urdf:

<?xml version="1.0" ?>
<robot name="panda" xmlns:xacro="http://www.ros.org/wiki/xacro">
  <link name="panda_link0">
    <visual>
      <geometry>
        <mesh filename="package://franka_description/meshes/visual/link0.dae"/>
      </geometry>
    </visual>
    <collision>
      <geometry>
        <mesh filename="package://franka_description/meshes/collision/link0.stl"/>
      </geometry>
    </collision>
  </link>

The following is the panda_link0 mesh that was imported by an artist into the engine and stored in the {project_name}/Content/Meshes directory. The substitute mesh has impressive visual qualities that are very useful in demos and presentations.

To substitute a mesh, save a static mesh as: {project_name}/Content/Meshes/{link_name.uasset} where the project_name is the subdirectory where the URDF project is stored (ie, the root of the project) and the link_name is the link you wish to substitute with a custom static mesh.

The RobotMesh.umap file contains the current mesh substitution components with a side-by-side comparison of the default render vs the custom components. The goal is to keep this file up to date when new components are available.

Custom Materials

As with custom meshes, it is possible to replace individual materials or supply a global material for the robot.

To create a global material, save a material in the Content/URDF/{project}/Material folder using the URDF name as the name of the material. For example, to create a material for carter.urdf, create a material called carter and save it as carter/Content/Materials/carter.usasset.

When the robot is spawned, all links use the new material you have defined.

To override the material per link, create a material using the name of the link in the same location as the global material. For example: {project_name}/Content/Materials/{link_name.usasset} where the project_name is the subdirectory where the URDF project is stored and the link_name is the link you wish to apply the custom material.

Test Maps

Each Robot contains a test map in the RobotBuilder/RobotTest subdirectory. The tests are used as examples of working robots as well as a means to experiment with the robot’s properties. In addition to the robot, you will see sliders in the upper left of the display to manipulate the joint drives.

../../_images/RB_RobotTest.png

The testing maps vary slightly depending on the type of robot. Mobile bases should move through the world when the forces set by the sliders are adjusted while the manipulators are typically anchored and associated joints actuate according to the forces applied. RobotBuilder also supports composite robots which are essentially manipulators mounted on mobile bases.

Tuning Parameters

To observe the default behavior of the robots, it is sufficient to run the associated map. The default is a stable robot. To tune parameters or manipulated the robot, you can open the level blueprint to edit.

Joint Monkey

You can enable Joint Monkey testing to drive both types of robots. For mobile bases, the robot should move or rotate in place. For manipulators, each joint should rotate through the upper and lower bounds of its range.

Alternatively, you can use the RobotUI to enable the Joint Monkey without modifying the level blueprint.