IsaacSim Unity3D

IsaacSim allows you to use Unity3D as the simulation environment for Isaac robotics. IsaacSim Unity3D provides an expandable test environment to evaluate the performance of the Isaac navigation stack. It also provides an infinite stream of procedurally generated, fully annotated training data for machine learning. Features include emulation of sensor hardware and robot base models, scene randomization, and scenario management.

See the Isaac SDK Setup page for details on installing IsaacSim Unity3D.

Unity3D overview diagram

Getting Started with Play Mode

This section explains how to use the “play mode” build of IsaacSim Unity3D with Isaac SDK.

Warehouse Navigation

The “play mode” build contains two scenes: “small_warehouse” and “medium_warehouse”. To select a scene to run, use the --scene command-line option followed by the name of the scene.

First, start the simulator with the “small_warehouse” scene:

bob@desktop:~/isaac_sim_unity3d$ cd builds
bob@desktop:~/isaac_sim_unity3d/builds$ ./isaac_sim_unity3d.x86_64 --scene small_warehouse

Next, run the Isaac SDK application with the Carter navigation stack:

bob@desktop:~/isaac$ bazel run //apps/navsim:navsim_navigate

The “navsim_navigate” application is similar to the Carter application for a real-world robot. Instead of the hardware subgraph, which launches hardware drivers, the “navsim_navigate” application uses the navsim subgraph, which communicates with the simulator via TCP sockets. The additional parameters above specify configuration files for the simulated warehouse map and robot used in the simulation.

Below is a top-down Unity camera view in Isaac Sight when the “small_warehouse” simulation is running: The goal pose is shown as a semi-transparent green Carter, and the global and local plans are displayed as red and blue lines, respectively.

Small warehouse overhead camera

Sight also displays the following windows:

  • Map View: Displays Carter localization using flatscan LIDAR.
  • Speed Profile: Displays commanded and observed speeds for the differential base.
  • Color Camera: Outputs a forward-facing, first-person view from Carter.
Small warehouse Sight output

Visualizing Data

Once an IsaacSim Unity3D scene is running, you can use the Isaac SDK “navsim_viewer_tcp” app to visualize via Websight camera data published by NavSim. To start the NavSim viewer, run the following command:

bob@desktop:~/isaac$ bazel run packages/navsim/apps:navsim_viewer_tcp

Note that the number and type of cameras differ by scene: For example, the “small_warehouse” and “medium_warehouse” scenes only have one color camera and one depth camera, while the “rng_warehouse” has instance and label cameras as well. Only enabled channels can be viewed in Websight.

Scenes in the Play Mode Build

The sample scenes contained in builds/isaac_sim_unity3d.x86_64 are detailed below:

Scene Description Isaac app
small_warehouse A small warehouse environment to test and demonstrate the Isaac navigation stack. By default, this scene uses Carter, but you can use other robots and scenarios as well. apps/carter/navsim /navsim_navigate
medium_warehouse A medium-sized warehouse enviornment to test and demonstrate the Isaac navigation stack. By default, this scene uses Carter, but you can use other robots and scenarios as well. apps/carter/navsim /navsim_navigate
pose_estimation_training A sample scene to generate training samples for a pose-estimation model for the autoencoder architecture. By default this scene uses a dolly model and randomizes camera positions, backgrounds, etc. every frame. packages/object _pose_estimation/ /apps/training. app.json
pose_estimation_codebook _generation A sample scene to generate a codebook for the autoencoder pose estimation training model. This scene receives teleportation coordinates for camera positions and captures views of the spawned object from all orientations. The packages/object _pose_estimation/ apps/codebook _generation
object_detection_training Object detection for the DetectNetv2 GEM uses this scene to generate simulated images (both positive and negative samples for object detections) for feeding into the TLT training pipeline. packages/ml/apps/ generate_kitti _dataset

Getting Started with Editor Mode

This section shows you how to use the “editor mode” build of IsaacSim Unity3D with Isaac SDK.

Opening the Sample Project

Once you have installed the Unity Editor (as described above), run the following command:

bob@desktop:~$: Unity/Hub/Editor/2018.3.11f1/Editor/Unity -projectPath isaac_sim_unity3d/projects/sample -importPackage isaac_sim_unity3d/packages/Samples/ThirdParty/substance.unitypackage -logfile

Note that the -importPackage argument is only necessary when you launch a project for the first time. This argument imports the Substance plugin, which IsaacSim Unity3D uses for material randomization.

When opening the sample project for the first time, Unity can take about 10 minutes to compile scripts and import assets. When the Unity Editor window opens, you should see the following items in the Project tab:

Unity editor Project tab

The following are important folders:

  • Assets/Allegorithmic: The imported Substance plugin folder. If you do not see this folder, the import was unsuccessful, and you will see errors in the Console tab. You can fix this by manually importing the plugin. In the Editor window toolbar, go to Assets > Import Package > Custom Package and choose isaac_sim_unity3d/packages/Samples/ThirdParty/substance.unitypackage.

  • Assets/StreamingAssets/navsim-pkg: The deployed Isaac SDK NavSim package, which contains the C API and the NavSim app to run inside Unity. If you have made changes to the Isaac SDK that affect the navsim-pkg or its dependencies (e.g. the navsim.app.json file, C API, C# binding, or engine), you should redeploy the navsim-pkg to IsaacSim Unity3D with the following command:

    bob@desktop:~/isaac$ ./engine/build/deploy.sh -p //packages/navsim/apps:navsim-pkg -d x86_64 -h localhost --deploy_path ~/isaac_sim_unity3d/projects/sample/Assets/StreamingAssets
    
  • Packages/NVIDIA IsaacSim for Unity3D (Core): The core package for IsaacSim Unity3D, containing all scripts, custom shaders, sensor prefabs, etc. It is included in the sample project with the following line in sample/Package/manifest.json:

    "com.nvidia.isaac_sim_core": "file:../../../packages/Nvidia/Core",
    
  • Packages/NVIDIA IsaacSim for Unity3D (Sample): The sample package for IsaacSim Unity3D, containing sample scenes and assets. It is included in the sample project with the following line in sample/Package/manifest.json:

    "com.nvidia.isaac_sim_sample": "file:../../../packages/Nvidia/Sample",
    

Running the medium_warehouse Scene

The first time you launch the sample project, Unity opens an empty scene. Follow these steps to open the “medium_warehouse” scene:

  1. Select the Project tab.
  2. Navigate to NVIDIA IsaacSim for Unity3D (Sample) > Warehouse > Scenes.
  3. Drag the “medium_warehouse” icon into the Hierarchy tab (as shown in the image below).
  4. Remove the default scene by clicking the icon next to the scene name and selecting Remove Scene.
Unity Editor with "medium_warehouse" scene loaded

Once the “medium_warehouse” scene loads, press Play to start the simulation. The Editor will switch to “Game” view automatically, and Carter will begin running and wait for commands from Isaac. Run the navigation app with the following command:

bob@desktop:~/isaac$ bazel run //apps/navsim:navsim_navigate -- --more packages/navsim/maps/medium_warehouse.json,packages/navsim/robots/carter.json

Carter should begin moving towards its goal as indicated by the semi-transparent green Carter. You can move the goal by dragging the green Carter to different locations: Carter will re-plan its path accordingly.

Unity Editor with "medium_warehouse" scene running

Communication with Isaac SDK

The IsaacSim Unity3D Core uses the Isaac SDK C API and C# binding to create an Isaac application inside Unity and pass messages between the Isaac application and Unity scripting components.

Overview of communication between Unity and Isaac

For a typical use case (shown in the diagram above), the Unity scene creates an Isaac application (navsim.app.json) with a TcpPublisher node. In the Unity scene, on every frame, a CameraSimulator C# script (1) attached to a Unity camera renders an RGB image to an off-screen buffer, creates an Isaac ColorCameraProto message with the image buffer, and publishes the message to the message ledger of the Isaac TcpPublisher node (2). A separate Isaac application running outside of Unity can subscribe (3) to the TcpPublisher, receive the simulated RGB image, and perform subsequent computations (4) on it.

The Isaac Application in Unity

The IsaacApplication script takes an Isaac-application JSON filename. When Start() is executed on this script after Unity starts running, the script creates an Isaac application from the specified application JSON file and starts it. The Isaac application is stopped and destroyed right before Unity stops running.

To add the IsaacApplication script and necessary C/C# API bindings to a scene, attach the Core package > Prefabs > isaac.alice prefab to the scene.

Message Acquisition Time

Unity time is measured in seconds since the start of the game, and is advanced on every frame. Thus, every Isaac message published by the IsaacComponent within the same frame in Unity has the same acqtime.

Isaac Application time is measure in nanoseconds since the start of the application. Since the Isaac Application running in Unity is started after the Unity simulation, there is an offset between the Unity clock and Isaac clock. This offset is measured once at the beginning of each Unity frame and compensated for with each Isaac message published/received in that frame.

In addition, the Isaac Application (navsim.app.json) running inside Unity (1) and the Isaac Navigation Application receiving simulation data (3) also have different clocks. The Isaac::Alice::TimeSynchronizer component is added to the TcpPublisher and TcpSubscriber nodes in both applications to compensate for this offset.

Message Conversion

Isaac SDK messages are based on Cap’n Proto, which is not well supported in C#. To bypass this limitation, Isaac SDK also supports conversion between JSON and Capnp proto. The IsaacComponent C# script generates messages in the JSON & buffers format, where most of the high-level data is formatted as JSON objects and large data blocks such as images and tensors are stored in byte buffers. The Isaac application running in Unity (navsim.app.json) converts these JSON messages to their corresponding Capnp proto messages before publishing them over TCP.

Message Routing

The following diagram outlines how messages connect IsaacSim Unity3D and different Isaac application components:

Unity3D messaging workflow

Each IsaacComponent script in Unity that publishes or receives Isaac messages must specify a message channel in the Isaac application in Unity. The message channel is specified with a node name, component name, and channel tag. In the diagram above, the Isaac application (navsim.app.json) has a “navsim” node, with an “output” component of type TcpPublisher (2). The CameraSimulator script (1) in Unity is configured to pulish to the channel navsim/output/color, where “color” is the channel tag. The TcpPublisher (2) publishes all messages with their corresponding tags to the same channel (navsim/output).

Coordinates

Isaac SDK uses a right-handed coordinate system (x:forward, y:left, z:up), while Unity uses a left-handed coordinate system (x:forward, y:up, z:left). Because of this, both rotation and position are converted to right-handed coordinates in Unity scripts before messages are published from Unity. Similarly, poses received from Isaac are converted to left-handed coordinates when messages are received in Unity. This difference in coordinate systems also applies to Isaac and Unity camera frames.

Isaac coordinates vs Unity coordinates

Left: Isaac coordinates – Right: Unity coordinates

Isaac camera frame vs. Unity camera frame

Left: Isaac camera coordinates – Right: Unity camera coordinates

Simulator Features

This section outlines important IsaacSim Unity3D features and describes how to use them.

Sensor

IsaacSim Unity3D supports simulation of camera, lidar, and IMU sensor data:

Camera

IsaacSim Unity3D can generate color and depth camera images, which can also be labeled.

Selecting Isaac camera prefabs

The core package contains the “Color Camera” and “Segementation Camera” prefabs in the Prefabs>Sensors directory.

Configuring Isaac camera components
  • If you enable the Capture Depth option, the prefab will also generate a depth image.
  • Use the Width and Height variables to specify the size (in pixels) of the output image.
  • You can specify the Field of View (vertical) of the camera in the Camera component.
  • If you wish to view the camera image in Game view rather than publishing it
    to Isaac, disable the Capture option. You may also need to switch the display of Game view to the “Target Display” of the “Camera” component.
  • Because the OpenGL frame arranges data from bottom to top, the Flip Vertical option must be
    enabled for publishing data to Isaac. Disable this option if you are viewing the camera image in Game view.
RGB(D) Camera

The UnityCamera.cs script publishes an RGB image (24 bit) and an optional depth image (32 bit float).

Segmentation Camera

The SegmentationCamera.cs script publishes a label image (8 bit) and an instance image (16 bit).

To generate the above images, add the BasicClassLabelManager prefab to the ‘medium_warehouse” scene and enable Carter > robot > SegmentationCamera. In this case, the floor consists of tiles of meshes, thus each tile gets a different instance value but the same label value.

Label Manager and Label Setter

To generate segmentation images, add the BasicClassLabelManager prefab to a scene.

Selecting the Isaac label-manager prefab

If you attach the LabelSetter.cs component to a GameObject, then label and instance values are assigned to all renderers of that GameObject and its children. Label values are assigned based on the Class Label Rules of the ClassLabelManager script, which is a regular expression used to match the Label Name in the LabelSetter script. When a SegmentationCameraProto message is published, the Class Labels in the ClassLabelManager script are converted into a labels list. Instance values are incremented for each renderer.

Configuring the label-manager component

The above image shows the ClassLabelManager script and two game objects with the LabelSetter script attached to them. The ClassLabelRules script assigns label value “1” to LabelSetter scripts with a Label Name of “floor”, “0” to LabelSetter scripts with a Label Name of “wall”, and “2” to all other LabelSetter scripts with a Label Name of one or more characters (e.g. with the “shelves_back” object).

Flatscan Lidar

The RaycastFlatscanLidar script implements a simulated 1D lidar using the Unity Physics.Raycast()_ method. In addition, range is added down the pipeline with the FlatscanNoiser codelet in Isaac. Since the Raycast() method is based on the Collider class, you must configure a mesh or other colliders on a GameObject for it to be visible to the lidar.

Unity flatscan lidar component

IMU

Inertial measurement unit (IMU) simulation is performed by the ImuSim codelet in IsaacSDK. To enable IMU simulation, add an “imu” GameObject with a Rigidbody component and attach it to the robot Rigidbody with a fixed joint. The RigidBodiesSink script publishes the Rigidbody state of the “imu” GameObject, which is used by the ImuSim codelet in Isaac to simulate IMU data.

Unity IMU component IMU raw output

The raw IMU output of simulated linear acceleration and angular velocity

Actuator

IsaacSim Unity3D supports simulation of a differential base with the Unity Wheel Colliders. The UnityDifferentialBaseSimulation script receives DiffBase commands, computes the desired wheel rotation speed from the commands, and applies torque to the wheels using a simple proportional controller. The script also retrieves the current speed of the robot from the Rigidbody.velocity property and computes the acceleration based on the velocity of the current and previous frame. It then publishes the DiffBase state.

../../_images/unity_differential_base.jpg

Monitor

The Pose and Collision script publish ground-truth states from the Unity Simulation to Isaac.

Pose

The IsaacPose and RigidBodiesSink helper script publish ground-truth pose and motion states from Unity.

The IsaacPose script publishes a PoseTreeEdge proto message containing the pose of its GameObject. This can be added to the pose tree of the receiving Isaac application with the PoseMessageInjector codelet.

../../_images/unity_pose_scripts.jpg

The RigidBodiesSink script publishes the rigidbody state (pose, velocity, and acceleration) of a list of GameObjects in the same reference frame. If the game object has a Rigidbody component, the Rigidbody.velocity is used–otherwise velocity is estimated from the position change between frames.

../../_images/unity_rigidbodiessink_script.jpg

Collision

Attach the CollisionMonitor script to a rigidbody to publish a CollisionProto message when the Rigidbody enters a collision.

../../_images/unity_collision_script.jpg

Scenario Management

The ScenarioManager codelet in Isaac can be used to request the Unity simulator to load a certain scene (level) and scenario while it is running. This allows you to reload a level whenever an Isaac application is connected. It also allows you to test different scenarios within the same level by reconfiguring the Isaac application without stopping the simulator.

SceneLoader

The SceneLoader script (attached to the isaac.alice prefab) receives requests for a scene and scenario and replies when the requested scene and scenario are loaded. Once a request is received, the SceneLoader script will unload the current active scene and load the requested scene. To keep the Isaac application running inside Unity during the scene switch, the SceneLoader script marks the isaac.alice prefab instance in the first scene as persistent and disables that isaac.alice prefab instance in the subsequently loaded scene.

../../_images/unity_sceneloader_script.jpg

Scenario

Each scene can contain scenarios for the robot to handle. A “scenario” refers to a choice made by the robot, start and goal poses, a list of static or dynamic actors, etc. A ScenarioManager script can selectively enable scenarios in its children.

../../_images/unity_scenarios.jpg

Scenarios can be hard-coded in a Unity scene, configured using JSON files (i.e the ScenarioFromFile prefab and script), or configured using Isaac ActorGroupProto messages (i.e. with ScenarioFromMessage).

../../_images/unity_scenarios_2.jpg

An example Unity scenario that allows the user to change the goal at run time

../../_images/unity_scenario_loading.jpg

Loading a scenario from message (left) or from file (right)

Randomization

IsaacSim Unity3D supports a rich set of asset randomization features inside Unity, allowing you to create infinite variants of a scene for testing the navigation stack and training perception models. This section gives a high-level overview of how to use randomization with IsaacSim. For implementation details, refer to the documentation and scripts in the Core package/Scripts/Runtime /Generation folder.

Asset Group

An asset group is a recursive data structure that stores a collection of assets. Randomization scripts can pick assets from this data structure. For example, GameObjectAssetGroup allows for spawning of game objects, while SubstanceMaterialAssetGroup allows for creation and randomization of materials using Substance Source. Asset groups are realized as ScriptableObjects, allowing them to be easily stored as asset files, which can be reused in many different spawners across many scenarios.

To create a new GameObjectAssetGroup, right-click the Project tab, select Create > GameObjectAssetGroup, and add prefabs to the “templates” list or another GameObjectAssetGroup to the “children list”. Asset groups are used as input variables to asset-spawner scrips.

Similarly, to create a new SubstanceMaterialAssetGroup, right-lcik the Project tab, select Create > SubstanceMaterialAssetGroup, and add Substance materials or another SubstanceMaterialAssetGroup to it.

GameObjectAssetGroup example Substance source example

Example of a GameObjectAssetGroup (top) and SubstanceMaterialAssetGroup (bottom)

Randomizer

A basic randomizer can randomize certain properties of a GameObject or component. For example, TransformRandomizer randomizes the transform (pose) of the GameObject. LightRandomizer randomizes the color and intensity of light. ColorGradientRandomizer randomizes the PoseProcessVolume component saturation, contrast, and exposure for image postprocessing.

AssetSpawner

An asset spawner is a mechanism to spawn one or more assets using an available geometric method. Assets can, for example, be spawned inside an area or volume (ColliderAssetSpanwer) or in a grid (TiledAssetSpawner). These spawners also have parameters that define transform randomization or the minimum required spacing of the spawned assets.

Note

We currently do not perform collision check to reject physically impossible poses.

../../_images/unity_spawner_and_randomizer.jpg

(Left) A ColliderAssetSpawner with a TransformRandomizer that spawns Gameobjects from a GameObjectAssetGroup – (right) A TileAssetSpawner with a SubstanceMaterialRandomizer to select random materials for the floor tiles.

You can also use AssetSpawner to randomly spawn GameObjects, which themselves have Randomizer scripts. The below example randomly spawns point lights in a box collider, while the color and intensity of the point lights are also randomized.

../../_images/unity_light_spawner.jpg

RandomizerGroup

All Randomizers (asset spawners, light randomizer, ect) implement the IRandomizer interface. Add the RandomizerGroup script to the root GameObject of a set of randomizers to control the random seed and randomization trigger (on start, on update, etc). The Randomness variable of a RandomizerGroup needs to be set to reference a Randomness script in the scene.

../../_images/unity_randomizer_group.jpg

Example of a RandomizerGroup in the “pose_estimation_training” scene.

Tools

This section outlines tools available in IsaacSim Unity3D.

Map Generation

The Isaac navigation stack requires a 2D occupancy map for localization. The Map Camera is a convenient tool to generate such a map for a Unity scene using the depth imaging of an orthographic camera. Follow these steps to use the Map Camera tool:

  1. Add the prefab in Core package/Prefabs/Sensors/Map Camera to your scene.
  2. Select the GameObject in the Inspector tab.
  3. Select Place Camera. This will set the camera position and near/far clip plane. All visible meshes between the near and far clip planes are projected as “obstacle” in the map.
  4. Switch the “Game” view to “Display 2” to preview the map.
  5. Once ready, select Save Map to save the PNG file and the corresponding map config JSON (for Isaac SDK).

Once the map is generated, remove or disable the “Map Camera” prefab to avoid rendering the map camera at run time.

../../_images/unity_map_generator_1.jpg ../../_images/unity_map_generator_2.jpg

AprilTag

AprilTag is a visual fiducial system commonly used in robotics. You can find the AprilTag prefab in Core package/Prefabs. This is implemented with a plane mesh, which at a scale of 1 is 10m by 10m.

../../_images/unity_apriltag.jpg

Visualization

The “isaac.minisight” prefab visualizes the local and global plan in Unity.

../../_images/unity_visualization.jpg

Build

Follow these steps to build a scene:

  1. In the Unity Editor, open File > Build Setting.
  2. Select Add Open Scenes to add scenes open in Unity Editor to the build setting.
  3. Click Build.

This should generate a <scene>.x86_64 executable and a <scene>_Data folder. Run the executable to launch the simulator. If you want to move or copy your build, you need to copy both the “.x86_64” binary and the “_Data” folder.

../../_images/unity_build_settings.jpg
Multiple scenes in a Single Binary

IsaacSim Unity3D provides a few helper functions to build multiple scenes into a single binary. These are in the BuildHelper and MultiSceneTools scripts. In the Unity Editor window, you can also use Isaac > Build Scenes in Folder to select a folder within your project Asset folder to create a build that includes all the scenes in that folder; alternatively, you can use Isaac > Build Scenes in File to select a JSON file with a list of scenes in your current project and create a build that includes them all.

Using Static Methods

You can also use the static methods in the BuildHelper script to create a build from the Unity command line. For example, you can create a warehouse.json file with the following content:

{
 "scenes": [
   "Packages/com.nvidia.isaac_sim_samples/Warehouse/Scenes/small_warehouse.unity",
   "Packages/com.nvidia.isaac_sim_samples/Warehouse/Scenes/medium_warehouse.unity",
   "Assets/tutorial.unity"
 ]
}

Then run the following command to create a build:

bob@desktop:~$: Unity/Hub/Editor/2018.3.14f1/Editor/Unity -quit -batchmode -logFile -projectPath isaac_sim_unity3d/projects/sample -executeMethod Isaac.BuildHelper.BuildFromFile --sceneFile warehouse.json

This creates a warehouse.x86_64 and warehouse_Data file in isaac_sim_unity3d/projects/sample/Builds containing the three scenes listed above. The name “warehouse” in the executable matches that of the warehouse.json scene file. Note that the paths in the JSON file are given relative to the Unity project path, and scenes in the package are referred to by the package name “com.nvidia.isaac_sim_samples” instead of the display name “NVIDIA IsaacSim for Unity3D (Sample)”. The build also only works for scenes that are part of the project.

Multi-Robot Simulation

IsaacSim Unity3D supports simulation of multiple robots in the same world, with each robot driven by an independent Isaac SDK navigation brain. This is implemented using a server-client structure with multiple simulation instances to allow for scaling to a large number of robots.

The figure below depicts communication between simulation instances and Isaac robot brains. Blue boxes represent Unity3D instances and green boxes represent Isaac SDK navigation app instances:

  • The Sim Server receives base commands for all robots, performs physics updates, and publishes the base state of each robot to its corresponding Sim Client. It also broadcasts the pose of all robots to all Sim Clients using the Teleport message.
  • The Sim Client receives the teleport message and teleports both the ego robot and other robots to their corresponding positions. It then performs sensor simulation (camera + lidar) and publishes the sensor data to the robot brain.
  • The Sim server/client structure is completely obscured from the robot brain: The same navigation app (navsim_navigate) can be used whether the simulator it connects to is in standalone or client mode.
Multi-robot simulatin diagram

The Sim Server and Sim Client are implemented using variants of the navsim.app.json application run by the simulation, as well as the ScenarioFromFile.cs script, which supports loading a scenario from file. In this case, the scenario file specifies the number of robots, their Isaac nodes, prefab variants, and spawn poses. The Isaac SDK provides sample apps and configs for running up to six robots in the apps/navsim/multirobot folder.

Running the Demo App

Follow these steps to launch the simulators and robot brains in the “medium_warehouse” scene:

  1. Follow the steps in the Build section to build the “medium_warehouse” scene.

  2. Deploy //apps/navsim/multirobot:navsim-pkg with the “medium_warehouse” scene:

    bob@desktop:~$ cd isaac
    bob@desktop:~/isaac$ ./engine/build/deploy.sh -p //apps/navsim/multirobot:navsim-pkg\
     -d x86_64 -h localhost --deploy_path ~/deploy/bob/medium_warehouse_Data/StreamingAssets
    

    The above command assumes the medium_warehouse.x86 scene is placed in ~/deploy/bob.

  3. Launch the Sim Server:

    bob@desktop:~$ cd ~/deploy/bob
    bob@desktop:~/deploy/bob$ ./medium_warehouse.x86_64\
       --app apps/navsim/multirobot/navsim_server.app.json\
       --scenario 0 --mode server\
       --scenarioFile apps/navsim/multirobot/scenarios/multirobot_server.json
    

    You should see a top-down view of the “medium_warehouse” scene, with six robots on the right.

Top down view of the medium_warehouse scene
  1. Launch Sim Client 0 and 1:

    bob@desktop:~$ cd ~/deploy/bob
    bob@desktop:~/deploy/bob$ ./medium_warehouse.x86_64\
       --app apps/navsim/multirobot/navsim_client.app.json\
       --more apps/navsim/multirobot/configs/navsim_client_00.json\
       --scenario 0 --mode client --clientIndex 0\
       --scenarioFile apps/navsim/multirobot/scenarios/multirobot_client.json&
    bob@desktop:~/deploy/bob$ ./medium_warehouse.x86_64\
       --app apps/navsim/multirobot/navsim_client.app.json\
       --more apps/navsim/multirobot/configs/navsim_client_01.json\
       --scenario 0 --mode client --clientIndex 1\
       --scenarioFile apps/navsim/multirobot/scenarios/multirobot_client.json
    
  2. Wait for all simulators to launch: You should now have three Unity windows open. The main camera is disabled in client mode, so the Unity windows for the two Sim Clients are black.

  3. Launch the robot brains for robot 0 and 1:

    bob@desktop:~$ cd isaac
    bob@desktop:~/isaac$ bazel run //apps/navsim:navsim_navigate -- --more apps/navsim/multirobot/configs/navsim_navigate_00.json,packages/navsim/maps/medium_warehouse.json,packages/navsim/robots/carter.json&
    bob@desktop: ~/isaac$ bazel run //apps/navsim:navsim_navigate -- --more apps/navsim/multirobot/configs/navsim_navigate_01.json,packages/navsim/maps/medium_warehouse.json,packages/navsim/robots/carter.json&
    

    The first two robots will start moving once both apps are launched. You can open Sight at localhost:3000 for robot 0 and localhost:3001 for robot 1.

Multi-robot navigation
  1. Kill all simulator and robot-brain processes with the following commands:

    bob@desktop:~$ for pid in $(pidof engine/alice/tools/main); do kill -9 $pid; done
    bob@desktop:~$ for pid in $(pidof ./medium_warehouse.x86_64); do kill -9 $pid; done
    

Use the commands in step 4-6 to launch additional robots. The maximum number of robots is dependent on your system hardware, as well as the complexity of the Unity scene. For the “medium_warehouse” scene on a Titan V, you can run up to three robots (Sim Client + robot brain) and the Sim Server.

Debugging a Sim Server or Sim Client

For debugging, you can run a Sim Server or Sim Client in Unity Editor. Open the “medium_warehouse” scene in Unity Editor and follow these steps:

Sim Server
  1. Deploy the //apps/navsim/multirobot:navsim-pkg into the sample project StreamingAssets folder:

    bob@desktop:~$ cd isaac
    bob@desktop:~/isaac$ ./engine/build/deploy.sh -p //apps/navsim/multirobot:navsim-pkg\
     -d x86_64 -h localhost --deploy_path ~/isaac_sim_unity3d/projects/sample/Assets/StreamingAssets
    
  2. In Scenario Manager, set the “Active Scenario” to the index of the “ScenarioFromFile” scenario in the child GameObjects–in this case, 0.

../../_images/unity_debugging_scenario_1.jpg
  1. Click on the “ScenarioFromFile” GameObject. In the “Scenario From File (Script)” component, change the “Mode” from “Standalone” to “Server”, and set the “Filename” to “apps/navsim/multirobot/scenarios/multirobot_server.json”.
../../_images/unity_debugging_scenario_2.jpg
  1. Click on “isaac.alice” and change the “App Filename” of the “Isaac Application” component to “apps/navsim/multirobot/navsim_server.app.json”
  2. Click “Run”. You should see six robots spawned on the right side of the scene, the “ScenarioFromFile” GameObject activated, and all six robots added to the “Objects” of the “Teleport” script, as shown below:
../../_images/unity_debugging_server.jpg

If you check each robot’s IsaacComponents, the node name should be “robot_0” for “carter_0”, etc.

If you encounter a problem, first check that the filenames in step 3 and 4 are correct and do not contain special characters. Then check out the Troubleshooting section below.

Sim Client

To test a Sim Client instance, follow the same steps as the Sim Server section above, but replace step 3 and 4 with the following:

  1. Click on the “ScenarioFromFile” GameObject. In the “Scenario From File (Script)” component, change the “Mode” from “Standalone” to “Client” and set the “Filename” to “apps/navsim/multirobot/scenarios/multirobot_client.json”
  2. Click on “isaac.alice” and change the “App Filename” of the “Isaac Application” component to “apps/navsim/multirobot/navsim_client.app.json”.
../../_images/unity_debugging_client.jpg

If you check each robot’s IsaacComponents, the node name should be “navsim”. The first robot, “carter_0”, should have IsaacComponents for sensors, while the other robots are “dummies”: meshes with no IsaacComponents attached.

Note

After debugging, deploy the default //package/navsim/apps:navsim-pkg into the StreamingAssets of the project so you can resume using standalone mode:

bob@desktop:~$ cd isaac
bob@desktop:~/isaac$ ./engine/build/deploy.sh -p //package/navsim/apps:navsim-pkg\
 -d x86_64 -h localhost --deploy_path ~/isaac_sim_unity3d/projects/sample/Assets/StreamingAssets

Using a Custom Scene

Follow these steps to enable multi-robot simulation in your own scene:

  1. Add the ScenarioFromFile prefab (in Core packages > Prefabs) to the scene. If you already have a scenario manager in the scene, add the Evaluation prefab as a child; otherwise, add it as a top-level GameObject.
  2. Update the robot poses in the apps/navsim/multirobot/scenarios/multirobot_server.json so that the initial poses of all robots are valid, e.g. inside the scene and not colliding with objects in the scene or other robots.

You can now run multi-robot simulation as detailed in Running the Demo App–just substitute “medium_warehouse” with your own scene. You can also run debugging in Unity Editor by following the Debugging a Sim Server or Sim Client steps above.

Using your Own Robot

If you want to replace Carter with your own robot, create the following prefabs for your robot:

  • “MyRobot_Dummy”: This is the base prefab, and should contain only the mesh renderer and collider for the robot, with no sensors or actuators. This is used in client mode for non-hero robots. See “Carter_Dummy.prefab” in Samples package > Robots as an example.
  • “MyRobot_Sensor”: This is a prefab variant of “Robot_Base”. It additionally contains IsaacComponents for sensors such as the lidar and color camera. This is used in client mode for the hero robot. See “Carter_Sensor.prefab” in Samples package > Robots as an example.
  • “MyRobot_Base”: This is a prefab variant of “Robot_Base”. It additionally contains the Differential Base actuator script and IMU. This is used in server mode for all robots. See “Carter_Base.prefab” in Samples package > Robots as an example.

Add all three prefabs to the “Robots” GameObjectAssetGroup in Samples package > Robots.

Open the multirobot_client.json and multirobot_server.json files in apps/navsim/multirobot/scenarios/. Replace "prefab": "Carter" with "prefab": "MyRobot".

Follow the steps detailed in Running the Demo App. You should see your own robots in the scene.

Command Line Arguments

The following command line arguments are used in IsaacSim Unity3D scripts when running a build.

Option Script Description
–scene name SceneLoader Name of the scene to load (e.g. “small_warehouse”)
–scenarioFile path ScenarioFromFile Path to the scenario file
–scenarioFile index ScenarioManager Index of the scenario to set to active
–timeScale number TimeScaler Timescale for Unity (only works for >=1)
–root path IsaacNative Path to the Isaac application root folder (navsim-pkg)
–app path IsaacApplication Path to the Isaac application JSON file relative to root
–more files IsaacApplication A comma-separated list of additional JSON files relative to root for the Isaac application to load
–buildTarget path BuildHelper Output filename of the build
–sceneFile path BuildHelper.BuildFrom File Path to the Isaac application root folder (navsim-pkg)
–mode [client/server/ standalone] ScenarioFromFile Which mode to run from the simulator

Unity also supports a set of command line arguments. Useful arguments include the following:

  • “-logFile”: Dumps Unity logs to the console.
  • “-screen-fullscreen 0[1]”: Disables/enables fullscreen.

Creating a Unity project with IsaacSim

This section describes how to create your own Unity project with IsaacSim. It assumes basic proficiency with the Unity Editor. If you are new to Unity, check out the Unity Learning resources and tutorials first.

Create a New Project

IsaacSim Unity3D has custom project settings and package dependencies. The easiest way to create a new project with Isaac is to copy the Assets, Packages, and ProjectSettings folders from the sample project:

bob@desktop:~/isaac_sim_unity3d$ mkdir projects/test
bob@desktop:~/isaac_sim_unity3d$ cp -r projects/sample/Assets projects/sample/Packages projects/sample/ProjectSettings projects/test/

Next, open the new project:

bob@desktop:~/isaac_sim_unity3d$ ~/Unity/Hub/Editor/2018.3.11f1/Editor/Unity -projectPath projects/test -logfile

Note

The 2018.3.11f1 directory in the above command may differ if you have a different version of Unity installed.

Wait for the project to load (this takes about 10 minutes). Once done, you should see an empty scene in Editor:

New project in Unity

Create a New Scene

First, you need to create a new scene:

  1. Right-click the Project tab.
  2. Choose Create > Scene.
  3. Rename the scene to “test”.
  4. Open the scene.
New scene
  1. Create an empty GameObject and name it “world”.
  2. Add a plane as the ground and a few cubes as walls.
  3. Add the “Map Camera” prefab to the scene.
  4. Save the map as “/tmp/test.png”.
  5. Remove the “Map Camera”.
New scene with map added

Add Random Obstacles

Next, you can add random obstacles to the scene:

  1. Create a new empty GameObject and name it “random scene”.
  2. Use Add Component in the Inspector to add the “Randomness” and “Randomizer Group” scripts.
  3. Drag the “Randomness” component into the “Randomness” field of the “Randomizer Group” script.
Unity scene with randomized obstacles
  1. Add an empty child GameObject named “random boxes”.
  2. Add a box collider and place it in the top-right empty space.
  3. Use the Add Component button to add a “Collider Asset Spawner” script and a “Transform Randomizer” script.
  4. Modify the following settings:
  1. Go to Project > NVIDIA IsaacSim for Unity3D (Samples) > Warehouse > Props > Groups
  2. Drag “CardboardBoxProps” into the “Asset Group” of “Collider Asset Spawner”.
  3. Drag the “Transform Randomizer” component into the “Transform Randomizer” field of the “Collider Asset Spawner”.
  4. Change the “Max Count” of “Collider Asset Spawner” to 6, and the “Rotation - Uniform Range - Y” of “Transform Randomizer” to 180.
  1. Go to the “random scene” GameObject and press the Randomize button. You should see 6 cardboard boxes drawn from the “CardboardBoxProps” group randomly spawned within the box.
  2. Press Randomize again to re-spawn the boxes.
Unity scene with spawned boxes
  1. Duplicate the “random boxes” GameObject.
  2. Rename the GameObject to “random trashcans”.
  3. Move the BoxCollider to the empty space in the lower left corner.
  4. Change the “Asset Group” of the “Collider Asset Spawner” to “None”.
  5. Go to Project > NVIDIA IsaacSim for Unity3D (Samples) > Warehouse > Props.
  6. Drag “TrashCan01” into the “Prefab” of “Collider Asset Spawner”.
  7. Go to “random scene” GameObject and press the Randomize button> You should see 6 trashcans randomly placed.
Unity scene with randomized trashcans

You can also place your own models in the scene. Make sure the models have a Collider attached and a minimum height of 0.7m so that the simulated lidar on Carter can see them.

Change the main camera position and rotation as shown below to get a bird’s eye view of the scene. If you press Play now, you should see a static scene in Game View. Press Stop.

Static Unity scene in Game View

Add Carter and Isaac Applications

Now you’re ready to add a robot to the scene:

  1. Go to Project > NVIDIA IsaacSim for Unity3D (Samples) > Robots.
  2. Drag the “Carter_Wheeled” prefab into the scene. You should now see a Carter in the center of the scene.
Unity scene with Carter
  1. Go to the Console tab and enable “Error Pause”.

Note

If you press Play now, you will see the game pause immediately and the error message “Access to IsaacNative singleton before Awake” because you have not added the “isaac.alice” prefab that loads the Isaac SDK C API and runs the NavSim app.

  1. Stop the game and go to Project > NVIDIA IsaacSim for Unity3D (Core) > Prefabs.
  2. Add the “isaac.alice” prefab to the scene.
  3. Add the “isaac.minisight” prefab to the scene to visualize the global and local plan.

Now, when you press Play, the simulation should start without error.

Unity scene with Carter playing

Open a terminal, go to the Isaac SDK folder, and run the navigate app with the map that we saved earlier for this scene:

bob@desktop:~/isaac$ bazel run apps/navsim:navsim_navigate -- --more /tmp/test.json,packages/navsim/robots/carter.json

Open Isaac Sight in Chrome browser at http://localhost:3000/:

Isaac sight showing custom Unity scene

You should see Carter localizing correctly and receiving Lidar and RGBD camera data. However, it remains stationary because the goal behavior is set to “pose” but no goal pose is being published from Simulation.

In Sight, go to Application Configuration > goals.goal_behavior, change the “desired_behavior” from “pose” to “random” and press Submit. You should see a goal pose displayed in the “Map View” in red, with Carter moving along the planned path.

Setting the Carter pose in Isaac sight Unity scene with Carter path

Troubleshooting

I double-click a Unity scene in the Sample package to open the scene, and this message pops up:

Unity "About to open another project!" message

Choose Cancel. Instead, drag the scene icon into the hierarchy and remove the previous scene.

I press Play and the Unity Editor crashes

In the Unity Editor Console tab, check if there’s any error before hitting “Play”. Enable Error Pause, this should pause the game if Unity catches any error. Lastly, if you launch the Unity Editor with the -logFile flag, Unity prints logs to the terminal. Check the terminal log for any errors: One common cause is that the Isaac SDK application running inside Unity crashes due to missing config files.

I press Play and get a lot of errors similar to “Could not invoke IsaacIsaacCreateApp”

Unity cannot load the isaac sdk c api lib. You need to deploy //packages/navsim/apps:navsim-pkg from the Isaac SDK into [your project]/Assets/StreamingAssets. For example, you would use this command to deploy to the sample project:

bob@desktop:~/isaac$ ./engine/build/deploy.sh -p //packages/navsim/apps:navsim-pkg -d x86_64 -h localhost --deploy_path ~/isaac_sim_unity3d/projects/sample/Assets/StreamingAssets

I add an IsaacComponent in Unity, but I’m not receiving data from the simulator

Check that the Isaac channel name in the IsaacComponent in Unity matches the channel name on the receiving Isaac app. Alternatively, sometimes the special character /x13 gets inserted in the channel name string by the Unity Editor. Search for and delete any instances of /x13 in the *.unity scene file or *.prefab file in your text editor/IDE.