Skip to navigation
Closed-Loop Testing

Isaac Sim Configuration

View as Markdown

Configuration files controlling Isaac Sim simulation behavior. Halos SIL runs on Isaac Sim 6.1 (nvcr.io/nvidia/isaac-sim:6.1.0) with Replicator Agent (IRA) config 1.7.0. Isaac Sim 6.1 refuses a 1.6.0 config, so a scene written for 6.0 shuts down at startup until its version is raised.

IRA 1.x (Isaac Sim 6.0 and later) is not config-compatible with IRA 0.x (Isaac Sim 5.1 and earlier). A 0.x config will not load. The three changes that matter most here: global.simulation_length (frames) became top-level simulation_duration (seconds); flat character/sensor blocks became named groups; and character behavior moved from text command files to JSON behavior trees. See the official IRA migration guide.

Configuration Files

Located in /isaac-sim/sil/configs/:

  • scenarios.yaml — one entry per scenario id, naming the four files below that make up a run
  • default_config_ros.yaml — IRA scene, characters, cameras, writer
  • cameras.yaml — camera prims, RTSP ports, and VST registration
  • robots.yaml — the fleet file: which trucks exist, where they spawn, how each drives, and the forklift control / odometry / safety-indicator and /clock Action Graphs
  • *-40x20.yaml — the same trio for the experimental two-dock, two-truck warehouse_40x20 scene
  • postprocessing.yaml — RTX sensor-anomaly presets (active: baseline = no degradation)
  • inspect_loop.bt.json, gather_bend_loop.bt.json, pickup_loop.bt.json — per-group character behavior trees

default_command.txt and default_robot_command.txt are the IRA 0.x omni.anim.people command files. Nothing in the 6.x pipeline reads them — they are kept only for reference against older recordings.

scenarios.yaml

One id describes a run, and Isaac Sim and every forklift controller read the same one, so they cannot be launched against different halves of it. Set it in the profile env file:

SCENARIO=warehouse_20x20_1fl
warehouse_20x20_1fl:
  ira:       default_config_ros.yaml   # IRA config; names the scene USD
  robots:    robots.yaml               # fleet file; its scene: key is checked against the IRA scene
  cameras:   cameras.yaml              # its scene: key is checked the same way
  waypoints: warehouse_20x20           # forklift-controller/waypoints/<dir>/<robot>.json
warehouse_40x20:
  ira:       default_config_ros_40x20.yaml
  robots:    robots-40x20.yaml
  cameras:   cameras-40x20.yaml
  waypoints: warehouse_40x20
  experimental: true                   # no published calibration: watch it, do not score it

A flag still wins over the scenario (-c, --robots-config, --cameras-config) for a one-off run. A different fleet in the same scene is a different run, so it gets an id of its own; WAYPOINTS_MAP is the one override, for a new route set in an existing scene.

default_config_ros.yaml

Main simulation configuration:

isaacsim.replicator.agent:
  version: 1.7.0
  seed: 42
  simulation_duration: 20.0
  environment:
    base_stage_asset_path: /isaac-sim/sil/scenes/indicator_warehouse_20x20_layout_overflow_test.usd
  character:
    groups:
      inspect_workers:
        num: 1
        asset_path: Isaac/People/Characters/
        behavior_tree: /isaac-sim/sil/configs/inspect_loop.bt.json
      gather_workers:
        num: 1
        asset_path: Isaac/People/Characters/
        behavior_tree: /isaac-sim/sil/configs/gather_bend_loop.bt.json
      pickup_workers:
        num: 1
        asset_path: Isaac/People/Characters/
        behavior_tree: /isaac-sim/sil/configs/pickup_loop.bt.json
    motion_library_path: Isaac/People/MotionLibrary/HumanMotionLibrary.usd
  sensor:
    groups:
      ceiling_cameras:
        num: 3
        aim_at_targets:
          height_range: [2.0, 3.0]
          look_down_angle_range: [0.0, 60.0]
          focal_length_range: [10.0, 15.0]
          distance_range: [6.5, 14.5]
  replicator:
    writers:
      IRABasicWriter:
        output_dir: ''
        rgb: false
        bounding_box_2d_tight: false
        # ... every other annotator also false ...
        start_frame: 0
        end_frame: 1800
        sensor_prim_list:
        - /World/Cameras/ceiling_cameras

Key Parameters

ParameterDefaultDescription
version1.7.0IRA config schema version. Isaac Sim 6.1 needs 1.7.0 and refuses 1.6.0.
seed42Random seed for reproducible scenarios (top-level in 1.x, was global.seed)
simulation_duration20.0Run length in seconds, not frames. IRA ticks the timeline at 30 Hz.
environment.base_stage_asset_pathUSD file pathWarehouse scene to load (was scene.asset_path)
character.groups.<name>.num1Digital humans in that group — 3 groups × 1 = 3 characters total
character.groups.<name>.behavior_treeJSON pathBehavior tree driving the group (replaces the 0.x command_file)
character.groups.<name>.asset_pathIsaac/People/Characters/Character USD directory, resolved against the asset root
character.motion_library_pathIsaac/People/MotionLibrary/HumanMotionLibrary.usdShared human motion library, resolved against the asset root
sensor.groups.ceiling_cameras.num3Cameras IRA places, using the aim_at_targets strategy
replicator.writers.IRABasicWriterall annotators offKept only to satisfy IRA’s writer requirement — see below

The writer no longer produces the camera streams. On IRA 0.x this file set writer: RTSPWriter and RTSP came out of the replicator writer. On 6.x the streams come from the RTSP multi-camera Action Graph built from cameras.yaml, so every IRABasicWriter annotator is switched off and nothing is written to disk.

Every camera is driven from a single OnPlaybackTick, routed through an IsaacSimulationGate rate divider before the render products. Its step input sets how many render-loop ticks are skipped between frames; at the shipped step: 1 every tick produces a frame, and the effective rate is whatever the render loop sustains. Raise step if the encoder starts duplicating frames on the wire and the DeepStream/VST pipeline backlogs.

cameras.yaml

Camera registration configuration for VST:

scene:
  - sil/scenes/indicator_warehouse_20x20_layout_overflow_test.usd
cameras:
  - name: Camera
    camera_prim: /World/Cameras/Camera
    port: 8554
    mount_path: /camera
    spawn:
      position: [0.5953581163925358, -15.33611879594478, 3.199676475989638]
      rotation_yxz_deg: [51.569855, 0.0, -92.38173]
      focal_length: 8.0
      horizontal_aperture: 20.955
      vertical_aperture: 15.2908
  - name: Camera_01
    camera_prim: /World/Cameras/Camera_01
    port: 8555
    mount_path: /camera_01
    # spawn: ...
  - name: Camera_02
    camera_prim: /World/Cameras/Camera_02
    port: 8556
    mount_path: /camera_02
    # spawn: ...
rtsp:
  host: ${HOST_IP}

Camera registration:

  • scene: The scene USDs these camera positions are measured in, checked against the scene loading (path-suffix match) before the stage opens
  • name: Camera identifier in VST
  • camera_prim: USD prim path of the camera in the stage
  • port: TCP port of the in-process RTSP server — unique per camera
  • mount_path: URL suffix (must start with /)
  • spawn: Optional. When present, camera_loader.py creates the Camera prim at camera_prim from these values at scene init, and the baked camera must be removed from the scene USD. Omit it to use a baked camera prim.
  • Full URL: rtsp://${HOST_IP}:${port}${mount_path}

When Isaac Sim starts with --enable-vst, cameras automatically register with VST using these definitions.

Two fields from the Isaac Sim 5.1 schema must not appear: per-camera rtsp_path (the RTSPWriter_World_Cameras_*_rgb string) and the top-level rtsp.port of the old external broker. vst_sensor_manager.py rejects a config that still carries them, and a scalar rtsp or cameras is a validation error.

robots.yaml

The fleet file. It drives the forklift Action Graph builders and the forklift controllers. One block per robot — adding a robot means adding a block here plus a controller service naming its ROBOT_ID, with no code change. What follows from the asset is stated once per model under models:; each robot names its model and keeps only its own choices. See Isaac Sim Action Graphs for what each builder produces and Forklift Controller for the drive: and proximity: keys.

scene: sil/scenes/indicator_warehouse_20x20_layout_overflow_test.usd   # checked before the stage opens
models:
  forklift_b:
    asset_path: https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/6.1/Isaac/Robots/IsaacSim/ForkliftB/forklift_b.usd
    drive_type: swivel
    drive_joint: back_wheel_drive       # velocity-controlled
    swivel_joint: back_wheel_swivel     # position-controlled steer
    wheelbase: 1.49
    wheel_radius: 0.15
    max_steer_deg: 45.0
    reverse_logic:
      flip_linear_x: true
      flip_angular_z: true
    robot_front: [-1.0, 0.0, 0.0]
    indicator:                          # the safety disc indicator_loader.py authors
      parent_prim_rel: body/body
      radius: 1.5
    drive:                              # forklift-controller, shared by every truck of this model
      heading_offset: 180.0
      no_invert: false
      angular_speed: 0.4
robots:
  - name: forklift_b                    # == the controller's ROBOT_ID == the prim leaf
    model: forklift_b
    articulation_prim: /World/forklift_b
    drive:
      speed: 1.5
      loop: true
    spawn:                              # authored into scenes/generated/, sublayering the scene
      position: [1.0, -13.39, 0.0]      # world metres; also the origin of every waypoint file
      yaw_deg: 180.0
    proximity:                          # PSF_APP=pxc / both
      reduce_speed: 0.5
      stop_hold_s: 2.0
      reduce_hold_s: 1.0
      stale_s: 10.0
    control:
      enabled: true
      cmd_vel_topic: forklift_b/cmd_vel # the controller reads this too
    odometry:
      enabled: true
      odom_topic: forklift_b/odom
      tf_topic: tf                      # frames forklift_b/odom -> forklift_b/base_link
    safety_indicator:
      enabled: true
      muted_topic: /safety/is_muted
proximity_line:
  enabled: false                        # draw each proximity decision's pair on the floor
clock:
  enabled: true
  clock_topic: clock

The reverse_logic flags exist because the forklift asset’s vehicle frame is opposite the ROS twist convention on both axes. Set them to false for a robot whose asset already matches ROS.

  • spawn — the scenes carry no forklift prim; forklift_overlay.py authors each truck from its spawn: block into a generated layer before IRA opens the stage. Declaring spawn: for a prim the scene already has is an error. The position must match the origin of the truck’s waypoint files; deployments/scripts/preflight.py compares the two.
  • safety_indicator — the disc shows /safety/is_muted under PSF_APP=atl, the proximity level (/<robot>/proximity/state: green / orange / red, grey before the first decision) under pxc, and STOP / REDUCE while proximity acts, otherwise the mute, under both. source: pins one. With two ATL zones, zone: 2 makes the disc follow the second bay.
  • proximity_line — off by default because it is scene geometry: the cameras, and therefore perception, see it too.

Forklift Motion

Forklift motion is driven by the forklift-controller service, not by a node inside the Action Graph. The controller reads a waypoint file and publishes cmd_vel (geometry_msgs/Twist); the Forklift Control Action Graph only converts that cmd_vel into wheel commands.

Author routes with the Forklift Waypoint Generator below. For the controller’s ROS interface, route-following behavior, and the drive: keys, see Forklift Controller.

sil/playback/segments.json (a list of duration / linear_x / angular_z steps) is left over from the pre-6.0 pipeline, where a playback script node inside the Action Graph read it directly. Nothing in the current codebase loads it.

Common Adjustments

Change Simulation Length

simulation_duration is in seconds:

isaacsim.replicator.agent:
  simulation_duration: 300.0   # 5 minutes

Test Without Humans

Set num: 0 on every character group, or delete the groups:

character:
  groups:
    inspect_workers:
      num: 0
    gather_workers:
      num: 0
    pickup_workers:
      num: 0

Change Character Behavior

Point a group at a different behavior tree. Trees are authored with omni.behavior.tree.ui and reference the omni.behavior.tree.core and omni.anim.behavior.tree node libraries — see the IRA configuration guide.

character:
  groups:
    inspect_workers:
      num: 1
      behavior_tree: /isaac-sim/sil/configs/my_loop.bt.json

Forklift Waypoint Generator

For curved, multi-segment forklift routes, author waypoints visually with the Forklift Waypoint Generator web tool (tools/waypoint-generator/). The tool overlays the warehouse floor plan (Top.png) and exports a JSON path that the SIL forklift controller consumes directly.

Forklift Waypoint Generator with a two-waypoint route drawn over the warehouse floor plan
A two-waypoint route. Left: mode selector and the waypoint list, each entry showing its odom and world coordinates. Right: the exported JSON, updated as you edit. The readout above the canvas tracks the cursor in pixel, world and odom coordinates at once, and the arrow on each marker is its heading.

Launch

Requires Node.js 20.19+ or 22.12+ (Vite 7 requirement).

cd tools/waypoint-generator
npm install
npm run dev
# open http://localhost:5173

Authoring a Path

  1. Set Origin — click Set Origin and click the forklift start on the map (or Use Default Forklift Start). This is the odom-frame reference.
  2. Add Waypoints — switch to Add Waypoint and click along the route. Use Shift+Scroll to set the heading before each click.
  3. Edit — click a waypoint to select, drag to move, or edit coordinates directly in the sidebar. To change the order, drag an item of the sidebar list onto another (it takes that item’s place) or use its Move up / Move down buttons; dragging a point on the map moves it and keeps the order.
  4. Save / Export — 💾 Save stores the path (Redux store + localStorage auto-save); Export downloads or copies the JSON.

Navigation: Alt+Drag or middle-mouse to pan, scroll to zoom, Shift+Scroll to adjust heading.

Coordinate Systems

  • Pixel — image coordinates (Top.png is 1920×1080)
  • World — Isaac Sim global coordinates (metres)
  • Odom — relative to the origin point

Export Format

The exported JSON is the input format of the SIL forklift controller:

{
  "origin": {"world_x": 1.0, "world_y": -13.39},
  "waypoints": [
    {"x": 12.44, "y": -0.01, "theta_deg": 0, "reverse": false,
     "world_x": 13.44, "world_y": -13.40, "velocity": 0.5, "note": "Waypoint 1"}
  ],
  "poses": [{"x": 1.0, "y": -13.39, "theta": 0.0}],
  "segments": [{"from_waypoint": 0, "to_waypoint": 1, "reverse": false, "pose_count": 21}]
}
  • origin — robot start in world coordinates
  • waypoints — user-defined points with odom (x, y) and world coordinates
  • poses — interpolated points for the smooth curved path (consumed by the controller)
  • segments — per-pair segment info, including the reverse flag

Using the Path in SIL

Each controller reads closed-loop-testing/forklift-controller/waypoints/<map id>/<ROBOT_ID>.json. Save the export as waypoints/warehouse_20x20/forklift_b.json to replace the default route, or under a directory of your own selected with WAYPOINTS_MAP (see Forklift Controller). The origin must be the truck’s spawn.position.

Calibration for New Scenes

The bundled public/Top.png and calibration match the closed-loop-testing loading-dock scene:

  • Scale factor: 49.51 pixels/metre
  • Default forklift start: (1.0, -13.39) in world coordinates

To target a different scene, replace public/Top.png and update src/config/calibration.json with that scene’s calibration, then set defaultForkliftStart from the truck’s spawn.position in robots.yaml.

Execution Script

Isaac Sim scenarios are executed using run_sdg.sh, a thin wrapper that locates Isaac Sim’s python.sh and runs run_actor_sdg.py. You can call either one.

Command Line Parameters

ParameterDescription
--scenarioScenario id from scenarios.yaml; defaults to SCENARIO from the environment and fills the IRA, robots and cameras configs
-c, --config_filePath to the IRA config file; optional when the scenario names one
--startAutomatically start data generation (no manual UI start needed)
--setup-onlySet the scene up but do not start generation
--headlessRun without a GUI window (for automated testing)
--enable-vstRegister the cameras as VST sensors
--cameras-configPath to cameras.yaml — drives both VST registration and the RTSP graph
--robots-configPath to robots.yaml — forklift control / odometry / safety plus the /clock graph
--srr-gtBuild the ground-truth /gt/*/tf publishers (default off; see Regression Testing Reporter Ground-Truth Graph (opt-in))
--width / --heightViewport resolution (default 1920×1080)
--vst-register-warmup-secSim-time seconds the timeline must advance under Play before VST registration (default 1.0)
--vst-register-timeout-secWall-clock ceiling before VST registers best-effort even if the render never warms (default 1200)
--postprocessing-config / --postprocessing-presetRTX sensor-anomaly presets (postprocessing.yaml); pin baseline for a reference run
--save_usdSave the USD scene after generation
--debug_printEnable debug output

The build steps are all on by default and can be switched off individually for debugging: --no-rtsp (RTSP graph), --no-forklift (control, odometry, safety indicator), --no-forklift-spawn (forklift prims from spawn:), --no-indicator-spawn (safety disc geometry), --no-clock (/clock publisher), --no-camera-spawn (dynamic camera prim spawn), --no-postprocessing, --no-runtime-patches.

Example execution:

# SCENARIO from sil.env supplies the IRA, robots and cameras configs
docker exec -d isaac-sim bash -lc 'cd /isaac-sim/sil/scripts && \
  ./run_sdg.sh --start --headless --enable-vst'

VST Integration:

  • --enable-vst registers cameras with VST automatically
  • Reads camera definitions from cameras.yaml
  • Constructs RTSP URLs: rtsp://${HOST_IP}:${port}${mount_path}
  • Requires the HOST_IP and VST_BASE_URL environment variables
  • Registration waits for the render to warm, which avoids the cold-bind has no caps failure on Isaac Sim 6.0 (see Isaac Sim Streaming)

Example with UI:

Allow X11 connections:

xhost +local:

Stop all SIL docker compose containers, then run cleanup and setup:

cd deployments
../closed-loop-testing/scripts/cleanup_all_datalog.sh sil
../closed-loop-testing/scripts/setup.sh sil

Start SIL docker compose:

docker compose --env-file profiles/sil.env up -d --build

Then start Isaac Sim with UI:

docker exec -it isaac-sim bash -lc 'cd /isaac-sim/sil/scripts && \
  ./run_sdg.sh --start --enable-vst'

Isaac Sim UI

Run headless for anything repeated. On a GeForce card the windowed UI leaks NVENC encode sessions across timeline stop/play and dies after roughly three replays (see Isaac Sim Streaming).

For the full IRA schema — character and robot groups, behavior trees, camera placement strategies, and writer options — see the IRA Configuration File Guide and the Actor Simulation tutorial.

Next Steps