Component API Overview

This following section contains a list of all components which are available in Isaac SDK. For each component, the incoming and outgoing message channels and the corresponding message types are listed. Additionally, all parameters with their names and types and corresponding default values are explained.

The following table gives an overview over all components. The columns ‘# Incoming’, ‘# Outgoing’ and ‘# Parameters’ indicate how many incoming message channels, outgoing message channels and parameters the corresponding module has.

isaac.AdafruitNeoPixelLedStrip

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• led_strip [LedStripProto]: The desired LED strip configuration message

Outgoing messages

(none)

Parameters

• bus [int] [default=1]: The I2C bus of the LED strip. Default value is 1

isaac.ArgusCsiCamera

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• image [ImageProto]: Channel to broad cast image extracted from argus feed
• intrinsics [CameraIntrinsicsProto]: Intrinsics including pinhole and distortion parameters

Parameters

• mode [int32_t] [default=]: Resolution mode of the camera. Supported values are:

  0: 2592 x 1944, 1: 2592 x 1458, 2: 1280 x 720

• camera_id [int32_t] [default=]: System device numeral for the camera. For example select 0 for /dev/video0.
• framerate [int32_t] [default=]: desired framerate
• focal_length [Vector2d] [default=]: Focal length of the camera in pixels
• optical_center [Vector2d] [default=]: Optical center in pixels

isaac.BlackFrameGenerator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• input_image [ImageProto]: Input frame

Outgoing messages

• output_image [ImageProto]: Output frame that’s either equal to input frame or pure black

Parameters

• opaque_tick_duration [int] [default=2]: The oscillation period is the sum of opaque and transparent ticks Defines the number of ticks in the period to publish a pure black frame
• transparent_tick_duration [int] [default=1]: Defines the number of ticks in the period to publish the unmodified input image

isaac.ImageComparison

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• input_image_a [ImageProto]: First input image
• input_image_b [ImageProto]: Second input image

Outgoing messages

(none)

Parameters

• correlation_threshold [float] [default=0.99]: The minimum correlation between two images where we will consider them the same
• down_scale_factor [int] [default=4]: Scaling of the displayed images in Sight

isaac.Joystick

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• js_state [JoystickStateProto]: The joystick message

Parameters

• deadzone [double] [default=0.05]: Size of the “deadzone” region, applied to both positive and negative values for each axis. For example, a deadzone of 0.05 will result in joystick readings in the range [-0.05, 0.05] being clamped to zero. Readings outside of this range are rescaled to fully cover [-1, 1]. In other words, the range [0.05, 1] is linearly mapped to [0, 1], and likewise for negative values.
• num_axes [int] [default=4]: Number of joystick axes (e.g., 4 axes might correspond to two 2-axis analogue sticks)
• num_buttons [int] [default=12]: Number of joystick buttons
• reconnect_interval [double] [default=1.0]: Reconnect interval, in seconds. This is the period between joystick connection attempts (i.e., attempts to open the joystick device file) when the initial attempt fails.
• input_timeout_interval [double] [default=0.1]: Input timeout interval, in seconds. This determines how long tick() will wait for input before giving up until tick() is called again. Note that stop() cannot succeed while tick() is waiting for input, so this timeout value should not be overly long.
• device [string] [default=”/dev/input/js0”]: Joystick device file (system-dependent)
• print_unsupported_buttons_warning [bool] [default=false]: Option controlling whether a warning will be logged when an event is received from an axis or button whose index exceeds num_axes or num_buttons, respectively

isaac.LivoxLidar

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• accumulated_point_cloud [PointCloudProto]: Output 3D point cloud samples. The point cloud is published when the point count reaches the configured minimum point count or when the time between message publishing is greater than the configured published interval.

Parameters

• device_ip [string] [default=”0.0.0.0”]: The IP address of the lidar device we want to connect to and receive data from. This parameter is changeable at configuration time.
• port_command [int] [default=50001]: The UDP port to send commands to the lidar. This parameter is changeable at configuration time.
• port_data [int] [default=50002]: The UDP port from which the data samples will be received. This parameter is changeable at configuration time.
• batch_count [int] [default=10]: Minimum number of accumulated point batches before publishing the point cloud. It can be configured and changed at runtime. The point cloud is published when the point count reaches the configured point batch count. Each batch is 100 data points per Livox communication protocol.

isaac.PanTiltDriver

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• command [StateProto]: Current command for pan/tilt unit

Outgoing messages

• state [StateProto]: The state of the pan tilt unit
• motors [DynamixelMotorsProto]: State of Dynamixel motors

Parameters

• use_speed_control [bool] [default=true]: If set to true dynamixels are controlled in speed mode, otherwise they are controlled in position mode
• usb_port [string] [default=”/dev/ttyUSB0”]: USB port used to connect to the bus (A2D2 USB adapter)
• pan_servo_id [int] [default=1]: Dynamixel ID for pan servo
• tilt_servo_id [int] [default=2]: Dynamixel ID for tilt servo
• tilt_min [double] [default=-0.5]: Minimum value valid for tilt
• tilt_max [double] [default=2.0]: Maximum value valid for tilt
• pan_min [double] [default=-Pi<double>]: Minimum value valid for pan
• pan_max [double] [default=Pi<double>]: Maximum value valid for pan
• pan_offset [double] [default=5.83]: Constant offset in the pan angle such as pan = 0 has the end effector looking forward (X axis)
• tilt_offset [double] [default=4.73]: Constant offset in the tilt angle such as tilt = 0 has the end effector looking horizontally
• baudrate [int] [default=static_cast<int>(dynamixel::Baudrate::k1M)]: Baudrate of the Dynamixel bus. See packages/dynamixel/gems/registers.hpp for options. TODO Remove when refactored to use DynamixelDriver class
• model [int] [default=static_cast<int>(dynamixel::Model::XM430)]: What kind of dynamixel model it is: (AX12A = 0, XM430 = 1, MX12W = 2) TODO(jberling) refactor pan tilt to use DynamixelDriver class and switch to enum
• pan_joint_frame [string] [default=”pan”]: Name of the pan joint frame. The edge pan_in_T_pan_out will be added to the PoseTree.
• tilt_joint_frame [string] [default=”tilt”]: Name of the tilt joint frame. The edge tilt_in_T_tilt_out will be added to the PoseTree.

isaac.RealsenseCamera

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• left_ir [ImageProto]: The left IR camera image
• right_ir [ImageProto]: The right IR camera image
• color [ImageProto]: The color camera image, which can be of type Image3ub for color or Image1ui16 for grayscale.
• depth [ImageProto]: The depth image (in meters) in the left IR camera frame
• color_intrinsics [CameraIntrinsicsProto]: Intrinsics including pinhole and distortion parameters for the color camera
• left_ir_intrinsics [CameraIntrinsicsProto]: Intrinsics including pinhole and distortion parameters for the left IR camera
• right_ir_intrinsics [CameraIntrinsicsProto]: Intrinsics including pinhole and distortion parameters for the right IR camera
• depth_intrinsics [CameraIntrinsicsProto]: Intrinsics including pinhole parameters for the depth camera

Parameters

• rows [int] [default=360]: The vertical resolution for both color and depth images.
• cols [int] [default=640]: The horizontal resolution for both color and depth images.
• ir_framerate [int] [default=30]: The framerate of the left and right IR sensors. Valid values are 90, 60, 30, 25, 15, 6. It should match the depth map framerate due to the RS 435 firmware constraints.
• color_framerate [int] [default=30]: The framerate of the RGB camera acquisition. Valid values are 60, 30, 15, 6.
• depth_framerate [int] [default=30]: The framerate of the depth map generation. Valid values are 90, 60, 30, 15, 6. It should match the IR framerate due to the RS 435 firmware constraints.
• align_to_color [bool] [default=true]: If enabled, the depth image is spatially aligned to the color image to provide matching color and depth values for every pixel. This is a CPU-intensive process and can reduce frame rates.
• frame_queue_size [int] [default=2]: Max number of frames you can hold at a given time. Increasing this number reduces frame drops but increase latency, and vice versa; ranges from 0 to 32.
• auto_exposure_priority [bool] [default=false]: Limit exposure time when auto-exposure is ON to preserve a constant FPS rate.
• laser_power [int] [default=150]: Amount of power used by the depth laser, in mW. Valid values are between 0 and 360, in increments of 30.
• enable_ir_stereo [bool] [default=false]: Enable acquisition and publication of the IR stereo pair. This setting can’t be changed at runtime.
• enable_color [bool] [default=true]: Enable acquisition and publication of the color frames. This setting can’t be changed at runtime.
• enable_depth [bool] [default=true]: Enable depth map computation and publication. This setting can’t be changed at runtime.
• enable_depth_laser [bool] [default=true]: Enable the depth laser projector to improve the depth image accuracy. Disabling it helps the visual odometry tracker by removing the dot pattern from the IR stereo pair. This setting can’t be changed at runtime.
• enable_auto_exposure [bool] [default=true]: Enable auto exposure. Disabling it can reduce motion blur
• dev_index [int] [default=0]: The index of the Realsense device in the list of devices detected. This indexing is dependent on the order the Realsense library detects the cameras, and may vary based on mounting order. By default the first camera device in the list is chosen. This camera choice can be overridden by the serial number parameter below.
• serial_number [string] [default=”“]: An alternative way to specify the desired device in a multicamera setup. The serial number of the Realsense camera can be found printed on the device. If specified, this parameter will take precedence over the dev_index parameter above.

isaac.SegwayRmpDriver

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• segway_cmd [StateProto]: Linear and angular speed command for driving segway (navigation::DifferentialBaseControl type)

Outgoing messages

• segway_state [StateProto]: State of the segway consisting of linear and angular speeds and accelerations (DifferentialBaseDynamics)

Parameters

• ip [string] [default=”192.168.0.40”]: Isaac will use this IP to talk to segway
• port [int] [default=8080]: Isaac will use this port to talk to segway
• flip_orientation [bool] [default=true]: If true, segway’s forward direction will be flipped
• speed_limit_linear [double] [default=1.1]: Maximum linear speed segway is allowed to travel with
• speed_limit_angular [double] [default=1.0]: Maximum angular speed segway is allowed to rotate with

isaac.SerialBMI160

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• imu [ImuProto]: IMU data including linear accelerations and angular velocities

Parameters

• device [string] [default=”/dev/ttyUSB0”]: Dev path where for the IMU device

isaac.SimpleLed

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• led_strip [LedStripProto]: The outgoing LED strip message for the driver to display

Parameters

(none)

isaac.SlackBot

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• slack_message [ChatMessageProto]: Messages to be sent to the slack server

Outgoing messages

• user_instruction [ChatMessageProto]: Messages received from the slack server

Parameters

• bot_token [string] [default=]: Slack bot token given on the slack app config page. A token can only be used by one Slackbot. Multiple robots on same token is not supported.
• slack_connect_url [string] [default=”https://slack.com/api/rtm.connect”]: Slack URL we will be sending the connection request too

isaac.TorchInferenceTestDisplayOutput

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• test_output [TensorProto]: Receives tensor output from the Torch inference

Outgoing messages

(none)

Parameters

(none)

isaac.TorchInferenceTestSendInput

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• test_input [TensorProto]: Sends tensors as input to Torch inference

Parameters

• input_value [float] [default=]:

isaac.V4L2Camera

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• frame [ImageProto]: Each frame output by the camera
• intrinsics [CameraIntrinsicsProto]: Intrinsics including pinhole and distortion parameters

Parameters

• device_id [int32_t] [default=0]: Which camera should be opened
• rows [int32_t] [default=720]: Parameters of the image requested by the camera These must match exactly with what the camera is able to produce. Number of pixels in the height dimension
• cols [int32_t] [default=1280]: Number of pixels in the width dimension
• rate_hz [int32_t] [default=30]: Frames per second.
• hardware_image_queue_length [int32_t] [default=3]: Buffers are queued with the V4L2 driver so that the driver can write out images at the specified frame rate without delays. This may be changed by the camera when we are initializing.
• focal_length [Vector2d] [default=(Vector2d{700.0, 700.0})]: Focal length (in pixels) for the pinhole camera model
• optical_center [Vector2d] [default=(Vector2d{360.0, 640.0})]: Optical center of the projection for the pinhole camera model
• brightness [int32_t] [default=]: Adjustable camera parameters. v4l2-ctl can be used to check values, e.g., “v4l2-ctl –device=/dev/video0 –list-ctrls”. Descriptions below are taken from video4linux API documentation. Picture brightness, or more precisely, the black level
• contrast [int32_t] [default=]: Picture contrast or luma gain
• saturation [int32_t] [default=]: Picture color saturation or chroma gain
• gain [int32_t] [default=]: Gain control
• white_balance_temperature_auto [bool] [default=]: If true, white balance temperature will be automatically adjusted.
• white_balance_temperature [int32_t] [default=]: This control specifies the white balance settings as a color temperature in Kelvin. White balance temperature needs to be between 2000 abd 6500. This parameter is inactive if white_balance_temperature_auto is true
• exposure_auto [int32_t] [default=]: Exposure time and/or iris aperture. 0: Automatic exposure time, automatic iris aperture. 1: Manual exposure time, manual iris. 2: Manual exposure time, auto iris. 3: Auto exposure time, manual iris.
• exposure_absolute [int32_t] [default=]: Determines the exposure time of the camera sensor. The exposure time is limited by the frame interval. Drivers should interpret the values as 100 mus units, where the value 1 stands for 1/10000th of a second, 10000 for 1 second and 100000 for 10 seconds.
• use_cuda_color_conversion [bool] [default=true]: Whether to convert from yuyv to RGB using CUDA, otherwise the CPU is for the conversion.

isaac.Vicon

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• vicon_pose_tree [PoseTreeProto]: Pose tree message containing information from Vicon scene volume
• vicon_markers [MarkerListProto]: Marker list message containing all markers visible in Vicon scene volume

Parameters

• vicon_hostname [string] [default=”localhost”]: Hostname of the Vicon system
• vicon_port [string] [default=”801”]: Port to which the Vicon data is streaming
• reconnect_interval [double] [default=1.0]: Amount of time to wait before attempting to reconnect to the Vicon system

isaac.ZedCamera

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• left_camera_rgb [ImageProto]: Left color image
• right_camera_rgb [ImageProto]: Right color image
• left_camera_gray [ImageProto]: Left gray image
• right_camera_gray [ImageProto]: Right gray image
• extrinsics [Pose3dProto]: Camera pair extrinsics (right-to-left)
• left_intrinsics [CameraIntrinsicsProto]: Intrinsics including pinhole and distortion parameters for the left camera
• right_intrinsics [CameraIntrinsicsProto]: Intrinsics including pinhole and distortion parameters for the right camera

Parameters

• auto_exposure [bool] [default=true]: Automatic exposure control
• auto_white_balance [bool] [default=true]: Automatic white balance control
• brightness [int] [default=4]: Brightness level. Valid values are between 0 and 8.
• resolution [sl::RESOLUTION] [default=sl::RESOLUTION::VGA]: The resolution to use for the ZED camera. The following values can be set:

• camera_fps [int] [default=60]: The image frame rate for the ZED camera. If set to 0, the highest FPS for the specified resolution will be used. The following are supported resolution/framerate combinations:
  • RESOLUTION_HD2K (2208*1242): 15 fps
  • RESOLUTION_HD1080 (1920*1080): 15, 30 fps
  • RESOLUTION_HD720 (1280*720): 15, 30, 60 fps
  • RESOLUTION_VGA (672*376): 15, 30, 60, 100 fps

• color_temperature [int] [default=2800]: The color temperature control. Valid values are between 2800 and 6500 with a step of 100.
• contrast [int] [default=4]: Contrast level. Valid values are between 0 and 8.
• exposure [int] [default=50]: Exposure control. Valid values are between 0 and 100. The parameter value is ignored if auto_exposure is enabled. The exposure time is interpolated linearly between 0.17072ms and the max time for a specific frame rate. The following are max times for common framerate:

• gain [int] [default=50]: Gain control. Valid values are between 0 and 100. The parameter value is ignored if auto_exposure is enabled.
• gamma [int] [default=sl::VIDEO_SETTINGS_VALUE_AUTO]: The gamma control. Valid values are between 1 and 9. The special value -1 commands the camera ISP to auto-tune the parameter accordingly to the lighting conditions.
• hue [int] [default=0]: The hue control. Valid values are between 0 and 11.
• saturation [int] [default=4]: The saturation control. Valid values are between 0 and 8.
• sharpness [int] [default=4]: The digital sharpening control. Valid values are between 0 and 8.
• device_id [int] [default=0]: The numeral of the system video device of the ZED camera. For example for /dev/video0 choose 0.
• enable_imu [bool] [default=false]: Turns on capture and publication of IMU data if camera supports it.
• sensors_required [bool] [default=false]: Force the motion sensors to open if the camera supports them. If set to true, the camera will fail to open if the sensors cannot be opened.
• settings_folder_path [string] [default=”./”]: The folder path to the settings file (SN#####.conf) for the zed camera. This file contains the calibration parameters for the camera.
• gpu_id [int] [default=0]: The GPU device to be used for ZED CUDA operations
• gray_scale [bool] [default=false]: Turns on gray scale images
• rgb [bool] [default=true]: Turns on RGB color images
• enable_factory_rectification [bool] [default=true]: Turns on rectification of images inside ZED camera
• lhs_camera_frame [string] [default=”zed_left_camera”]: The left camera frame used to define the left_T_right camera transform in the PoseTree
• rhs_camera_frame [string] [default=”zed_right_camera”]: The right camera frame used to define the left_T_right camera transform in the PoseTree

isaac.atlas.Pose2Comparer

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• first_lhs_frame [string] [default=]: Name of the left hand side frame of the first pose
• first_rhs_frame [string] [default=]: Name of the right hand side frame of the first pose
• second_lhs_frame [string] [default=]: Name of the left hand side frame of the second pose
• second_rhs_frame [string] [default=]: Name of the right hand side frame of the second pose
• threshold [Vector2d] [default=]: This codelet reports success if this parameter is set and the relative difference between the two poses is less than this threshold in position and angle.

isaac.atlas.PoseFromFile

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• lhs_frame [string] [default=]: Name of the reference frame of the left side of the pose
• rhs_frame [string] [default=]: Name of the reference frame of the right side of the pose
• filename [string] [default=]: Path to the file that contains the pose json. Expected layout:

isaac.atlas.PoseMessageInjector

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• pose [PoseTreeEdgeProto]: Incoming pose messages to inject into the pose tree

Outgoing messages

(none)

Parameters

• frame_name_map [FrameNameMap] [default={}]: Map from frame names received to frame names to write. For example,

isaac.atlas.PoseToFile

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• lhs_frame [string] [default=]: Name of the reference frame of the left side of the pose
• rhs_frame [string] [default=]: Name of the reference frame of the right side of the pose
• filename [string] [default=]: Path for the file to create with pose as json

isaac.atlas.PoseToMessage

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• pose [PoseTreeEdgeProto]: Outgoing pose message from pose tree

Parameters

• lhs_frame [string] [default=]: Name of the reference frame of the left side of the pose
• rhs_frame [string] [default=]: Name of the reference frame of the right side of the pose

isaac.atlas.PoseTreeRelink

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• current_lhs_frame [string] [default=]: Name of the current reference frame of the left side of the pose
• desired_lhs_frame [string] [default=]: Name of the desired reference frame of the left side of the pose
• rhs_frame [string] [default=]: Name of the reference frame of the right side of the pose

isaac.atlas.PoseTreeRemoveLink

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• lhs_frame [string] [default=]: Name of the reference frame of the left side of the pose
• rhs_frame [string] [default=]: Name of the reference frame of the right side of the pose

isaac.atlas.PublishMapRoi

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• map_lattice [LatticeProto]: Lattice defining the structure of the extracted evidence grid map
• world_T_robot_gt [PoseTreeEdgeProto]: Express the relative pose of the region of interest with respect to the map

Outgoing messages

• roi [ImageProto]: The region of interest extracted from the binary map

Parameters

• map_node_name [string] [default=”map”]: Name of the map node that contains the occupancy map
• robot_gt_frame [string] [default=”robot_gt”]: Name of the robot frame

isaac.behavior_tree.WaitUntilMessage

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• channel_name [string] [default=”message”]: Name of input channel

isaac.cask.Recorder

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• base_directory [string] [default=”/tmp/isaac”]: The base directory used as part of the log directory (see class comment)
• batch_accumulation_duration [double] [default=0.1]: Maximum duration to wait before writing out messages, in seconds
• tag [string] [default=”“]: A tag used as part of the log directory (see class comment)
• use_compression [bool] [default=false]: If enabled compression is used when writing to the log file
• max_buffer_size_mb [int] [default=4096]: If messages can not be written out fast enough they are kept in memory. If more than a certain amount of memory is used no new messages are accepted until all messages are written out to the disk.
• max_bytes_written_per_tick_mb [double] [default=100]: Maximum total size of messages written per tick.
• min_bytes_per_tick_per_channel_mb [double] [default=0.2]: Minimum total size of messages written per tick and channel
• start_recording_automatically [bool] [default=true]: If true, the recording will automatically start when this Codelet starts.

isaac.cask.Replay

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• cask_directory [string] [default=”“]: The cask directory used to replay data from
• replay_time_offset [int64_t] [default=0]: Time offset to start a replay from between a log
• use_recorded_message_time [bool] [default=false]: Decides whether to use recorded message pubtime and acqtime or replay current time as pubtime and synchronize the acqtime using the starting time of the replay.
• loop [bool] [default=false]: If this is enabled replay will start from the beginning when it was replayed
• auto_stop_application [bool] [default=false]: If enabled automatically stops the application when the log was fully replayed
• report_success [bool] [default=false]: If enabled sets node status to: success the log was fully replayed, failure in case or errors

isaac.cask.ReplayBridge

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• request [nlohmann::json]: Request to replay node

Outgoing messages

• reply [nlohmann::json]: Reply from replay node

Parameters

• replay_component_name [string] [default=]: Replay component name in format node/component. Ex: replay/isaac.alice.Replay
• websight_replay_channel [string] [default=”websight/WebsightServer/replay”]: The full WebSight channel name to take record request from
• websight_replay_reply_channel [string] [default=”websight/WebsightServer/replay_reply”]: The full WebSight channel name to send record reply to

isaac.composite.CompositeAtlas

Description

Type: Component - This component does not tick and only provides certain helper functions.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• cask [string] [default=]: Cask filename which contains composite protos.

isaac.composite.CompositeMetric

Description

Type: Component - This component does not tick and only provides certain helper functions.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• use_config_schema [bool] [default=false]: If true, the schema from the config below is used to compute distance. Otherwise, caller should set the schema
• schema [Schema] [default=]: The schema to compute distance for. See json_formatter.hpp for json representation of Schema. This is use only when use_config_schema is to True.
• norms [VectorXd] [default=VectorXd::Constant(1, 2.0)]: The list of p-norms to compute distance for each quantity in schema. The size should match the list of quantities in schema, or one in which case it’s applied to all quantities.
• weights [VectorXd] [default=VectorXd::Constant(1, 1.0)]: Optional param to multiply with entity distance to get the total distance.

isaac.composite.CompositePublisher

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• path [CompositeProto]: Publishes all the waypoints in path as a batch. A new message is published only when the path changes.

Parameters

• atlas [string] [default=]: Name of the node containing the CompositeAtlas component.
• path [std::vector<std::string>] [default={}]: The list of waypoint names as the path to follow. Reports failure if any waypoint on the path does not exist in CompositeAtlas. A new message is sent if this config changes.
• use_config_schema [bool] [default=false]: If true, use the schema set in config below. Otherwise use schema of the first waypoint in the path read from cask as the schema for the whole path, and ignore the schema set in config.
• schema [Schema] [default=]: Sets the schema to publish. This must be a subset of schemas in the path waypoints. Only used if use_waypoint_schema is set to false.
• report_success [bool] [default=false]: If true, report success after publishing a valid path.

isaac.composite.FollowPath

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• path [CompositeProto]: Receives the path to follow given as a batch of waypoints
• state [CompositeProto]: Receives current state, used to check if we arrive at the current waypoint.

Outgoing messages

• goal [CompositeProto]: Publishes the desired goal waypoint.

Parameters

• wait_time [double] [default=1.0]: Seconds to wait after arriving at the current waypoint before publishing the next waypoint.
• loop [bool] [default=false]: If set to true we will repeat the path once completed. Otherwise will report success on completion.
• tolerance [double] [default=0.1]: Tolerance for arrival. A waypoint is reached when the distance between state and current waypoint computed by the CompositeMetric component is below this limit.

isaac.controller.AckermannControl

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• plan [CompositeProto]: Receives a plan that contains a trajectory of the vehicle. It contains positions as well as the derivative of the vehicle and the angle of the tires.
• odometry [Odometry2Proto]: Receives the estimated current speed, used to stop vehicle when no plan is received

Outgoing messages

• command [CompositeProto]: Publishes the control to be applied in order to follow the trajectory

Parameters

• command_delay [double] [default=0.01]: A look-ahead time (in seconds) to get the current command from the received plan. The expected position or speed at time = current_time + command_delay from the received plan is published as the command
• force_stop [bool] [default=false]: Parameter to force the robot to stop.
• stop_time [double] [default=0.05]: Time used to stop the vehicle if there is not available plan.
• maximum_acceleration [double] [default=2.0]: Maximum acceleration used when stopping the vehicle
• use_pid_controller [bool] [default=false]: Whether or not we should use a pid controller to track the trajectory.
• speed_proportional_gain [double] [default=2.0]: Gain of the proportional error of the speed pid controller.
• speed_differential_gain [double] [default=0.5]: Gain of the differential error of the speed pid controller.
• speed_integral_gain [double] [default=0.25]: Gain of the integral error of the speed pid controller.
• maximum_speed_integral [double] [default=1.0]: Maximum value allowed for the integral error of the speed pid controller.
• lateral_proportional_gain [double] [default=1.0]: Gain of the proportional error of the lateral pid controller.
• lateral_differential_gain [double] [default=0.15]: Gain of the differential error of the lateral pid controller.
• lateral_integral_gain [double] [default=0.25]: Gain of the integral error of the lateral pid controller.
• maximum_lateral_integral [double] [default=1.0]: Maximum value allowed for the integral error of the lateral pid controller.

isaac.controller.DifferentialBaseControl

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• plan [DifferentialTrajectoryPlanProto]: Input: the plan to follow (contain a list of state at a given timestamp)

Outgoing messages

• cmd [StateProto]: Output a navigation::DifferentialBaseControl state message.

Parameters

• cmd_delay [double] [default=0.2]: Expected delay between the command sent and the execution (in second)
• use_pid_controller [bool] [default=true]: Whether or not use the pid controller
• manual_mode_channel [string] [default=”“]: Channel publishing whether or not the robot is in manual mode
• pid_heading [Vector7d] [default=Vector7d((double[]){1.0, 0.1, 0.0, 0.25, -0.25, 1.0, -1.0})]: Parameters of pid controller that controls the heading error
• pid_pos_y [Vector7d] [default=Vector7d((double[]){1.0, 0.1, 0.0, 0.25, -0.25, 2.0, -2.0})]: Parameters of pid controller that controls the lateral error
• pid_pos_x [Vector7d] [default=Vector7d((double[]){1.0, 0.1, 0.0, 0.25, -0.25, 2.0, -2.0})]: Parameters of pid controller that controls the forward error
• controller_epsilon_gain [double] [default=1.0]: Gains used to compute the forward gain
• controller_b_gain [double] [default=1.0]: Gains used to compute the heading gain
• robot_frame [string] [default=”robot”]: Name of robot’s frame. Output command message will be published in this frame.
• static_frame [string] [default=”odom”]: Name of a static frame. Used to convert input plan message to robot frame.
• use_tick_time [bool] [default=true]: This flag controls whether or not this task uses tick time or the acquisition time to know which command to output. Note: Acquisition time should be used when the DifferentialTrajectoryPlanProto comes with a not synchronized source. cmd_delay should be used to estimate the full delay from when the odometry was computed to when the command is going to be executed on the system.

isaac.controller.HolonomicBaseControl

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• plan [DifferentialTrajectoryPlanProto]: Plan (position/time) the controller is trying to follow. TODO: Should not take a DifferentialTrajectoryPlanProto

Outgoing messages

• cmd [StateProto]: Output a navigation::DifferentialBaseControl state message.

Parameters

• cmd_delay [double] [default=0.2]: Expected delay between the command sent and the execution (in second)
• use_pid_controller [bool] [default=true]: Whether or not use the pid controller
• manual_mode_channel [string] [default=”“]: Channel publishing whether or not the robot is in manual mode
• pid_heading [Vector7d] [default=Vector7d((double[]){1.0, 0.1, 0.0, 0.25, -0.25, 1.0, -1.0})]: Parameters of pid controller that controls the heading error
• pid_pos_y [Vector7d] [default=Vector7d((double[]){1.0, 0.1, 0.0, 0.25, -0.25, 2.0, -2.0})]: Parameters of pid controller that controls the lateral error
• pid_pos_x [Vector7d] [default=Vector7d((double[]){0.2, 0.05, 0.0, 0.1, -0.1, 2.0, -2.0})]: Parameters of pid controller that controls the forward error
• use_tick_time [bool] [default=true]: This flag controls whether or not this task uses tick time or the acquisition time to know which command to output. Note: Acquisition time should be used when the DifferentialTrajectoryPlanProto comes with a not synchronized source. cmd_delay should be used to estimate the full delay from when the odometry was computed to when the command is going to be executed on the system.

isaac.controller.MultiJointController

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• plan [CompositeProto]: Receives a plan that contains a trajectory of joint positions or/and speeds. If control_mode is position, the trajectory must contain joint positions. If control_mode is speed, the trajectory must contain joint speeds
• state [CompositeProto]: Receives current state of the joints. In position control, this is sent back to the robot in case of emergency stop, so that the robot doesn’t move

Outgoing messages

• command [CompositeProto]: Publishes the current joint position or speed command

Parameters

• kinematic_tree [string] [default=]: Node name containing the map:KinematicTree component. This is used to get the names of active joints to parser/serialize composite proto.
• control_mode [ControlMode] [default=ControlMode::kPosition]: Use “position” or “speed” to specify whether to send joint position and speed command
• command_delay [double] [default=0.05]: A look-ahead time (in seconds) to get the current command from the received plan. The expected position or speed at time = current_time + command_delay from the received plan is published as the command

isaac.deepstream.Pipeline

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• pipeline [string] [default=”videotestsrc ! video/x-raw]: The Deepstream/GStreamer media pipelines. Note the usage of an appsink element for the entry points to Isaac. Equivalently, the appsrc for exit. The name of the element becomes the channel name. For example: appsrc name=<RX CHANNEL NAME>. For pipeline syntax, please read the command manual for gst-launch-1.0 or read this page: https://gstreamer.freedesktop.org/documentation/tools/gst-launch.html For supported capabilities, formats, memory models, and equivalent Isaac messages, please refer to the component detailed documentation.

isaac.detect_net.DetectNetDecoder

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• bounding_boxes_tensor [TensorProto]: Tensor from DetectNetv2 inference. Contains bounding boxes per class per grid box and is of size (4*N, R, C), where N = number of classes, R = grid rows, C = grid cols.
• confidence_tensor [TensorProto]: Tensor from DetectNetv2 inference. Contains confidence values per object per grid box and is of size (N, R, C), where N = number of classes, R = grid rows, C = grid cols.

Outgoing messages

• detections [Detections2Proto]: Output detections with bounding box, label, and confidence

Parameters

• non_maximum_suppression_threshold [double] [default=0.6]: Non-maximum suppression threshold. The greater this value is, the stricter the algorithm is when determining if two bounding boxes are detecting the same object.
• confidence_threshold [double] [default=0.6]: Confidence threshold of the detection. Decreasing this value allows less confident detections be considered.
• min_bbox_area [int] [default=100]: Bounding box area threshold of the detection. Decreasing this value allows for smaller bounding box detections
• labels [std::vector<std::string>] [default=]: Names of the classes trained by the network. The order and length of this list must correspond to the order and length of the labels given during training.
• output_scale [Vector2d] [default=]: Output scale in [rows, cols] for the decoded bounding boxes output. For example, this could be the image resolution before downscaling to fit the network input tensor resolution.
• bounding_box_scale [double] [default=35.0]: Bounding box normalization for both X and Y dimensions. This value is set in the DetectNetv2 training specification.
• bounding_box_offset [double] [default=0.5]: Bounding box offset for both X and Y dimensions. This value is set in the DetectNetv2 training specification.

isaac.dummy.DummyPose2dConsumer

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• pose [Pose2dProto]:

Outgoing messages

(none)

Parameters

• threshold [int] [default=5]:

isaac.dummy.DummyPose2dProducer

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• pose [Pose2dProto]:

Parameters

(none)

isaac.dummy.DummyStatus

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• status [int] [default=0]:

isaac.dynamixel.DynamixelDriver

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• command [StateProto]: The desired angular speeds for each motor

Outgoing messages

• state [StateProto]: The measured angular speeds for each motor

Parameters

• port [string] [default=”/dev/ttyUSB0”]: USB port where Dynamixel controller is located at. usb_port varies depending on the controller device, e.g., “/dev/ttyACM0” or “/dev/ttyUSB0”
• baudrate [Baudrate] [default=Baudrate::k1M]: Baud rate of the Dynamixel bus. This is the rate of information transfer.
• servo_model [Model] [default=Model::MX12W]: Model of servo (AX12A, XM430, MX12W, XC430)
• control_mode [DynamixelMode] [default=DynamixelMode::kVelocity]: If set to true dynamixels are controlled in speed mode, otherwise they are controlled in position mode
• servo_ids [std::vector<int>] [default=]: Unique identifier for Dynamixel servos. Each motor needs to be assigned a unique ID using the software provided by Dynamixel. This is a mandatory parameter.
• torque_limit [double] [default=1.0]: Servo maximum torque limit. Caps the amount of torque the servo will apply. 0.0 is no torque, 1.0 is max available torque
• max_speed [double] [default=6.0]: Maximum (absolute) angular speed for wheels
• command_timeout [double] [default=0.3]: Commands received that are older than command_timeout seconds will be ignored. Kaya will stop if no message is received for command_timeout seconds.
• debug_mode [bool] [default=false]: Enables debug mode in which all motors are driving with constant speed independent from incoming messages.
• debug_speed [double] [default=1.0]: If debug mode is enabled, all motors will rotate with this speed.

isaac.egm_fusion.BinaryToEvidenceMap

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• binary_map [ImageProto]: Incoming binary map which will be converted to an evidence map

Outgoing messages

• evidence_grid [ImageProto]: Computed evidence grid map as an Image3f

Parameters

(none)

isaac.egm_fusion.EvidenceGridMapViewer

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• egm_lattice [LatticeProto]: Lattice information for the evidence grid map to be visualized
• egm [ImageProto]: Evidence grid map to be visualized

Outgoing messages

(none)

Parameters

• uncertainity_visualization_color [Pixel3ub] [default=Pixel3ub(0, 133, 100)]: RGB color for the uncertain region in an evidence grid. By default, uncertain regions are visualized with dark green color

isaac.egm_fusion.EvidenceMapFusion

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• local_map_lattice [LatticeProto]: The lattice grid information associated to the output local evidence map. It contains the information about the cell size as well as the reference frame of the local egm.
• fused_map_lattice [LatticeProto]: The lattice grid information associated to the larger fused evidence map. It contains the information about the cell size as well as the reference frame of the fused evidence map.

Outgoing messages

• evidential_local_map [ImageProto]: The latest evidence map representing the environment.

Parameters

• fusion_operator [FusionOperator] [default=FusionOperator::kPCR6]: Fusion rule to fuse evidence masses
• num_observation_channels [size_t] [default=1]: Number of channels connected to this codelet in the application graph
• odom_frame [string] [default=”odom”]: The name of the coordinate frame used to continuously update the fused map.
• base_frame [string] [default=”robot”]: The name of the coordinate frame from which local map and fused map frames are computed
• observation_channel_prefix [string] [default=”sensor_”]: Prefix for the channels that will accept evidence grid maps from the sensors Indexing for the lattice channels begin from 1
• lattice_channel_prefix [string] [default=”lattice_”]: Prefix for the channels that will accept lattices for the grid maps from the sensors Indexing for the lattice channels begin from 1
• use_gpu [bool] [default=true]: If cuda enabled kernels need to be used for fusion and extraction of local map Currently only PCR6 fusion operation is accelerated with CUDA

isaac.egm_fusion.EvidenceMapInpaint

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• egm_lattice [LatticeProto]: An evidence grid map lattice information
• original_egm [ImageProto]: An evidence grid map which needs to be painted

Outgoing messages

• painted_egm [ImageProto]: A painted evidence grid map

Parameters

• rectangles_frame [string] [default=]: The coordinate frame of the rectangular region to paint
• paint_mass [Vector3f] [default=Vector3f(1.0f, 0.0f, 0.0f)]: The mass values that will be used to paint every cell within the rectangular region
• rectangles [std::vector<geometry::RectangleD>] [default=]: Set of rectangular regions to paint in the evidence grid, each one with the format [[x_min,x_max], [y_min,y_max]] in meters

isaac.egm_fusion.EvidenceMapOverlay

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• map_1 [ImageProto]: Evidence grid map to overlay with map_2
• map_2 [ImageProto]: Evidence grid map to overlay with map_1

Outgoing messages

• combined_egm [ImageProto]: Evidence grid obtained after combining map_1 and map_2 respecting obstacle occlusion semantics

Parameters

(none)

isaac.egm_fusion.EvidenceToBinaryMap

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• evidence_grid [ImageProto]: Incoming evidence map which will be converted to binary map

Outgoing messages

• binary_map [ImageProto]: Computed binary map (Image1ub)

Parameters

• free_threshold [float] [default=0.1]: Minimum difference between the free and occupied mass of a cell for it to be labelled free
• occupied_threshold [float] [default=0.1]: Minimum difference between the occupied and free mass of a cell for it to be labelled occupied
• is_optimistic [bool] [default=false]: If enabled, uncertain cells will be treated as free, otherwise they are considered to be blocked.

isaac.egm_fusion.RangeScanToEvidenceMap

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• flatscan [FlatscanProto]: The observation_map is created based on flat range scans.
• evidence_map_lattice [LatticeProto]: The lattice information of the Evidence Map

Outgoing messages

• evidence_map [ImageProto]: Image3f containing the free, occupied and uncertain masses

Parameters

• wall_thickness [double] [default=0.20]: When integrating a flatscan an area of the given thickness behind a hit is marked as solid. This value should be at least in the order of the chosen cell size.
• sensor_frame [string] [default=”lidar”]: The name of the reference frame in which range scans arriving on the flatscan channel are defined.

isaac.evaluators.AprilTagsEvaluator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• color [ImageProto]: The camera image which should be of type Image3ub and in color. This is used only to save a sample PNG image to the sensor certification test logs when running this test.
• fiducials [FiducialListProto]: The list of april tag fiducials the test will verify are valid, meaning the fiducials have at least 4 points (corners), and that based on those points the fiducials lie within an ROI.

Outgoing messages

(none)

Parameters

• roi_ratio [double] [default=0.75]: The ratio between the dimensions of the ROI to the resolution of the image provided by the camera under test.
• test_timeout [int] [default=]: The amount of time (seconds) the test is set to be ran for.
• num_frames [int] [default=]: The number of FiducialListProto messages to be captured.

isaac.evaluators.ColorEvaluator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• color_listener [ImageProto]: Color images to validate

Outgoing messages

• cv_image [ImageProto]: CV Masked image

Parameters

• frame_rate [double] [default=]: The frame rate of the color channel
• image_height [int] [default=]: Image dimensions
• image_width [int] [default=]:
• target_color [int] [default=0]: Expected lego color of target area: 0-red, 1-green, 2-blue
• target_width [int] [default=10]: Width of target area in pixels
• target_height [int] [default=10]: Height of target area in pixels
• target_cmd [int] [default=0]: Direction to move target: 0-none, 1-up, 2-down, 3-left, 4-right, 5-done

isaac.evaluators.ColorSanityEvaluator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• color_listener [ImageProto]: Color images to validate
• color_intrinsics_listener [CameraIntrinsicsProto]:

Outgoing messages

(none)

Parameters

• rows [int] [default=]: Expected number of rows
• cols [int] [default=]: Expected number of columns
• channels [int] [default=]: Expected number of channels
• focal_x [float] [default=]: Expected focal length
• focal_y [float] [default=]:
• num_frames [int] [default=]: Number of frames to test
• colorspace [string] [default=]: Colorspace

isaac.evaluators.DepthEvaluator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• depth_listener [ImageProto]: Depth messages to evaluate
• color_listener [ImageProto]: After setup is complete, the first color message received on this channel is saved as a png to record the scene setup

Outgoing messages

(none)

Parameters

• frame_rate [double] [default=]: The frame rate of the depth channel, used to determine how many frames to evaluate
• image_height [int] [default=]: Image dimensions
• image_width [int] [default=]:
• target_depth [double] [default=0.5]: Target depth value in meters
• target_width [int] [default=10]: Width of target area in pixels
• target_height [int] [default=10]: Height of target area in pixels
• mean_tolerance [double] [default=1.0]: Tolerance for average depth (% error)
• cov_tolerance [double] [default=0.05]: Tolerance for coefficient of variation (standard deviation / mean)
• target_cmd [int] [default=0]: Direction to move target: 0-none, 1-up, 2-down, 3-left, 4-right, 5-done

isaac.evaluators.DepthSanityEvaluator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• depth_listener [ImageProto]: Depth images to validate
• depth_intrinsics_listener [CameraIntrinsicsProto]:

Outgoing messages

(none)

Parameters

• rows [int] [default=]: Expected number of rows
• cols [int] [default=]: Expected number of columns
• focal_x [float] [default=]: Expected focal length
• focal_y [float] [default=]:
• num_frames [int] [default=]: Number of frames to test

isaac.evaluators.FrameRateEvaluator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• color_listener [ColorCameraProto]: Color channel to listen to for framerate
• depth_listener [DepthCameraProto]: Depth channel to listen to for framerate

Outgoing messages

(none)

Parameters

• target_fps [double] [default=30.0]: Target framerate
• fps_tolerance [double] [default=1.0]: Tolerance for framerate (% error)
• test_depth [bool] [default=false]: Whether to test depth frame rate, otherwise, test the color framerate
• test_duration [double] [default=5.0]: Length of time (seconds) to run framerate test

isaac.fiducials.AprilTags3Detection

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• image [ImageProto]: RGB input image. Image should be undistorted prior to being passed in here.
• intrinsics [CameraIntrinsicsProto]: Intrinsics including pinhole and distortion parameters

Outgoing messages

• april_tags [FiducialListProto]: Output, List of AprilTag fiducials

Parameters

• tag_dimensions [double] [default=0.18]: Tag dimensions, translation of tags will be calculated in same unit of measure
• tag_family [string] [default=”tag36h11”]: Tag family, currently ONLY tag36h11 is supported

isaac.fiducials.AprilTagsDetection

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• image [ImageProto]: RGB input image. Image should be undistorted prior to being passed in here.
• intrinsics [CameraIntrinsicsProto]: Intrinsics including pinhole and distortion parameters

Outgoing messages

• april_tags [FiducialListProto]: Output, List of AprilTag fiducials

Parameters

• max_tags [int] [default=50]: Maximum number of AprilTags that can be detected
• tag_dimensions [double] [default=0.18]: Tag dimensions, translation of tags will be calculated in same unit of measure
• tag_family [string] [default=”tag36h11”]: Tag family, currently ONLY tag36h11 is supported

isaac.fiducials.FiducialAsGoal

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• fiducials [FiducialListProto]: The input channel where fiducial detections are publishes

Outgoing messages

• goal [Goal2Proto]: The target fiducial as a goal
• plan [Plan2Proto]: The target fiducial as a simple plan with one waypoint

Parameters

• target_fiducial_id [string] [default=”tag36h11_9”]: The ID of the target fiducial
• give_up_duration [double] [default=1.0]: If the robot does not see the fiducial for this time period the robot is stopped
• mode [Mode] [default=Mode::kCenter]: Specifies how the robot will use the fiducial to compute its goal location.
• max_goal_tag_distance [double] [default=1.0]: The maximum distance the goal with be away from the tag
• robot_frame [string] [default=]: The name of the robot coordinate frame
• camera_frame [string] [default=]: The name of the camera coordinate frame

isaac.flatscan_localization.GradientDescentLocalization

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• flatscan [FlatscanProto]: Range scan used to localize the robot

Outgoing messages

(none)

Parameters

• map [string] [default=”map”]: Reference frame in which the algorithm tracks the target.
• flatscan_frame [string] [default=”lidar”]: Flat range scans arriving on the flatscan channel are relative to this coordinate frame.
• robot_init_frame [string] [default=”robot_init_global_localizer”]: In the start function the algorithm is initialized with the pose of this coordinate frame.
• robot_frame [string] [default=”robot”]: The frame which is tracked by this algorithm.

isaac.flatscan_localization.GridSearchLocalizer

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• exclude_restricted_areas [bool] [default=true]: If false, robot may localize inside a restricted area defined in the map configuration
• robot_radius [double] [default=0.25]: The radius of the robot. This parameter is used to exclude poses which are too close to an obstacle.
• max_beam_error [double] [default=0.50]: The maximum beam error used when comparing range scans.
• num_beams_gpu [int] [default=256]: The GPU accelerated scan-and-match function can only handle a certain number of beams per range scan. The allowed values are {32, 64, 128, 256, 512}. If the number of beams in the range scan does not match this number a subset of beams will be taken.
• batch_size [int] [default=512]: This is the number of scans to collect into a batch for the GPU kernel. Choose a value which matches your GPU well.
• sample_distance [float] [default=0.1]: Distance between sample points in meters. The smaller this number, the more sample poses will be considered. This leads to a higher accuracy and lower performance.
• map [string] [default=”map”]: Name of map node to use for localization
• flatscan_frames [std::vector<std::string>] [default={“lidar”}]: The names of the reference frames for the flatscan sources in which range scans arriving on the flatscan channels are defined. The order of the frame names corresponds to the numbering of the expected incoming flatscan channels flatscan_<n>. While n counts from 1, the channel flatscan_1 is just referred to as flatscan.
• flatscan_channel_names [std::vector<std::string>] [default={“flatscan”}]: The names of the input channels used to receive flatscan messages. Each flatscan channel is associated to the frame name in flatscan_frames at the same position. Both lists need to have the same amount of entries. Channel names may not be duplicated in this list.
• output_frame [string] [default=”robot_init_global_localizer”]: The estimated pose of the robot will be written to PoseTree as world_T_output_frame
• patch_size [Vector2i] [default=Vector2i(512, 512)]: The size in gridmap rows/columns each patch used for tracing should contain.
• robot_frame [string] [default=”robot”]: Name of the frame of the robot

isaac.flatscan_localization.MonitorEvaluatorFrequency

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• flatscan [FlatscanProto]: Incoming flatscan messages to calculate a message frequency from

Outgoing messages

• grade [CompositeProto]: Outgoing grade to send to a Monitor instance

Parameters

• threshold [double] [default=5.0]: Measure defining 1.0 deviation of the actual from the expected frequency, in Hz. This means that for each threshold the actual frequency differs from the expected_frequency, the resulting evaluator grade increases (frequency smaller than expected) or decreases (frequency higher than expected) by 1.0. Fractions apply.
• expected_frequency [double] [default=10.0]: The expected frequency with which flatscan messages are received, in Hz

isaac.flatscan_localization.MonitorEvaluatorMapscan

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• flatscan [FlatscanProto]: Incoming flatscan messages used to monitor the current robot pose

Outgoing messages

• grade [CompositeProto]: Outgoing grade to send to a Monitor instance

Parameters

• map [string] [default=”map”]: Name of the map node which contains the reference map
• range_scan_model [string] [default=”shared_robot_model”]: Name of the node which contains the RangeScanModel component used to compare flatscans
• flatscan_frame [string] [default=”lidar”]: Name of the coordinate frame of the sensor which produces the flatscan messages
• robot_frame [string] [default=”robot”]: Name of the coordinate frame of the robot base
• beam_distance_threshold [double] [default=0.2]: Beams where measured distance and expected distance are within this tolerance are considered “good” beams.
• good_beams_threshold [double] [default=0.75]: Measure defining what percentage of good beams is a grade deviation of 1.0

isaac.flatscan_localization.MonitorEvaluatorSvio

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• grade [CompositeProto]: Outgoing grade to send to a Monitor instance

Parameters

• ref1_frame [string] [default=]: Name of the reference frame of the first input pose
• pose1_frame [string] [default=]: Name of the target frame of the first input pose
• ref2_frame [string] [default=]: Name of the reference frame of the second input pose
• pose2_frame [string] [default=]: Name of the target frame of the second input pose
• delta_time [double] [default=]: Optional delta time that defines the time interval between poses to subtract; the codelet’s start time is used if this parameter is not set instead
• threshold [Vector2d] [default=]: Maximum relative difference between the two poses tolerated, in position (meters) and angle (radians)

isaac.flatscan_localization.ParticleFilterLocalization

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• flatscan [FlatscanProto]: Incoming range scan used to localize the robot
• flatscan_2 [FlatscanProto]: A second range scan which can be used to localize the robot.

Outgoing messages

• samples [Pose2Samples]: The current weight samples the particle is tracking

Parameters

• num_particles [int] [default=75]: The number of particles used in the particle filter
• absolute_predict_sigma [Vector3d] [default=Vector3d(0.04, 0.04, DegToRad(5.0))]: Standard deviation of Gaussian noise “added” to the estimated pose during the predict

• relative_predict_sigma [Vector3d] [default=Vector3d(0.10, 0.10, 0.10)]: Standard deviation of Gaussian noise which is applied relative to the current speed of

• initial_sigma [Vector3d] [default=Vector3d(0.3, 0.3, DegToRad(20.0))]: Standard deviation of Gaussian noise which is applied to the initial pose estimate when the particle filter is (re-)seeded.
• output_best_percentile [double] [default=0.10]: The final pose estimate is computed using the average of the best particles. For example a value of 0.10 would mean that the top 10% of particles with highest scores are used to compute the final estimate.
• reseed_particles [bool] [default=false]: Set to true to request reseeding particles. This will be reset to false when the particle filter was reseeded.
• map [string] [default=”map”]: Node of the map which contains map data. The map is used to compute which range scan would be expected from a hypothetical robot pose.
• range_scan_model [string] [default=”shared_robot_model”]: Name of the node which contains a component of type RangeScanModel which is then used to compare range scans when evaluating particles against a new incoming message.
• flatscan_frame [string] [default=”lidar”]: Flat range scans arriving on the flatscan channel are relative to this coordinate frame.
• flatscan_2_frame [string] [default=”lidar_2”]: Similar to flatscan_frame but for the second flatscan channel flatscan_2.
• robot_init_frame [string] [default=”robot_init_global_localizer”]: In the start function particles are initialized around this coordinate frame.
• robot_frame [string] [default=”robot”]: The frame which is tracked by the particle filter.

isaac.flatscan_localization.ParticleSwarmLocalization

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• flatscan [FlatscanProto]: The current sensor measurement based on which we try to localize in the map

Outgoing messages

(none)

Parameters

• num_particles [int] [default=1000]: The number of particles used by PSO
• pso_omega [double] [default=0.5]: Omega parameter of PSO
• pso_phi [Vector3d] [default=(Vector3d{0.05, 0.05, 0.1})]: Phi parameter of PSO (values are for for dx, dy, da)
• pso_phi_p_to_g [double] [default=1.0]: PSO parameter to express ratio between phi_p and phi_g
• map [string] [default=”map”]: Map node to use for localization
• output_frame [string] [default=”robot_init_global_localizer”]: The estimated pose of the robot will be written to PoseTree as world_T_output_frame

isaac.flatsim.DifferentialBasePhysics

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• command [CompositeProto]: Actuator commands for the wheels of a differential base

Outgoing messages

• bodies [CompositeProto]: Resulting physics state of the differential base body

Parameters

• robot_model [string] [default=”shared_robot_model”]: Name of the robot model node
• joint_name_left_wheel [string] [default=”left_wheel”]: Name of the joint for left wheel
• joint_name_right_wheel [string] [default=”right_wheel”]: Name of the joint for right wheel
• wheel_acceleration_noise [double] [default=0.03]: Each step a random normal-distributed noise with the given sigma will be added to the desired wheel acceleration. The sigma will be scaled based on the time step and wheel speed.
• wheel_acceleration_noise_decay [double] [default=0.995]: The wheel acceleration noise is additive simulating a random walk. To keep the noise bounded around zero it is multiplied with a decay factor at every timestep.
• slippage_magnitude_range [Vector2d] [default=Vector2d(0.00, 0.05)]: A random friction value is applied which effectively reduces the effect of wheel speed on wheel distance driven. A friction value of 0 zero means full transmission, while a friction value of 1 means full slippage. Slippage is computed randomly using a uniform distribution with the given minimum and maximum value.
• slippage_duration_range [Vector2d] [default=Vector2d(0.50, 1.25)]: The slippage value is maintained constant for a certain duration and then changed to a new value. The duration of the slippage is computed using a uniform distribution with given minimum and maximum value.
• robot_init_pose_name [string] [default=”robot_init_gt”]: Name of the pose in pose tree to use as the initial pose for robot

isaac.flatsim.DifferentialBaseSimulator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• diff_base_command [StateProto]: Input command message with desired body speed (should be of type: DifferentialBaseControl)
• physics_bodies [CompositeProto]: Input state of the base rigid body as computed by physics

Outgoing messages

• physics_actuation [CompositeProto]: Output actuator message with desired accelerations for each wheel
• diff_base_state [StateProto]: Output state of differential base (DifferentialBaseDynamics)

Parameters

• max_wheel_acceleration [double] [default=10.0]: The maximum acceleration for a wheel
• power [double] [default=0.20]: How fast the base will accelerate towards the desired speed
• flip_left_wheel [bool] [default=false]: If this is enabled the direction of the left wheel will be flipped
• flip_right_wheel [bool] [default=false]: If this is enabled the direction of the right wheel will be flipped
• robot_model [string] [default=”shared_robot_model”]: Name of the robot model node
• joint_name_left_wheel [string] [default=”left_wheel”]: Name of the joint for left wheel
• joint_name_right_wheel [string] [default=”right_wheel”]: Name of the joint for right wheel

isaac.flatsim.FlatscanNoiser

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• flatscan [FlatscanProto]: Recive a FlatScan proto: this is a list of beams (angle + distance)

Outgoing messages

• noisy_flatscan [FlatscanProto]: Output a noisy FlatScan proto: this is a list of beams (angle + distance) with

Parameters

• min_range [double] [default=0.25]: The minimum range at which obstacles are detected
• max_range [double] [default=50.0]: The maximum range of the simulated LIDAR
• range_sigma_rel [double] [default=0.001]: Standard deviation of relative range error
• range_sigma_abs [double] [default=0.03]: Standard deviation of absolute range error
• beam_invalid_probability [double] [default=0.05]: Probability that a beam will be simulated as invalid
• beam_random_probability [double] [default=0.00001]: Probability that a beam will return a random range
• beam_short_probability [double] [default=0.03]: Probability that a beam will return a smaller range

isaac.flatsim.PolygonMapGenerator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• polygon_soup [Polygon2SoupProto]: Receive an aggregate message containing coordinates of the polygons that need to be painted
• polygon_lattice [LatticeProto]: Receive the structure defining the polygon map

Outgoing messages

• polygon_map [ImageProto]: Publish a binary occupancy map with all the areas covered by the polygons marked as occupied

Parameters

(none)

isaac.flatsim.SimRangeScan

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• flatscan [FlatscanProto]: Output a FlatScan proto: this is a list of beams (angle + distance)

Parameters

• num_beams [int] [default=360]: The number of beams in the range scan
• min_range [double] [default=0.25]: The minimum range at which obstacles are detected
• max_range [double] [default=50.0]: The maximum range of the simulated LIDAR
• min_angle [double] [default=0.0]: The min angle of simulated beams
• max_angle [double] [default=TwoPi<double>]: The max angle of simulated beams
• map [string] [default=”map”]: Map node to use for tracing range scans
• lidar_frame [string] [default=”lidar_gt”]: Name of the frame of the simulated LiDAR sensor

isaac.flatsim.TraceOccupancyMap

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• occupancy_map [ImageProto]: Incoming occupancy grid map that needs to be raycasted
• occupancy_map_lattice [LatticeProto]: Structure of the occupancy grid map

Outgoing messages

• flatscan [FlatscanProto]: Output a FlatScan proto with a list of beams (angle + distance)

Parameters

• num_beams [int] [default=900]: The number of beams in the range scan
• min_range [double] [default=0.00]: The minimum range at which obstacles are detected
• max_range [double] [default=1000.0]: The maximum range of the simulated LIDAR
• min_angle [double] [default=0.0]: The min angle of simulated beams
• max_angle [double] [default=TwoPi<double>]: The max angle of simulated beams
• lidar_frame [string] [default=”lidar”]: Name of the frame of the simulated LiDAR sensor

isaac.fuzzy.EfllFuzzyEngineExample

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

(none)

isaac.fuzzy.LfllFuzzyEngineExample

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

(none)

isaac.gtc_china.PanTiltGoto

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• current_state [StateProto]: Input containing the current pan-tilt state

Outgoing messages

• target_state [StateProto]: Output state proto containing target pan and tilt angle information

Parameters

• target_pan_angle [double] [default=0.0]: Target pan angle for the camera
• target_tilt_angle [double] [default=0.0]: Target tilt angle for the camera
• tolerance [double] [default=0.05]: Parameter which defines at least how close the current pan and tilt angles have to be to the target angles to be considered as having reached the target. If the absolute differences between the current and target pan and tilt angles are lesser than or equal to this value, we consider that the pan-tilt unit has reached the required angles.

isaac.hgmm.HgmmPointCloudMatching

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• cloud [PointCloudProto]: Takes as input point clouds from sensor like Lidar or Depth Camera

Outgoing messages

(none)

Parameters

• levels [int] [default=2]: The number of levels to build the HGMM tree. This number depends on the complexity of the scene geometry and the number of points in the point clouds. Typically, 2 works well for simple scenes/point clouds, though 3 empirically works better for denser point clouds like velodyne-32 or more. The higher the level, the more accurate the registration, but divergence becomes more probable. (the model overfits and becomes unstable.) 4+ typically reserved for high-fidelity 3d reconstructions, not 6 dof registration
• convergence_threshold [float] [default=0.001]: The lower, the longer the algorithm takes to converge, but performance becomes better. 0.01: fast to converge but worse accuracy 0.001-0.0001: slow to converge but often better accuracy
• max_iterations [int] [default=30]: Max iterations regardless of convergence. Most problems take on the order of 10-35 iterations per level for normal convergence tolerance ranges.
• noise_floor [float] [default=0.000]: TODO Noise parameter (currently turned off). Used if data contains extreme outliers. In the meantime, basic filtering of input needs to be performed outside of HGMM model creation and registration
• zero_x_y_minimal_z [float] [default=1]: Minimal z-coordinate for valid points with zero x-y coordinates. Used to drop invalid points from erroneous source
• regularization [float] [default=0.01]: Regularization to prevent singularities and overfitting If solution is diverging, parameter is too low. 0.0001: highly accurate but often unstable 0.001: highly accurate but possible divergence 0.01: robust convergence but higher error 0.1: very robust but possibly biased result
• axis_length [double] [default=1.0]: Ego frame axis length
• skip [int] [default=51]: Skipping points to reduce overload of visualization
• history_size [int] [default=10]: Keeps past several history point clouds for visualization
• max_distance [double] [default=10.0]: Visualizes no points beyond the distance

isaac.imu.IioBmi160

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• imu_raw [ImuProto]: ImuProto is used to publish IMU data read from the buffer

Parameters

• i2c_device_id [int] [default=1]: I2C device ID: matches ID of /dev/i2c-X
• imu_T_imu [SO3d] [default=SO3d::FromAxisAngle(Vector3d{1, 0, 0}, Pi<double>)]: IMU Mounting Pose In the base case, the IMU is mounted on it’s back. Rotate 180 degrees about X-axis (flip Y and Z axes)

isaac.imu.ImuCalibration2D

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• imu [ImuProto]: Imu Data

Outgoing messages

(none)

Parameters

• imu_calibration_file [string] [default=”imu_calibration.out.json”]: path to output calibration file. This file will be created if it does not exist and overwritten if it exists.
• imu_variance_stationary [double] [default=0.2]: Threshold for stationary variance
• imu_window_length [int] [default=100]: Number of samples in window

isaac.imu.ImuCorrector

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• raw [ImuProto]: Receive raw IMU data

Outgoing messages

• corrected [ImuProto]: Publish corrected IMU data

Parameters

• calibration_file [string] [default=]: Optional calibration file. If a calibration file is provided, biases from the file will be removed from the IMU data. Otherwise we will calibrate in the beginning.
• calibration_variance_stationary [double] [default=0.1]: Stationary variance for calibration
• calibration_window_length [int] [default=100]: Number of samples in window for calibration

isaac.imu.ImuSim

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• bodies [RigidBody3GroupProto]: Input states of the rigid bodies as computed by physics engine

Outgoing messages

• imu_raw [ImuProto]: Imu proto is used to publish raw Imu data received from simulator

Parameters

• imu_name [string] [default=”imu”]: Name of the IMU rigid body This param is required and should match the config file for the sim
• gravity_norm [double] [default=9.80665]: Imu specific parameters Norm of local gravitational constant
• sampling_rate [double] [default=30.0]: Sampling Frequency
• accel_bias [Vector3d] [default=Vector3d::Zero()]: Accelerometer Bias
• accel_noise [Vector3d] [default=Vector3d::Zero()]: Accelerometer (zero mean) noise std dev
• gyro_bias [Vector3d] [default=Vector3d::Zero()]: Gyroscope Bias
• gyro_noise [Vector3d] [default=Vector3d::Zero()]: Gyroscope (zero mean) noise std dev

isaac.isaac_rc.IsaacRCServer

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• color0 [ImageProto]: Up to 4 image streams to a connected client
• color1 [ImageProto]:
• color2 [ImageProto]:
• color3 [ImageProto]:
• detection_listener [Detections2Proto]: 2D bounding box detections
• polyline_listener [Polyline2dProto]: Polylines - this is added an example of another data type being sent

Outgoing messages

• joystick [JoystickStateProto]: joystick controls

Parameters

• width [int32_t] [default=]: Width of the streaming video
• height [int32_t] [default=]: Height of the streaming video
• framerate [int32_t] [default=]: Framerate of the streaming video
• sync_on_detections [bool] [default=false]: Data is sent in one JSON document at a time, each document corresponding to an image. If this param is true, the latest data will be sent out when detections are received. If this param is false, the latest data will be sent out when an image (on the first color channel) is received. When set to true, there will be improved synchronization between detections and images but potentially higher latency. This must be set to false if there is no edge detections will be received on.
• gpu_id [int32_t] [default=0]: GPU to use for IsaacRC component CUDA operations

isaac.json.JsonMockup

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• json [JsonProto]: Outputs a JsonProto message containing the json_mock parameter serialized to text.
• raw_json [nlohmann::json]: Outputs a copy of the json_mock parameter in a RawMessage. Use when interacting with components that do not accept capnp messages, or use with a JsonToProto component to generate capnp messages of any type.

Parameters

• json_mock [json] [default=]: The JSON to publish
• report_success [bool] [default=false]: Reports success after publishing the message, if enabled
• num_successful_publishes [int] [default=1]: The number of times the message is published before the codelet reports success, if enabled.
• raw_type [uint64_t] [default=]: The type id to set for the raw json message. If this parameter is not set, raw json messages are not published

isaac.json.JsonReplay

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• json_proto [JsonProto]: JSON message read from the file

Parameters

• jsonfile_path [string] [default=”/tmp/input.jsonl”]: Path to JSON file

isaac.json.JsonTcpClient

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• host [string] [default=]: The IP address or hostname of the remote host from which messages will be received.
• port [int] [default=]: The TCP port number on which the remote host is publishing messages.
• reconnect_interval [double] [default=0.5]: If a connection to the remote can not be established or breaks we try to re-establish the connection at this interval (in seconds).
• max_buffered_messages [int] [default=32]: The maximum amount of unsent messages to store before dropping the oldest message
• default_channel [string] [default=”json”]: If a message is received from the remote server with no channel specified, the codelet will publish the message on this channel

isaac.json.JsonToProto

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• json [nlohmann::json]: JSON messages received on this channel are converted to proto messages.

Outgoing messages

• proto [MessageHeaderProto]: Publish proto messages in registered proto definition as specified by incoming json message proto id. FIXME(yangl) This is the wrong proto type.

Parameters

(none)

isaac.json.JsonWriter

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• json [JsonProto]: Incoming JSON message to write that is received as a proto
• raw_json [Json]: Incoming JSON message to write that is received as a raw message

Outgoing messages

(none)

Parameters

• filename [string] [default=]: Path to write file. If the file already exists, it will be overwritten by this codelet.
• indent [int] [default=-1]: Sets the indent of nlohmann::basic_json::dump(). Leave as -1 for the compact representation with no newlines. Set to positive for newlines with indent level.
• itemize_top_level_array [bool] [default=false]: If enabled each item in the top-level array will be written as an individual line.

isaac.json.ProtoToJson

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• proto [MessageHeaderProto]: Receives any proto message FIXME(yangl) This is the wrong proto type.

Outgoing messages

• json [nlohmann::json]: Publishes converted Json message

Parameters

(none)

isaac.kaya.KayaBaseDriver

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• base_command [StateProto]: The desired motion of Kaya of type messages::HolonomicBaseControls
• wheel_state [StateProto]: The measured angular speeds for each wheel received from the motor driver. The order of wheels is the same as for the wheel_command channel.

Outgoing messages

• base_state [StateProto]: The state of Kaya of type messages::HolonomicBaseDynamics
• wheel_command [StateProto]: The desired angular speeds for each wheel to be sent to the motor driver. The order of wheels in the message is: front right, front left, back

Parameters

• wheel_base_length [double] [default=0.125]: Distance of the wheel center to the robot center of rotation
• wheel_radius [double] [default=0.04]: The radius of Kaya wheels
• max_linear_speed [double] [default=0.3]: Maximum allowed linear speed
• max_angular_speed [double] [default=0.5]: Maximum allowed angular speed

isaac.kinova_jaco.KinovaJaco

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• arm_command [CompositeProto]: Command for the arm, parsed based on control mode. In joint control mode, requires a scalar quantity for each of the seven joints with entity matching the kinematic tree link names, and measure matching the control mode (position or speed). In end effector control, the entity name matching the end_effector_name set in config. In end effector pose control, expects a position quantity of translation vector(Vector3) and a rotation quantity of quaternion (Vector4), in base coordinate. In end effector velocity control, expects a speed quantity of linear velocity (Vector3) in base coordinate and a angularSpeed quantity of angular speed (Vector3) in end-effector coordinate.
• finger_command [CompositeProto]: Receives command to open or close fingers

Outgoing messages

• arm_state [CompositeProto]: Current state for the arm, includes end effector pose, joint positions and velocities. The schema includes the following: end effector position (Vector3) and rotation quaternion (Vector4), in base coordinate; a scalar quantity for position and for speed for each of the seven joints, entity name is the link name in kinematic tree;
• finger_state [CompositeProto]: Current position of finger

Parameters

• kinova_jaco_sdk_path [string] [default=]: Path to Kinova Jaco SDK for Gen7 7 DoF arm on Ubuntu. Driver is tested for use with Kinova Jaco SDK 1.4.2 and SDK 1.5.1. See SDK resources for details of the Kinova Jaco driver.

• initialize_home [bool] [default=false]: If true, initializes arm and finger to home position in start.
• control_mode [ControlMode] [default=ControlMode::kJointPosition]: Set control mode for arm. This can be changed at runtime
• kinematic_tree [string] [default=]: Name of the node containing the map:KinematicTree component
• end_effector_name [string] [default=”end_effector”]: Name of the end effector for parsing/creating composite message.
• finger_names [std::vector<std::string>] [default=std::vector<std::string>({“finger1”, “finger2”, “finger3”})]: Name of fingers used in composite proto
• finger_use_raw_value [bool] [default=false]: If true, use raw (un-normalized) values received in command and for publishing state for the fingers. For kinova 3-finger hand, these are typically in the range of [0, 7340] for finger position, and [-6000, 6000] for speed. If false, use 0.0/1.0 for open/close fingers, and the raw position state is mapped to binary using the finger_close_threshold param below.
• finger_close_threshold [float] [default=3000.0f]: Threshold for the raw finger position reported by kinova api to map to binary open/close state. Values below this threshold are considered open, above closed. This is only used if param finger_use_raw_value is false.

isaac.laikago.LaikagoDriver

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• base_command [StateProto]: The desired motion of Laikago of type messages::HolonomicBaseControls

Outgoing messages

• base_state [StateProto]: The state of Laikago of type messages::HolonomicBaseDynamics
• imu [ImuProto]: The imu proto has linear and angular acceleration data

Parameters

• speed_limit_linear [double] [default=0.6]: Maximum linear speed robot is allowed to travel with
• speed_limit_angular [double] [default=0.8]: Maximum angular speed robot is allowed to rotate with
• min_command_speed [double] [default=0.01]: Minimum command speed to switch to walking mode
• scale_back_speed [double] [default=2.0]: Scale backward speed. The laikago walks backward slower than forward given the same command value. To approximately compensate the bias, we scale up the command speed
• scale_side_speed [double] [default=3.0]: Scale side walk speed. The laikago side walk speed is about three times slower than command. To approximately compensate the tracking error, we scale up the command speed

isaac.lidar_slam.Cartographer

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• flatscan [FlatscanProto]: Cartographer uses a 2D LIDAR scan to build the map

Outgoing messages

(none)

Parameters

• lua_configuration_directory [string] [default=”“]: Folders to search for Cartographer lua scripts, separated by comma
• lua_configuration_basename [string] [default=”“]: File name of the specific Cartographer lua script to load
• output_path [string] [default=”/tmp”]: Folder to write submaps and generated map
• background_size [Vector2i] [default=Vector2i(1500, 1500)]: The size of the canvas for visualizing the map in sight (in grid cells)
• background_translation [Vector2d] [default=Vector2d(-75, -75)]: Translation to apply on background image (in meters)
• num_visible_submaps [int] [default=8]: Only the most recent submaps are visualized with sight for performance reasons.

isaac.lidar_slam.GMapping

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• flatscan [FlatscanProto]: GMapping uses a 2D LIDAR scan to build the map
• odometry [Odometry2Proto]: Odometry can either be read from this message or from the pose tree. The message is only used if the parameter use_pose_tree is set to false, otherwise odometry is read from the pose tree.

Outgoing messages

(none)

Parameters

• file_path [string] [default=”/tmp”]: Directory path used to save map snapshots
• sensor_frame [string] [default=”lidar”]: The reference frame in which range scans arriving on the flatscan channel are defined. Sensor needs to be fixed with respect to the robot. Pose is read in the beginning only.
• build_map_period [double] [default=2.0]: How often the map is recomputed, in seconds
• laser_matcher_resolution [double] [default=DegToRad(3.0)]: Resolution to be used in scan matcher angles
• map_x_max [double] [default=100.]: Maximum x value of the initial map
• map_y_max [double] [default=100.]: Maximum y value of the initial map
• map_x_min [double] [default=-100.]: Minimum x value of the initial map
• map_y_min [double] [default=-100.]: Minimum y value of the initial map
• map_resolution [double] [default=0.1]: Distance between each pixel in the map
• max_range [double] [default=32.0]: The maximum range of the lidar. This value should be close to the physical range of the lidar to exploit as much of the available information. This value limits the out of range threshold coming from the input flatscan message.
• map_update_range [double] [default=30.0]: The range within which the map is updated. The update range must be smaller or equal to the maximum range parameter as it relies on the lidar range information. The value chosen allows the tradeoff between the map global consistency and its sharpness.
• number_particles [int] [default=40]: Number of particles used to estimate position of the robot
• linear_distance [double] [default=0.3]: Linear threshold used to attempt scan matching
• angular_distance [double] [default=0.1]: Angular threshold used to attempt scan matching
• use_pose_tree [bool] [default=false]: Whether robot pose is read from pose tree or RX channel

isaac.lqr.AckermannLqrPlanner

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• odometry [Odometry2Proto]: Contains the odometry information required for planning (current speed, acceleration, etc.)
• vehicle_state [CompositeProto]: Contains the vehicle steering angle
• global_plan [Plan2Proto]: Contains the target plan the local planner attempts to follow

Outgoing messages

• plan [CompositeProto]: Contains a series of poses to form the trajectory that is optimal to follow

Parameters

• odom_frame [string] [default=”odom”]: Name of the odometry frame the pose information in received state message is relative to
• static_frame [string] [default=”world”]: Name of the static frame used to transform the plan into the odometry frame
• time_between_command [double] [default=0.1]: Step size to be used in integrating the state
• num_controls [int] [default=50]: Number of controls used to generate the path.
• decay [double] [default=0.99]: Other parameters: Decay apply to each steps (decay < 1 means we accord higher importance to the beginning of the path while decay > 1 emphasizes the end of the path).
• wheel_base [double] [default=3.0]: Wheelbase for the bicycle model (distance between front and rear axis in meter)
• state_planner_cost_name [string] [default=]: This is the name of the root PlannerCostBuilder component for the state. The cost of this

• control_planner_cost_name [string] [default=]: This is the name of the root PlannerCostBuilder component for the controls. The cost of this builder is computed on the control vector and is added for each position along the path. The default PlannerCostBuilder is a RangeConstraintsCostBuilder, but you can modify the cost function or add new ones by using a AdditionCostBuilder (see state_planner_cost_name description for more information on how you can customize a cost function).
• final_planner_cost_name [string] [default=]: This is the name of PlannerCostBuilder components. The cost of this builder is computed on the state vector for final position. There is no default one used, but you can add one (see state_planner_cost_name description for more information on how you can customize a cost function).
• path_cost_name [string] [default=]: Name fo the PolylineDistanceQuadraticCostBuilder in state planner cost to set the global path for the plan to follow.
• target_cost_name [string] [default=]: Name fo the BoundedQuadraticCostBuilder in the planner cost to set the target pose.
• stop_robot_on_collision [bool] [default=true]: Force the robot to stop when it’s colliding. This can prevent the robot from moving when it gets too close to an obstacle (a wall for example) if the robot model is not accurate enough or if the localization is off. If set to false, the robot will try to move away from the colliding state, and if it fails it will stop.
• plan_visualization_width [double] [default=]: If set, plan will have this width on Sight
• plan_visualization_color [Vector4ub] [default=(Vector4ub{118, 185, 0, 100})]: Color to visualize plan with

isaac.lqr.DifferentialBaseLqrPlanner

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• odometry [Odometry2Proto]: Contains the odometry information required for planning (current speed, acceleration, etc.)
• global_plan [Plan2Proto]: Contains the target plan the local planner attempts to follow

Outgoing messages

• plan [DifferentialTrajectoryPlanProto]: Contains a series of poses to form the trajectory that is optimal to follow

Parameters

• time_between_command_ms [int] [default=100]: Step size to be used in integrating the state
• lqr_iterations [int] [default=100]: Upper limit on the number of iterations made by the lqr solver
• line_search_iterations [int] [default=50]: Upper limit on the number of iterations made by the lqr solver during the line search.
• lqr_solver_minimum_change [double] [default=1e-5]: The minimum change in the lqr score before stopping the solver.
• num_controls [int] [default=50]: Upper limit on the number of steps in the output trajectory plan
• min_num_controls [int] [default=5]: Lower limit on the number of steps in the output trajectory plan (only used if the flag use_adaptive_num_controls is set to true).
• use_adaptive_num_controls [bool] [default=false]: If true, the number of controls will adapt to the distance (but remain between min_num_controls and num_controls).
• adaptive_num_controls_factor [double] [default=5.0]: If use_adaptive_num_controls is true, this parameter will help to compute the number of control points to use: num_controls = adaptive_num_controls_factor * distance / (max_speed * dt)
• gain_lat_acceleration [double] [default=0.2]: parameters for the DifferentialLqr parameters Gain of a quadratic cost to penalize the lateral acceleration
• gain_to_target [double] [default=0.1]: Gain of a custom cost to penalize the robot according to its distance to the target
• gain_to_end_position_x [double] [default=20.0]: Gain of a quadratic cost to penalize the last position in forward/backward direction relative to the target
• gain_to_end_position_y [double] [default=50.0]: Gain of a quadratic cost to penalize the last position in lateral direction relative to the target
• gain_to_end_angle [double] [default=1.0]: Gain of a quadratic cost to penalize the robot if its orientation does not match the target
• gain_to_end_speed [double] [default=10.0]: Gain of a quadratic cost to penalize the robot if it is still moving
• gain_to_end_angular_speed [double] [default=10.0]: Gain of a quadratic cost to penalize the robot if it is still rotating
• max_speed [double] [default=0.75]: Soft limit on how fast we would like to move
• min_speed [double] [default=-0.0]: Soft limit on how slow we are allowed to move
• distance_to_target_sigma [double] [default=1.0]: Other parameters: Parameter that controls the strength of the gradient depending on the distance of the target The error cost is of the form: d^2/(d^2 + s^2). It behaves as a quadratic cost close to the target and as a constant value far away from the target.
• decay [double] [default=1.01]: Decay apply to each steps (decay < 1 means we accord higher importance to the beginning of the path while decay > 1 emphasizes the end of the path).
• distance_to_waypoint [double] [default=1.0]: Maximum distance the end of the plan needs to be from a waypoint before trying to move to the next waypoint.
• angle_to_waypoint [double] [default=DegToRad<double>(20.0)]: Maximum angle the end of the plan needs to be from a waypoint before trying to move to the next waypoint.
• state_planner_cost_name [string] [default=]: This is the name of the root PlannerCostBuilder component for the state. The cost of this

• control_planner_cost_name [string] [default=]: This is the name of the root PlannerCostBuilder component for the controls. The cost of this builder is computed on the control vector and is added for each position along the path. The default PlannerCostBuilder is a RangeConstraintsCostBuilder, but you can modify the cost function or add new ones by using a AdditionCostBuilder (see state_planner_cost_name description for more information on how you can customize a cost function).

• final_planner_cost_name [string] [default=]: This is the name of PlannerCostBuilder components. The cost of this builder is computed on the state vector for final position. There is no default one used, but you can add one (see state_planner_cost_name description for more information on how you can customize a cost function).

• path_cost_name [string] [default=]: Name fo the PolylineDistanceQuadraticCostBuilder in state planner cost to set the global path for the plan to follow.

• use_predicted_position [bool] [default=true]: Indicates whether or not the predicted position or actual position is used while planning. If true, this produces a more stable path, however it relies on a good controller to keep the robot on track. If false, then this codelet also acts as a controller.

• reset_robot_position [int] [default=0]: How frequently (in term of ticks) do we reset the robot position to the odometry: - 0: Disable it, never reset the robot position unless use_predicted_position is set to false. - 1: Always reset the robot position regardless of the value of use_predicted_position. - 10 Assuming use_predicted_position = true, means every 10 ticks we reset the robot position

  to where the odometry predict the robot to be.

• max_predicted_position_error [double] [default=0.5]: The distance from the predicted position we tolerate. If we exceed this value, the actual robot position is used.

• manual_mode_channel [string] [default=”“]: Channel publishing whether or not the robot is in manual mode

• print_debug [bool] [default=false]: Specifies whether to show extra information in Sight for debug purposes

• reuse_lqr_plan [bool] [default=true]: Specifies whether or not to use the previous plan as starting point for the lqr

• restart_planning_cycle [int] [default=10]: How frequently (in term of ticks) do we restart the planning from scratch: - 0: Disable it, never restart it (unless reuse_lqr_plan is set to false) - 1: Never reuse the plan (regardless of the value of reuse_lqr_plan). - 10 Assuming reuse_lqr_plan = true, means every 10 ticks we drop the previous plan and

  replan from a stopped position.

• static_frame [string] [default=”world”]: Name of a frame which is static. This is used to compensate for the odometry drift.

• stop_robot_on_collision [bool] [default=true]: Force the robot to stop when it’s colliding. This can prevent the robot from moving when it gets too close to an obstacle (a wall for example) if the robot model is not accurate enough or if the localization is off. If set to false, the robot will try to move away from the colliding state, and if it fails it will stop.

• plan_visualization_width [double] [default=]: If set, plan will have this width on Sight

• plan_visualization_color [Vector4ub] [default=(Vector4ub{118, 185, 0, 100})]: Color to visualize plan with

isaac.lqr.MultiJointLqrPlanner

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• acceleration_max [VectorXd] [default=]: Maximum acceleration for each joint for lqr solver
• acceleration_min [VectorXd] [default=]: Minimum acceleration for each joint for lqr solver
• speed_max [VectorXd] [default=]: Maximum speed for each joint for lqr solver
• speed_min [VectorXd] [default=]: Minimum speed for each joint for lqr solver
• gain_control [double] [default=1.0]: Gain for the control (to prevent high change of acceleration) for lqr solver
• gain_final_state [double] [default=10.0]: Gain for the the final state (for position, speed and acceleration) for lqr solver
• gain_limits [double] [default=100.0]: Gain when the speed or acceleration is outside the target range for lqr solver

isaac.map.AdditionFlatmapCost

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

(none)

isaac.map.DirectedAreaFlatmapCost

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• color [Vector3ub] [default=(Vector3ub{120, 240, 120})]: Color to visualize the polygon in the map.
• arrows_distance [Vector2d] [default=(Vector2d{1.1, 1.1})]: The distance between two arrows, in rows and columns.
• arrow_length [double] [default=0.8]: The length of the vector arrow.
• arrow_opening_angle [double] [default=0.6]: The opening angle of the vector arrow.
• polygon [std::vector<Vector2d>] [default=]: Corners of the polygon covering the directed area.
• target_angle [double] [default=]: The angle of the area’s direction.
• penality_angle [double] [default=1.0]: The scaling factor of the cost associated to the delta angle.
• center_tendency_weight [double] [default=10.0]: Weight for how strong the tendency is to go to the center of the polygon.
• half_cost_fraction [double] [default=0.25]: Determines at which lateral displacement fraction to either side of the polygon relative to its center half of the center_tendency_weight should be reached. This decides on the aggressiveness with which a plan is pushed into the middle lane of the directed area.
• inside_offset [double] [default=1.0]: The base offset when close to the polygon.
• outside_weight [double] [default=1.0]: The weight when we are too far away from the polygon.
• inside_backward_penalty [double] [default=]: The penalty associated to backward motion inside the polygon. If empty, then backward is not allowed inside.
• outside_backward_penalty [double] [default=]: The penalty associated to backward motion outside the polygon. If empty, then backward is not allowed outside.
• delta_angle_range [Vector2d] [default=]: It contains the valid range of the angle relative to the target_angle. If not provided all the orientations will be valid.
• use_robot_center [bool] [default=true]: If true, we only use the robot center to decide if a point is inside the area. This is much faster than using the robot_shape. If false, then the robot shape and orientation will be used to decide whether the position is inside the area.

isaac.map.KinematicTree

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• kinematic_file [string] [default=]: Path to a file to load the kinematic tree from

isaac.map.KinematicTreeToPoseTree

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• state [CompositeProto]: Current joint state

Outgoing messages

(none)

Parameters

• kinematic_tree [string] [default=]: Name of the node containing the map::KinematicTree component
• root_frame [string] [default=]: Name of the root frame of the kinematic tree. This frame is not part of the kinematic tree, since it depends on where the root link of the kinematic tree is mounted on. This config is only used in start.

isaac.map.Map

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• graph_file_name [string] [default=”/tmp/graph_changes.json”]: Filename under which to store the current graph whenever there is an update to the map.
• config_file_name [string] [default=”/tmp/config_changes.json”]: Filename under which to store the current configuration whenever there is an update to the map.

isaac.map.MapBridge

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• request [nlohmann::json]: Request to the MapBridge

Outgoing messages

• reply [nlohmann::json]: Reply from the MapBridge

Parameters

(none)

isaac.map.MinimumFlatmapCost

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

(none)

isaac.map.MultiplicationFlatmapCost

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

(none)

isaac.map.ObstacleAtlas

Description

Type: Component - This component does not tick and only provides certain helper functions.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• static_frame [string] [default=”world”]: Frame which can be considered static, it is used to do time synchronization of obstacles.

isaac.map.OccupancyFlatmapCost

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• map_name [string] [default=”map/isaac.navigation.DistanceMap”]: Name of the component that contains the DistanceMap used to check for detection and get the dimension of the world.
• occupancy_grid_layer [string] [default=”map/occupancy”]: Name of the component that contains the DistanceMap used to check for detection and get the dimension of the world.
• target_distance [double] [default=0.5]: The distance before we start penalizing the node. Any node further away than this distance to an obstacle won’t be penalized, however if the distance is smaller, then a penality of (sampling_distance^2 - distance^2) * target_distance_penality will be assigned to the position
• penality [double] [default=4.0]: Scale used to penalized node too close to obstacles (see target_distance for more information).
• offset_weight [double] [default=1.0]: Offset weight, this is the minimum weight returned by this function which corresponds to position in the map further away than the target distance.
• backward_penalty [double] [default=]: The penalty associated to backward motion. If empty, then backward is not allowed.

isaac.map.OccupancyGridMapLayer

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• filename [string] [default=]: Filename of grayscale PNG which will be loaded as the occupancy grid map
• cell_size [double] [default=]: Size of one map pixel in meters
• threshold [double] [default=0.4]: Threshold used to compute the distance map. Cells with a value larger than this threshold are assumed to be blocked.

isaac.map.PolygonFlatmapCost

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• polygon_layer_name [string] [default=]: Name of the component that contains the PolygonMapLayer used to get the list of polygons;
• polygon_names [std::vector<std::string>] [default=]: List of the names of the polygon in the PolygonMapLayer to be used to compute the cost. If it is not provided all the polygons will be used.
• inside_weight [double] [default=]: The weight of a position inside the polygon areas.
• outside_weight [double] [default=]: The weight of a position outside the polygon areas.
• use_robot_center [bool] [default=true]: If true, we only use the robot center to decide if a point is inside the area. This is much faster than using the robot_shape. If false, then the robot shape and orientation will be used to decide whether the position is inside the area.
• inside_backward_penalty [double] [default=]: The penalty associated to backward motion inside the polygon. If empty, then backward is not allowed inside.
• outside_backward_penalty [double] [default=]: The penalty associated to backward motion outside the polygon. If empty, then backward is not allowed outside.

isaac.map.PolygonMapLayer

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• polygons [json] [default=nlohmann::json::object()]: A json object from configuration containing the polygons.

            

            
Layout:
  {
    "poly1": {
       "points": [[<polygon point1>], [<polygon point2>]],
     },
  }
        

• color [Vector3i] [default=(Vector3i{120, 120, 120})]: Layer color.
• frame [string] [default=”world”]: Frame the polygons are defined in.
• obstacle_max_distance [double] [default=1.5]: The maximum distance to consider to create the obstacle from the polygons
• obstacle_pixel_size [double] [default=0.1]: The resolution of the map used to create the obstacle from the polygons

isaac.map.PolylineFlatmapCost

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• color [Vector3ub] [default=(Vector3ub{120, 240, 120})]: Color to visualize the polygon in the map.
• polyline [std::vector<Vector2d>] [default=]: Color to visualize the areas in the map.
• width [double] [default=]: The distance we consider the position to be influenced by the polyline
• penality_angle [double] [default=1.0]: The scaling factor of the cost associated to the delta angle.
• penality_distance [double] [default=1.0]: The scaling factor of the cost associated to the distance to the polyline.
• inside_offset [double] [default=1.0]: The base offset when close to the polyline.
• outside_weight [double] [default=1.0]: The weight when we are too far away from the polyline
• inside_backward_penalty [double] [default=]: The penalty associated to backward motion close to the polyline. If empty, then backward is not allowed in the area of influence of the polyline.
• outside_backward_penalty [double] [default=]: The penalty associated to backward motion far from the polyline. If empty, then backward is not allowed.
• delta_angle_range [Vector2d] [default=]: It contains the valid range of the angle relative to the target_angle. If not provided all the orientations will be valid.

isaac.map.Spline

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• json [JsonProto]: Outgoing message that defines this spline

Parameters

• filename [string] [default=]: If set, spline points will be read from file. Expected layout:

isaac.map.SurfletModelAtlas

Description

Type: Component - This component does not tick and only provides certain helper functions.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• cask [string] [default=]: Cask filename which contains surflet models.

isaac.map.SurfletModelPublisher

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• surflets [CompositeProto]: The requested surflet model as a CompositeProto.

Parameters

• model_atlas [string] [default=]: Name of surflet model atlas database. This has the form node_name/component_name.
• model_names [std::vector<std::string>] [default=]: List of names of surflet models in atlas database
• model_frame_names [std::vector<std::string>] [default=]: List of names of coordinate frames to use for surflet models
• reference_frame_name [string] [default=]: Name of reference frame of the published surflet model
• tick_count [int] [default=1]: Number of times to publish before reporting success.

isaac.map.WaypointMapLayer

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• waypoints [json] [default=nlohmann::json::object()]: A json object from configuration containing the waypoints.

            

            
Layout:
  {
    "wp1": { "pose": [1,0,0,0,0,0,0], "radius": 0.5 },
    "wp3": { "pose": [1,0,0,0,0.1,-1.2,0], "color": [54.0, 127.0, 255.0] }
  }
        

isaac.message_generators.BinaryTensorGenerator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• assignment [TensorProto]: Produces tensor of type int with entries set randomly to 0 or 1.

Parameters

• dimensions [Vector2i] [default=Vector2i(1, 100)]: Dimensions of the generated rank 2 tensor
• probability [double] [default=0.5]: Probability for assignment

isaac.message_generators.CameraGenerator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• color_left [ImageProto]: Random left color image
• color_right [ImageProto]: Random right color image
• depth [ImageProto]: Random depth image

Parameters

• rows [int] [default=1080]: The number of rows for generated data
• cols [int] [default=1920]: The number of columns for generated data

isaac.message_generators.CameraIntrinsicsGenerator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• image [ImageProto]: Output message will be published with the acqtime of this input message

Outgoing messages

• intrinsics [CameraIntrinsicsProto]: Intrinsics including pinhole and distortion parameters

Parameters

• distortion_model [string] [default=”brown”]: Image undistortion model. Must be ‘brown’ or ‘fisheye’
• focal_length [Vector2d] [default=]: Focal length in pixels
• optical_center [Vector2d] [default=]: Optical center in pixels
• distortion_coefficients [Vector5d] [default=Vector5d::Zero()]: Distortion coefficients

isaac.message_generators.ConfusionMatrixGenerator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• confusion_matrix [ConfusionMatrixProto]: Output segmentation prediction with regulated probabilities

Parameters

(none)

isaac.message_generators.Detections2Generator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• mock_detections [Detections2Proto]: Output mocked detection proto message

Parameters

• detection_configuration [json] [default={}]: Parameter defining the configuration of the detections we need to mock.

isaac.message_generators.Detections3Generator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• output_poses [Detections3Proto]: publish dummy poses to test pose refinement

Parameters

• pose [Pose3d] [default=]: Randomized pose to be added to dummy pose to generate different start positions
• label [string] [default=”object”]: object label param for pose
• reference [string] [default=”camera”]: reference label param for pose
• num_detections [int] [default=1]: Number of detections at output

isaac.message_generators.DifferentialBaseControlGenerator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• command [StateProto]: A StateProto representing a navigation::DifferentialBaseControl state which is populated with values specified via parameters

Parameters

• linear_speed [double] [default=0.0]: Linear speed in outgoing state message
• angular_speed [double] [default=0.0]: Angular speed in outgoing state message

isaac.message_generators.DifferentialBaseStateGenerator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• state [StateProto]: Output state of differential base (DifferentialBaseDynamics)

Parameters

• linear_speed [double] [default=1.0]: Linear speed in outgoing state message
• angular_speed [double] [default=0.1]: Angular speed in outgoing state message
• linear_acceleration [double] [default=-0.1]: Linear acceleration in outgoing state message
• angular_acceleration [double] [default=0.05]: Angular acceleration in outgoing state message

isaac.message_generators.FlatscanGenerator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• flatscan [FlatscanProto]: Outgoing “flat” range scan

Parameters

• invalid_range_threshold [double] [default=0.2]: Beams with a range smaller than or equal to this distance are considered to have returned an invalid measurement.
• out_of_range_threshold [double] [default=100.0]: Beams with a range larger than or equal to this distance are considered to not have hit an obstacle within the maximum possible range of the sensor.
• beam_count [int] [default=1800]: Number of beams in outgoing message
• angles_range [Vector2d] [default=Vector2d(0.0, TwoPi<double>)]: Azimuth angle range for the beams
• range_mean [double] [default=20.0]: Mean value for the ranges.
• range_standard_deviation [double] [default=]: Standard deviation for the range values. Requires an alice::Random component in the same node.

isaac.message_generators.HolonomicBaseControlGenerator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• command [StateProto]: A StateProto representing a navigation::HolonomicBaseControls state which is populated with values specified via parameters

Parameters

• speed_angular [double] [default=0.0]: Angular speed in counter-clockwise direction
• speed_linear_x [double] [default=0.0]: Linear speed in forward direction
• speed_linear_y [double] [default=0.0]: Linear speed in left direction

isaac.message_generators.ImageFeatureGenerator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• image [ImageProto]: Output mockup image
• coordinates [TensorProto]: Output keypoint coordinates
• features [TensorProto]: Output features ids. A rank-1 int tensor with dimensions (number of features).

Parameters

• cell_size [int] [default=50]: Chessboard cell size in pixel
• image_rows [int] [default=512]: Image height in pixels
• image_cols [int] [default=512]: Image width in pixels
• animation_speed [float] [default=2.0]: Chessboard shift per tick in pixels

isaac.message_generators.ImageLoader

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• color [ImageProto]: Output the color camera image proto
• depth [ImageProto]: Output the depth image proto

Parameters

• color_filename [string] [default=]: Path of the color image file. The image is expected to be a 3-channel RGB PNG.
• depth_filename [string] [default=]: Path of the depth image file. The image is expected to be a 1-channel 16-bit grayscale PNG.
• color_glob_pattern [string] [default=]: Path of the color image directory. The directory is expected to contain only 3-channel RGB PNG.

• loop_images [bool] [default=true]: The images in the specified directory plays in a loop if set to true. Otherwise it plays once.
• sort_by_number [bool] [default=false]: The images in the directory are sorted by NUMBER when they are ‘NUMBER.jpg’ or ‘NUMBER.png’.
• depth_scale [double] [default=0.001]: A scale parameter to convert 16-bit depth to f32 depth
• distortion_model [string] [default=”brown”]: Image undistortion model. Must be ‘brown’ or ‘fisheye’
• focal_length [Vector2d] [default=]: Focal length in pixels
• optical_center [Vector2d] [default=]: Optical center in pixels
• distortion_coefficients [Vector5d] [default=Vector5d::Zero()]: Distortion coefficients (see the DistortionProto in Camera.capnp for details)
• min_depth [double] [default=0.0]: Minimum depth
• max_depth [double] [default=10.0]: Maximum depth

isaac.message_generators.LatticeGenerator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• gridmap_lattice [LatticeProto]: Output lattice proto. This contains relevant information about the corresponding gridmap.

Parameters

• cell_size [double] [default=0.05]: Parameter defining the cell size in metres.
• dimensions [Vector2i] [default=Vector2i(256, 256)]: The dimensions of the grid map in pixels
• lattice_frame_name [string] [default=”gridmap_frame”]: The name of the lattice coordinate frame. This will be used to write the pose of the gridmap relative to the reference frame in the pose tree.
• reference_frame_name [string] [default=”ref”]: Name of the reference frame
• relative_offset [Vector2d] [default=Vector2d(0.0, -0.5)]: Percentage offset of robot relative to the map. The offset determines the position of the robot (or the reference frame) with respect to the grid map created. The origin of the grid map is considered to be at the top-left of the grid. The x parameter defines the percentage offset for the rows (positive is in the upward direction and negative is in the downward direction), and the y parameter defines the offset for the columns (positive is in the left direction and negative is in the right direction). Determining the offset using a percentage basis makes it agnostic to the dimensions of the map. The default value fixes the reference frame at the top-center of the grid map.

isaac.message_generators.PanTiltStateGenerator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• target [StateProto]: Target state

Parameters

• pan_frequency [double] [default=1.0]: Frequency of pan reference (Hz)
• pan_max_angle [double] [default=1.0471975511965976]: Max panning on one side in radians
• pan_offset_angle [double] [default=0.0]: Degree offset for panning
• pan_speed [double] [default=0.1]: Speed for panning in revolutions/second
• pan_mode [WaveMode] [default=WaveMode::kSinus]: Wave function for panning
• tilt_frequency [double] [default=1.0]: Frequency of tilt reference (Hz)
• tilt_max_angle [double] [default=0.5235987755982988]: Max tilting on one side in radians
• tilt_offset_angle [double] [default=0.0]: Degree offset of tilting
• tilt_speed [double] [default=0.1]: Speed for tilting in revolutions/second
• tilt_mode [WaveMode] [default=WaveMode::kSinus]: Wave function for tilting

isaac.message_generators.ParkingSpotListGenerator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• parking_spot_list [ParkingSpotListProto]: Output the generated parking spot list

Parameters

• num_detections [int] [default=3]: The number of parking spot detections to generate
• spot_width [double] [default=3.0]: The width(meters) of the generated parking spot detections
• spot_length [double] [default=6.0]: The length (meters) of the generated parking spot detections
• spot_pitch [double] [default=3.1]: The pitch (meters) between the generated parking spot detections
• spot_entry_line_index [int] [default=3]: The entry line index (0-3) for the generated parking spot detections

isaac.message_generators.Plan2Generator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• plan [Plan2Proto]: The plan generated as specified via parameters
• goal [Goal2Proto]: Last waypoint of the plan

Parameters

• waypoints [std::vector<Pose2d>] [default=]: List of waypoint poses in the form of (angle, x, y).

• frames [std::vector<std::string>] [default=]: List of waypoints as frame names defined in PoseTree. Either ‘waypoints’ or ‘frames’ parameter needs to be set.
• plan_frame [string] [default=”world”]: Frame for the waypoints. Sets the plan frame in outgoing message.
• static_frame [string] [default=”world”]: Name of a frame that is not moving. Used to decide whether a new plan message needs to be published.
• report_success [bool] [default=false]: Report success after publishing the plan
• erase_path_on_stop [bool] [default=true]: Whether or not to erase the path in sight when stopping
• new_message_threshold [Vector2d] [default=Vector2d(1e-3, DegToRad(0.01))]: A new message will be published whenever change in poses exceeds this threshold. Values are for Euclidean distance and angle respectively.

isaac.message_generators.PointCloudGenerator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• point_cloud [PointCloudProto]: Outgoing proto messages used to publish the point cloud messages.

Parameters

• point_count [int] [default=10000]: Total number of point to generate.
• point_per_message [int] [default=100]: Maximum number of points in a single given message.
• has_normals [bool] [default=false]: Whether there should be normals in the messages, as many as the number of points.
• has_colors [bool] [default=false]: Whether there should be colors in the messages, as many as the number of points.
• has_intensities [bool] [default=false]: Whether there should be intensities in the messages, as many as the number of points.

isaac.message_generators.Polyline2Generator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• polyline [Polyline2dProto]: Output generated polyline

Parameters

• prototype [std::vector<Vector2d>] [default={}]: The prototype of the generated polyline

isaac.message_generators.PoseGenerator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• lhs_frame [string] [default=]: Name of the reference frame of the left side of the pose
• rhs_frame [string] [default=]: Name of the reference frame of the right side of the pose
• initial_pose [Pose3d] [default=Pose3d::Identity()]: Initial pose
• step [Pose3d] [default=Pose3d::Translation({1.0, 0.0, 0.0})]: The pose delta for every tick

isaac.message_generators.RangeScanGenerator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• scan [RangeScanProto]: Outgoing range scan

Parameters

• azimuth_angle_range [Vector2d] [default=Vector2d(0.0, TwoPi<double>)]: Azimuth angle range for the beams. (2pi, 0) would produce counter-clockwise rotation.
• num_slices [int] [default=16]: Number of (horizontal) ray slices that cover azimuth_angle_range
• num_slices_per_message [int] [default=0]: Number of (horizontal) ray slices published with each message. 0 means publish num_slices each message. Needs to be smaller than num_slices.
• vertical_beam_angles [std::vector<double>] [default=std::vector<double>({DegToRad(-15.0), DegToRad(-7.0), DegToRad(-3.0), DegToRad(-1.0), DegToRad(+1.0), DegToRad(+3.0), DegToRad(+7.0), DegToRad(+15.0)})]: The (vertical) beam angles to use for every slice
• max_range [double] [default=100.0]: Out of range threshold
• min_range [double] [default=0.0]: Invalid range threshold
• range_domain_max [double] [default=110.0]: Max value of the range domain. Used when normalizing range values.
• delta_time [int] [default=50‘000]: Delay in microseconds between firing. Default is 20 Hz.
• intensity_denormalizer [double] [default=1.0]: Scale factor which can be used to convert an intensity value from an 8-bit integer to meters.
• height [double] [default=1.0]: The height of the lidar over the ground plane
• segments [std::vector<geometry::LineSegment2d>] [default={}]: Lines in range / height plane which define the world

isaac.message_generators.RigidBody3GroupGenerator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• bodies [RigidBody3GroupProto]: Output group with a single body

Parameters

• body_name [string] [default=”dummy_body”]: Name of the body
• reference_frame [string] [default=”world”]: Reference frame for the body
• pose [Pose3d] [default=Pose3d::Identity()]: Pose of the body with respect to the reference frame
• linear_velocity [Vector3d] [default=Vector3d::Zero()]: Linear velocity of the body
• angular_velocity [Vector3d] [default=Vector3d::Zero()]: Angular velocity of the body
• linear_acceleration [Vector3d] [default=Vector3d::Zero()]: Linear acceleration of the body
• angular_acceleration [Vector3d] [default=Vector3d::Zero()]: Angular acceleration of the body

isaac.message_generators.TensorGenerator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• sample [TensorProto]: Produced random list of tensors with the specified dimensions

Parameters

• dimensions [Vector3i] [default=Vector3i(3, 640, 480)]: Dimensions of the generated rank 3 tensor
• element_type [TensorGeneratorElementType] [default=TensorGeneratorElementType::kFloat32]: The element type for the tensor

isaac.message_generators.TrajectoryListGenerator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

• trajectories [Vector3TrajectoryListProto]: The output channel to send all generated trajectories.

Parameters

• frame [string] [default=”world”]: Reference frame for the generated trajectories.
• position_count [int] [default=60]: Number of positions in the generated trajectory.
• helix_radius [double] [default=5.0]: The radius of the vertical helix created as the made up trajectory.
• position_delta_angle [double] [default=0.1]: The delta angle between each positions in the generated trajectory.

isaac.ml.BoundingBoxPadding

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• input_detections [Detections2Proto]: Input list of bounding boxes that need to be padded

Outgoing messages

• output_detections [Detections2Proto]: Detections with padded bounding boxes

Parameters

• bbox_padding_range [Vector2i] [default=]: If set, the bounding box is padded with pixels and the number of pixels padded on each side of the bbox is given by a random number in the range specified by the parameter with uniform distribution. If set to [3, 7] - random padding on each side with pixel number between 3 and 7. If set to [3, 3] - bounding box is padded by 3 pixels on each of the 4 sides.
• image_dimensions [Vector2i] [default=]: Dimensions of the image corresponding to the bounding box This parameter is needed to ensure that the bounding boxes after padding are within the image dimensions.

isaac.ml.ColorCameraEncoderCpu

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• rgb_image [ImageProto]: Input RGB color image

Outgoing messages

• tensor [TensorProto]: A rank 3 tensor with image data normalized and transformed according to parameters.

Parameters

• rows [int] [default=960]: The image is resized before it is encoded. Currently, only downsampling is supported for this. Number of pixels in the height dimension of the downsampled image.
• cols [int] [default=540]: The image is resized before it is encoded. Currently, only downsampling is supported for this. Number of pixels in the width dimension of the downsampled image.
• pixel_normalization_mode [ImageToTensorNormalization] [default=ImageToTensorNormalization::kNone]: Type of Normalization to be performed.
• tensor_index_order [TensorTransposeOp] [default=TensorTransposeOp::k012]: The indexing order, default is {row, column, channel}

isaac.ml.ColorCameraEncoderCuda

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• rgb_image [ImageProto]: Input RGB color image

Outgoing messages

• tensor [TensorProto]: A rank 3 tensor with image data normalized and transformed according to parameters.

Parameters

• rows [int] [default=960]: The image is resized before it is encoded. Currently, only downsampling is supported for this. Number of pixels in the height dimension of the downsampled image.
• cols [int] [default=540]: The image is resized before it is encoded. Currently, only downsampling is supported for this. Number of pixels in the width dimension of the downsampled image.
• keep_aspect_ratio [bool] [default=true]: The aspect ration of the image is preserved during resizing, the ROI is centered and padded.
• pixel_normalization_mode [ImageToTensorNormalization] [default=ImageToTensorNormalization::kUnit]: Type of Normalization to be performed. Todo: Add additional normalization modes besides unit for cuda
• tensor_index_order [TensorTransposeOp] [default=TensorTransposeOp::k012]: The indexing order, default is {row, column, channel}

isaac.ml.ConfusionMatrixAggregator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• sample_metrics [ConfusionMatrixProto]: incoming metrics message

Outgoing messages

• accumulated_metrics [ConfusionMatrixProto]: outgoing accumulated metrics message

Parameters

• confusion_matrix_slice_index [int] [default=0]: Index to specify which slice of the confusion matrix we want to visualize. The slicing is done along the third dimension. Hence each slice of the matrix represents a 2D tensor which is a single confusion matrix for a particular intersection over union threshold.

isaac.ml.Detection3Encoder

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• detection3 [Detections3Proto]: Input detection proto.

Outgoing messages

• tensor [TensorProto]: A tensor of dimensions (N, 8) where N is the number of rigid bodies. Channels are as follows:

Parameters

• class_names [std::vector<std::string>] [default={}]: List of class names to detect as string

isaac.ml.DetectionComparer

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• ground_truth_detection [Detections2Proto]: Ground truth object detection input
• predicted_detection [Detections2Proto]: Predicted detection input

Outgoing messages

• metrics [ConfusionMatrixProto]: Output containing the evaluated metrics for the detections in a single sample.

Parameters

• intersection_over_union_thresholds [std::vector<double>] [default=std::vector<double>({0.5, 0.8, 0.95})]: List of intersection over union thresholds over which the metrics are computed
• class_names [std::vector<std::string>] [default={}]: The allowed class names for the ground truth and predicted detections. The confusion matrix is constructed by assigning an index to each of these classes and one for the background class. If a sample contains any class other than the ones specified here, it gets dropped.

isaac.ml.DetectionEncoder

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• detection [Detections2Proto]: Input detection proto.

Outgoing messages

• tensor [TensorProto]: Detection encoded as a (N, 5) tensor where N is the number of bounding boxes. Channels are: (bb_min_x, bb_min_y, bb_max_x, bb_max_y, class_id).

Parameters

• class_names [json] [default={}]: The class names of our detection objects.
• area_threshold [double] [default=10.0]: The minimum area of bounding boxes

isaac.ml.Detections3Comparer

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• reference_poses [Detections3Proto]: Reference pose
• predicted_poses [Detections3Proto]: Predicted pose

Outgoing messages

• statistics [JsonProto]: Outputs statistics about the reference_poses and the predicted_poses.

Parameters

• symmetry_rotations [std::vector<SO3d>] [default={}]: List of rotation symmetry of the object For example, settin gparam to [{ “axis”: [0, 0, 1], “angle_degrees”: 180 }] indicates the object is symmetric about z axis and the symmetry angle is 180 degrees. TODO: Extend to multiple object classes

isaac.ml.FilterDetections

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• input_detections [Detections2Proto]: A list of detections which may have different labels.

Outgoing messages

• output_detections [Detections2Proto]: A subset of the input detections, filtered by matching the included and excluded labels.

Parameters

• allowlist_labels [std::vector<std::string>] [default=]: The labels for which to include only detections by string match. Includes all if not set. NOTE Set either allowlist_labels or denylist_labels, not both.
• denylist_labels [std::vector<std::string>] [default=]: The labels for which to exclude some detections by string match. Excludes none if not set. NOTE Set either allowlist_labels or denylist_labels, not both.
• max_detections [int] [default=1]: Maximum number of detections to output. If number of incoming detections exceeds this number, smaller detections are filtered out.
• image_cols [int] [default=1280]: Width of the image corresponding to the detections. This is used to compute the metric used to filter the top N detections.
• focus_factor [double] [default=0.5]: Fraction of the image width. This parameter is used to filter the top N detections with maximum area in the region of interest.

isaac.ml.GenerateKittiDataset

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• image [ImageProto]: Image input (must be a three-channel color image of image3ub)
• detections [Detections2Proto]: Detections associated with the image input

Outgoing messages

(none)

Parameters

• num_training_samples [int] [default=1000]: The total number of training samples to generate
• num_testing_samples [int] [default=100]: The total number of testing samples to generate
• path_to_dataset [string] [default=]: Path to the root of the KITTI dataset. See https://docs.nvidia.com/metropolis/TLT/tlt-getting-started-guide/index.html#kitti_file for file structure and organization.

isaac.ml.HeatmapDecoder

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• tensor [TensorProto]: Input tensor containing heatmap of probabilities

Outgoing messages

• heatmap [HeatmapProto]: Output heatmap proto

Parameters

• grid_cell_size [double] [default=2.0]: Cell size (in metres) of every pixel in heatmap
• map_frame [string] [default=”world”]: The pose map frame for the heatmap

isaac.ml.HeatmapEncoder

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• heatmap_proto [HeatmapProto]: Input heatmap proto containing heatmap image of probabilities

Outgoing messages

• heatmap_tensor [TensorProto]: Heatmap encoded as a tensor

Parameters

(none)

isaac.ml.ImageDetectionExtraction

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• detections [Detections2Proto]: Input list of bounding boxes to crop to. Labels and confidence are not used.
• image [ImageProto]: Input RGB color image or segmentation image that needs to converted to tensor

Outgoing messages

• tensor [TensorProto]: Cropped and resized output as batch of image tensors with dimension (Nd, rows, cols, Nc) or (Nd, Nc, rows, cols) depending on the TensorIndexOrder if Nc =3. (Nd, rows, cols) if input image proto contains a single channel image. (Nc = 1) Nd is the number of input detection, rows, cols are the rows and columns of the output image and Nc is number of channels which is either 3 for rgb or 1 for segmentation image.

Parameters

• downsample_size [Vector2i] [default=]: Target dimensions (rows, cols) for downsample after crop.
• pixel_normalization_mode [ImageToTensorNormalization] [default=ImageToTensorNormalization::kNone]: Type of Normalization to be performed. Valid modes are {kNone, kUnit, kPositiveNegative, kHalfAndHalf}
• tensor_index_order [TensorTransposeOp] [default=TensorTransposeOp::k012]: The indexing order of the output tensor, default is {row, column, channel}

isaac.ml.LabelToBoundingBox

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• class_segmentation [ImageProto]: Input image with class labels.
• instance_segmentation [ImageProto]: Input image with instance labels.
• class_labels [LabelProto]: Label mapping of class indices to class names

Outgoing messages

• detections [Detections2Proto]: Computed bounding boxes - one AABB for every occuring instance/label combination.

Parameters

• resolution [int] [default=1]: The target resolution when computing bounding boxes. A value of 1 means bounding boxes are pixel-accurate. A value of 3 would mean bounding boxes are accurate up to 3 pixels.
• min_bbox_size [int] [default=1]: Minimum size in pixels across the two dimensions of the rectangle to be considered as non-zero size bounding box. A value of 1 means the bounding box rectangle length and breadth must be at least one pixel.

isaac.ml.ResizeDetections

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• detections [Detections2Proto]: Input detections associated with an image of dimensions specified by the input_image_dimensions ISAAC_PARAM

Outgoing messages

• resized_detections [Detections2Proto]: Output detections associated with an image of dimensions specified by the output_image_dimensions ISAAC_PARAM

Parameters

• input_image_dimensions [Vector2d] [default=]: Resolution of the image (rows, cols) that the input detections were computed for.
• output_image_dimensions [Vector2d] [default=]: Resolution of the image (rows, cols) that the output detections should be transformed to.

isaac.ml.RigidbodyToDetections3

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• rigid_bodies [RigidBody3GroupProto]: Input list of 3D rigid body poses in Isaac SDK coordinate frame

Outgoing messages

• detections [Detections3Proto]: Output list of 3D rigid body poses with respect to desired rigid body coordinate frame from the input list. Index of this reference rigid body in the input list is given by input parameter, ref_frame_id.

Parameters

• ref_frame_id [int] [default=0]: Index of the rigid body in input list that is used as reference coordinate frame for publishing the poses of all the rigid bodies in the input list. If ref_frame_id < 0, object poses are published with respect to Isaac SDK coordinate frame.

isaac.ml.SampleAccumulator

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

(none)

Outgoing messages

(none)

Parameters

• sample_buffer_size [int] [default=256]: Number of training samples to keep in the buffer
• randomize_samples [bool] [default=true]: Randomize the order of samples in the buffer when true
• channel_names [std::vector<std::string>] [default={“samples”}]: Names of input channels. A sample will contain one tensor for each input channel in the given order.

isaac.ml.SegmentationComparer

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• ground_truth [TensorProto]: Ground truth object segmentation input
• prediction [TensorProto]: Predicted segmentation input

Outgoing messages

• metrics [ConfusionMatrixProto]: Output containing the evaluated metrics for the segmentations in a single sample.

Parameters

• argmax_threshold [double] [default=0.5]: The discretization threshold that was used to convert a 3 dimensional tensor to a 2 dimensional tensor in TensorArgMax. This parameter is repeated here so as to fill the “thresholds” parameter in the ConfusionMatrixProto. During evaluation, it’s important to know the confidence threshold that was used to decide if a prediction was valid or not, since the confusion matrix produced could vary depending on this threshold. Hence, although the inference results provided to this comparer codelet have already been filtered based on a threshold, we repeat the parameter for information so that it can be propagated downstream.
• number_of_classes [int] [default=0]: Number of classes expected from ground truth and prediction. This information is needed to build a confusion matrix of the appropriate size and to determine the class indices to compare. For example, if this value is set to 2, we’d consider classes 0 and 1 while comparing the ground truth and predicted segmentations.

isaac.ml.SegmentationDecoder

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• tensors [TensorProto]: The input tensor contains semantic segmentation label prediction where each pixel has a probability distribution over all classes. Dimensions: (rows, cols, number of classes)

Outgoing messages

• segmentation_prediction [SegmentationPredictionProto]: Output segmentation prediction proto which contains the class information

Parameters

• class_names [json] [default={}]: name of the classes in an array. Each class is represented by a string. The number of classes must match the number of classes in the tensor input.

isaac.ml.SegmentationEncoder

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• class_segmentation [ImageProto]: Input segmentation image.
• labels [LabelProto]: Label mapping for the input segmentation image

Outgoing messages

• tensor [TensorProto]: Ouput tensor encoding the segmentation image. Dimensions are (row, cols, channels) where the number of channels is 1 if the number of classes is 2 (as in binary segmentation), or (n + 1) if the number of classes is ‘n’ (as in multiclass segmentation).

Parameters

• rows [int] [default=256]: Height of the downsampled segmentation image
• cols [int] [default=512]: Width of the downsampled segmentation image
• offset [int] [default=1]: Offset by which the actual pixel value differs from the label index. The segmentation image comes with a per-pixel integer labelling which denotes the object it belongs to. This integer is in turn mapped to a string label name in the format <object_name>:<label_index>. In some cases (such as NavSim), the label index could be equal to the pixel value, and in other cases (such as Isaac Sim) they could differ by a fixed offset value.
• class_label_indices [std::vector<int>] [default=]: The pixel values that are to be encoded as valid classes, in case the labels are not provided. These are only used in the absence of the strings labels to be encoded, and if we specify the input mode as “NoLabelsAvailable”
• input_mode [InputMode] [default=InputMode::kLabelsAvailable]: Expected data input mode. When the string labels are available, the mode would be “LabelsAvailable” When there are no string labels available, the mode would be “NoLabelsAvailable”. In this case, the encoder looks for the pixel values which are to be encoded as valid classes.
• output_type [OutputMode] [default=OutputMode::kDistribution]: Parameter defining the format of the output tensor. If the mode is “Index”, we publish a 3D integer tensor, where each element at position (row, col, 0) represents the index of the class that the corresponding pixel at position (row, col) belongs to. If the mode is “Distribution”, we publish a tensor representing the probability distribution over the classes.
• class_label_names [std::vector<std::string>] [default={}]: A list of string labels representing the classes which need to be encoded. Typically, the input proto message contains - * An image where each pixel has a numerical value that represents its class. * The mapping of these numerical values to their string labels. We might want to encoded a subset of these classes in the output tensor. This subset is determined by the class_label_names parameter. Pixels which belong to classes other than the ones specified in this parameter are counted as “everything else”.

isaac.ml.Teleportation

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• relative_frame [Pose3dProto]: Proto used to receive a reference frame (pose)

Outgoing messages

• rigid_command [RigidBody3GroupProto]: Proto used to publish rigid body pose to the sim bridge
• relative_frame_cmd [Pose3dProto]: Proto used to publish rigid body pose to another teleportation codelet as a reference frame

Parameters

• enable_on_relative_frame [bool] [default=false]: Flag to tick on relative frame message
• name [string] [default=”“]: Name of actor to teleport
• min_scale [double] [default=0.0]: Mimimum multiplicational scale factor of corresponding objects in simulation
• max_scale [double] [default=1.0]: Maximum multiplicational scale factor of corresponding objects in simulation
• enable_scale [bool] [default=false]: Flag to enable scale
• base_mode [BaseMode] [default=BaseMode::kBox]: Parameter for step 1. Specifies how the base pose will be generated. Please see codelet summary for a list of modes.
• min [Vector3d] [default=Vector3d::Zero()]: Parameter for “box” mode of step 1. Minimum translation in X, Y, Z coordinates
• max [Vector3d] [default=Vector3d(1.0, 1.0, 1.0)]: Parameter for “box” mode of step 1. Maximum translation in X, Y, Z coordinates
• enable_translation_x [bool] [default=true]: Parameter for “box” mode of step 1. Flag to enable translation (X)
• enable_translation_y [bool] [default=true]: Parameter for “box” mode of step 1. Flag to enable translation (Y)
• enable_translation_z [bool] [default=true]: Parameter for “box” mode of step 1. Flag to enable translation (Z)
• min_roll [double] [default=0.0]: Parameter for “box” mode of step 1. Minimum roll change after a teleoperation
• max_roll [double] [default=TwoPi<double>]: Parameter for “box” mode of step 1. Maximum roll change after a teleoperation
• enable_roll [bool] [default=false]: Parameter for “box” mode of step 1. Flag to enable rotation (roll)
• min_pitch [double] [default=0.0]: Parameter for “box” mode of step 1. Minimum pitch change after a teleoperation
• max_pitch [double] [default=TwoPi<double>]: Parameter for “box” mode of step 1. Maximum pitch change after a teleoperation
• enable_pitch [bool] [default=false]: Parameter for “box” mode of step 1. Flag to enable rotation (pitch)
• min_yaw [double] [default=0.0]: Parameter for “box” mode of step 1. Minimum yaw change after a teleoperation
• max_yaw [double] [default=TwoPi<double>]: Parameter for “box” mode of step 1. Minimum yaw change after a teleoperation
• enable_yaw [bool] [default=false]: Parameter for “box” mode of step 1. Flag to enable rotation (yaw)
• spline_distance [double] [default=0.02]: Parameter for “spline” mode of step 1. We will travel for this fraction of the spline distance before uniformly randomly sampling a new point on the spline again.
• spline_speed [double] [default=0.005]: Parameter for “spline” mode of step 1. Speed of travel. Unit is fraction of spline length per second. Negative speed corresponds to driving backwards.
• spline_flip_probability [double] [default=0.5]: Parameter for “spline” mode of step 1. With this probability, the direction of the tangent will be flipped.
• translation_standard_deviation [Vector3d] [default=Vector3d::Zero()]: Parameter for step 2. A noise for the translation with this standard deviation will be applied.
• roll_pitch_yaw_standard_deviation [Vector3d] [default=Vector3d::Zero()]: Parameter for step 2. A noise for the angles with this standard deviation will be applied.
• offset_pose [Pose3d] [default=Pose3d::Identity()]: Parameter for step 3. Offset pose to be applied to the combined pose.

isaac.ml.TensorArgMax

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• input [TensorProto]: Input rank-3 tensor with dimensions (rows, columns, channels).

Outgoing messages

• argmax [TensorProto]: Output rank-2 tensor with dimensions (rows, columns) which stores the index of the max channel.

Parameters

• threshold [double] [default=0.5]: Threshold which defines if the argmax channel index for a pixel will be assigned to the 2 dimensional tensor. If the value at the argmax index is less than this threshold, the corresponding element in the discretized tensor is assigned the unknown class index.
• non_max_index [int] [default=-1]: Value to be assigned for the unknown class.

isaac.ml.TensorChannelSum

Description

Type: Codelet - This component ticks either periodically or when it receives messages.

Incoming messages

• tensor [TensorProto]: Input tensor having dimensions (rows, columns, channels)

Outgoing messages