NuRec gRPC API: sensorsim#

SensorsimService defines the RPC interface for interacting with the sensor simulator. It provides methods for rendering RGB images and lidar data, querying available scenes, cameras, trajectories, and ego masks, as well as shutting down the simulator.

rpc `batch_render_rgb`

Request: sensorsim.BatchRGBRenderRequest
Response: sensorsim.BatchRGBRenderReturn

Deprecated: use batch_render_camera instead. Render a batch of RGB requests.

rpc `create_harmonizer_session`

Request: common.Empty
Response: sensorsim.HarmonizerSessionId

Open a temporal-harmonizer session holding the frame history for ONE coherent render sequence (a single trajectory). Use one session per sequence — reusing it across separate or interleaved sequences corrupts the temporal conditioning. Send the session's frames in trajectory order, one at a time (await each render before sending the next); concurrent or out-of-order requests build the history wrongly and are not rejected. Returns FAILED_PRECONDITION unless the server runs --enable-harmonizer-temporal.

rpc `end_harmonizer_session`

Request: sensorsim.HarmonizerSessionId
Response: common.Empty

End a temporal Harmonizer session and release its stored state.

rpc `get_external_asset_objects`

Request: sensorsim.ExternalAssetObjectsRequest
Response: sensorsim.ExternalAssetObjectsReturn

Return the external asset objects available in a scene.

rpc `edit_assets`

Request: sensorsim.EditAssetsRequest
Response: sensorsim.EditAssetsResponse

Apply the requested asset replacements and insertions to a scene.

rpc `restore_model_parameters`

Request: sensorsim.RestoreModelParametersRequest
Response: common.Empty

Restore the original model parameters after asset editing.

rpc `get_dynamic_objects`

Request: sensorsim.AvailableDynamicObjectsRequest
Response: sensorsim.AvailableDynamicObjectsReturn

Return dynamic objects and their trajectories for a scene.

rpc `get_server_config`

Request: common.Empty
Response: sensorsim.ServerConfig

Return the running server configuration.

rpc `open_ipc_session`

Request: ipc.OpenIpcSessionRequest
Response: ipc.OpenIpcSessionResponse

CUDA IPC session lifecycle. After open_ipc_session, render_rgb may be called with use_cuda_ipc=true to deliver the payload into one of the server-owned, pre-allocated buffers exported as cudaIpcMemHandle_t bytes in the open response.

rpc `close_ipc_session`

Request: ipc.CloseIpcSessionRequest
Response: ipc.CloseIpcSessionResponse

Close a same-node CUDA IPC session and release its server-owned buffers.

rpc `render_rgb`

Request:

sensorsim.RGBRenderRequest

Response:

sensorsim.RGBRenderReturn

Deprecated: use render_camera instead. Renders an RGB image based on the provided request parameters.

rpc `render_camera`

Request:

sensorsim.CameraRenderRequest

Response:

sensorsim.CameraRenderReturn

rpc `batch_render_camera`

Request:

sensorsim.BatchCameraRenderRequest

Response:

sensorsim.BatchCameraRenderReturn

rpc `render_lidar`

Request:

sensorsim.LidarRenderRequest

Response:

sensorsim.LidarRenderReturn

Renders lidar data based on the provided request parameters.

rpc `get_version`

Request:

common.Empty

Response:

common.VersionId

Returns the version of the sensor simulator.

rpc `get_available_scenes`

Request:

common.Empty

Response:

common.AvailableScenesReturn

Returns the list of available scenes.

rpc `get_loaded_scenes`

Request:

common.Empty

Response:

sensorsim.LoadedScenesReturn

Returns cached / currently resident scene instance counts from BackendCache. Unlike get_available_scenes, scenes may appear with loaded_instance_count > 1 when multiple cached instances exist for the same scene.

rpc `get_available_cameras`

Request:

sensorsim.AvailableCamerasRequest

Response:

sensorsim.AvailableCamerasReturn

Returns the list of available cameras for a given scene.

rpc `shut_down`

Request:

common.Empty

Response:

common.Empty

Shuts down the sensor simulator.

rpc `get_available_trajectories`

Request:

sensorsim.AvailableTrajectoriesRequest

Response:

sensorsim.AvailableTrajectoriesReturn

Returns the list of available trajectories for a given scene.

rpc `get_available_ego_masks`

Request:

common.Empty

Response:

sensorsim.AvailableEgoMasksReturn

Returns the list of available ego masks.

Messages

message AvailableCamera (Nested in sensorsim.AvailableCamerasReturn )

Metadata for a single available camera.

	Field	Type	Description
1	intrinsics	`sensorsim.CameraSpec`	Camera intrinsic and extrinsic parameters.
2	rig_to_camera	`common.Pose`	The pose from the rig to the camera.
3	logical_id	`string`	Logical ID for the camera.
4	trajectory_idx	`uint32`	Index of the trajectory associated with this camera.

message AvailableCamerasRequest

AvailableCamerasRequest specifies the scene for which available cameras are requested.

	Field	Type	Description
1	scene_id	`string`	The scene ID for which to list available cameras.

message AvailableCamerasReturn

AvailableCamerasReturn contains the list of available cameras for a scene.

	Field	Type	Description
1	available_cameras	`repeated sensorsim.AvailableCamerasReturn.AvailableCamera`	List of available cameras.

message AvailableEgoMasksReturn

AvailableEgoMasksReturn contains metadata for all available ego masks.

	Field	Type	Description
1	ego_mask_metadata	`repeated sensorsim.AvailableEgoMasksReturn.EgoMaskMetadata`	List of available ego mask metadata.

message AvailableTrajectoriesRequest

AvailableTrajectoriesRequest specifies the scene for which available trajectories are requested.

	Field	Type	Description
1	scene_id	`string`	The scene ID for which to list available trajectories.

message AvailableTrajectoriesReturn

AvailableTrajectoriesReturn contains the list of available trajectories for a scene.

	Field	Type	Description
1	available_trajectories	`repeated sensorsim.AvailableTrajectoriesReturn.AvailableTrajectory`	List of available trajectories.

message AvailableTrajectory (Nested in sensorsim.AvailableTrajectoriesReturn )

Metadata for a single available trajectory.

	Field	Type	Description
1	trajectory_idx	`uint32`	Index of the trajectory.
2	trajectory	`common.Trajectory`	The trajectory data.

message BatchCameraRenderRequest

	Field	Type	Description
1	items	`repeated sensorsim.BatchCameraRenderRequestItem`

message BatchCameraRenderRequestItem

	Field	Type	Description
1	camera_name	`string`	Identifier to match response
2	request	`sensorsim.CameraRenderRequest`

message BatchCameraRenderReturn

Batch camera render response

	Field	Type	Description
1	items	`repeated sensorsim.BatchCameraRenderReturnItem`

message BatchCameraRenderReturnItem

	Field	Type	Description
1	camera_name	`string`	Matches request camera_name
2	result	`sensorsim.CameraRenderReturn`
3	success	`bool`
4	error_message	`string`

message BboxColor

RGB color for bounding box wireframe edges. Unset = server default (0, 255, 0). Each channel is 0-255; values outside this range are rejected with INVALID_ARGUMENT.

	Field	Type
1	r	`uint32`
2	g	`uint32`
3	b	`uint32`

message BivariateWindshieldModelParameters

BivariateWindshieldModelParameters describes the parameters for a bivariate polynomial windshield distortion model.

	Field	Type	Description
1	reference_poly	`sensorsim.BivariateWindshieldModelParameters.ReferencePolynomial`	The reference polynomial type.
2	horizontal_poly	`repeated double`	Coefficients for the horizontal polynomial.
3	vertical_poly	`repeated double`	Coefficients for the vertical polynomial.
4	horizontal_poly_inverse	`repeated double`	Coefficients for the inverse horizontal polynomial.
5	vertical_poly_inverse	`repeated double`	Coefficients for the inverse vertical polynomial.

message CameraRenderOutput

	Field	Type	Description
1	image_bytes	`bytes`
2	image_height	`uint32`
3	image_width	`uint32`
4	render_product	`sensorsim.CameraRenderProduct`	Which product this buffer carries. Matches the corresponding entry in the request's render_products list (or COLOR for an empty request list).
5	semantics_class_mapping	`repeated sensorsim.SemanticsClassEntry`	when output is semantics
6	distance_colormap_vmax_m	`optional float`	when output is distance

message CameraRenderRequest

	Field	Type	Description
1	scene_id	`string`
2	resolution_h	`uint32`
3	resolution_w	`uint32`
4	camera_intrinsics	`sensorsim.CameraSpec`
5	frame_start_us	`fixed64`
6	frame_end_us	`fixed64`
7	sensor_pose	`sensorsim.PosePair`
8	dynamic_objects	`repeated sensorsim.DynamicObject`
9	image_format	`sensorsim.ImageFormat`
10	image_quality	`float`
11	insert_ego_mask	`bool`
12	ego_mask_id	`sensorsim.EgoMaskId`	if `insert_ego_mask` is true, `ego_mask_id` has to correspond to one of the ego masks in the AvailableEgoMasksReturn
13	render_products	`repeated sensorsim.CameraRenderProduct`	Image products to render. Empty list defaults to a single COLOR output. Entries must be unique; the server returns one CameraRenderOutput per requested product in the same order.
14	overlay_bounding_boxes	`sensorsim.OverlayBoundingBoxSource`	Draw 3D cuboid wireframes on every requested output (color, distance, semantics); selects the source of tracks. Set to UNSPECIFIED to disable.
15	distance_colormap_vmax_m	`optional float`	Upper bound of the distance colormap (meters). Unset = server default (80 m).
16	bbox_color	`optional sensorsim.BboxColor`	RGB color for bounding box wireframe edges. Unset = server default (0, 255, 0).
17	harmonizer_session_id	`optional string`	On a server with temporal Harmonizer enabled, this field is required when a colour request sets apply_harmonizer true; the render uses the session's HarmonizerState. When apply_harmonizer is false or unset, supplying this field resets that session's history so a later opt-in starts a fresh coherent sequence rather than conditioning on stale frames. If the server is non-temporal, this field is silently ignored. The id must identify a single coherent render sequence (see create_harmonizer_session). Only meaningful for the CAMERA_RENDER_PRODUCT_COLOR product.
21	apply_harmonizer	`optional bool`	Request Harmonizer postprocessing for this render. Unset or false leaves the color output unharmonized. On a temporal server, apply_harmonizer=true requires harmonizer_session_id. When apply_harmonizer is false or unset, a supplied ID resets that session; without an ID, no temporal session state changes. If apply_harmonizer is true and the server was not started with a Harmonizer mode enabled, the request fails with FAILED_PRECONDITION. Only meaningful for CAMERA_RENDER_PRODUCT_COLOR.

message CameraRenderReturn

	Field	Type	Description
1	camera_render_outputs	`repeated sensorsim.CameraRenderOutput`

message CameraSpec

CameraSpec describes the full specification of a camera, including intrinsics, extrinsics, and distortion models.

	Field	Type	Description
1	temporary_camera_spec	`string`	Deprecated: Temporary camera spec string.
5	logical_id	`string`	Logical ID for the camera (user-defined).
6	trajectory_idx	`uint32`	Index of the trajectory associated with this camera.
7	resolution_h	`uint32`	Image height in pixels.
8	resolution_w	`uint32`	Image width in pixels.
9	shutter_type	`sensorsim.ShutterType`	The shutter type for the camera.

oneof external_distortion
10	bivariate_windshield_model_param	`sensorsim.BivariateWindshieldModelParameters`	Bivariate windshield model parameters.

oneof camera_param
2	ftheta_param	`sensorsim.FthetaCameraParam`	F-theta camera parameters.
3	opencv_pinhole_param	`sensorsim.OpenCVPinholeCameraParam`	OpenCV pinhole camera parameters.
4	opencv_fisheye_param	`sensorsim.OpenCVFisheyeCameraParam`	OpenCV fisheye camera parameters.

message DynamicObject

DynamicObject represents a dynamic object in the scene with a unique track ID and pose pair.

	Field	Type	Description
1	track_id	`string`	Unique identifier for the dynamic object.
2	pose_pair	`sensorsim.PosePair`	The pose pair for the dynamic object.

message EgoMaskId

EgoMaskId uniquely identifies an ego mask by camera and rig configuration.

	Field	Type	Description
1	camera_logical_id	`string`	The logical ID of the camera (e.g., camera_front_wide_120fov).
2	rig_config_id	`string`	The rig configuration ID (e.g., hyperion 8.0, 8.1).

message EgoMaskMetadata (Nested in sensorsim.AvailableEgoMasksReturn )

Metadata for a single ego mask.

Field Type Description

1

ego_mask_id


          
          sensorsim.EgoMaskId

The unique identifier for the ego mask.

Future: extra metadata like nominal camera_to_rig pose.

message FthetaCameraParam

FthetaCameraParam describes the parameters for an F-theta camera model.

	Field	Type	Description
1	principal_point_x	`double`	The x-coordinate of the principal point in pixels.
2	principal_point_y	`double`	The y-coordinate of the principal point in pixels.
3	reference_poly	`sensorsim.FthetaCameraParam.PolynomialType`	The reference polynomial type for the model.
4	pixeldist_to_angle_poly	`repeated double`	Coefficients for the pixel distance to angle polynomial.
5	angle_to_pixeldist_poly	`repeated double`	Coefficients for the angle to pixel distance polynomial.
6	max_angle	`double`	The maximum angle (in radians or degrees, as defined by the model).

message LidarRenderRequest

LidarRenderRequest specifies the parameters for rendering lidar data.

	Field	Type	Description
1	scene_id	`string`	The scene ID to render.
2	lidar_config	`sensorsim.LidarSpec`	Lidar configuration parameters.
3	frame_start_us	`fixed64`	Start time of the frame in microseconds.
4	frame_end_us	`fixed64`	End time of the frame in microseconds.
5	sensor_pose	`sensorsim.PosePair`	The pose of the sensor for this frame.
6	dynamic_objects	`repeated sensorsim.DynamicObject`	List of dynamic objects present in the scene.

message LidarRenderReturn

LidarRenderReturn contains the rendered lidar point cloud and intensities.

	Field	Type	Description
1	point_xyzs	`repeated float`	The xyz coordinates of the lidar points in the end-of-spin lidar space. [x1, y1, z1, x2, y2, z2, ...]
2	point_intensities	`repeated float`	The intensity of the points (range [0, 1]).

message LidarSpec

LidarSpec describes the configuration of a lidar device.

	Field	Type	Description
2	lidar_type	`sensorsim.LidarDeviceType`	The type of lidar device to use.

message LoadedSceneEntry

Per-scene cached backend instance counts.

	Field	Type	Description
1	scene_id	`string`	Scene ID for this group of cached instances.
2	loaded_instance_count	`uint64`	Cached instances for this scene: active + reusable.
3	reusable_instance_count	`uint64`	Spare instances for this scene available for checkout without loading.

message LoadedScenesReturn

Snapshot of cached scene instances and global cache capacity.

	Field	Type	Description
1	scenes	`repeated sensorsim.LoadedSceneEntry`	Per-scene counts, sorted by scene_id.
2	loaded_instance_capacity	`uint64`	Global cached-instance capacity from BackendCache.maxsize.

message OpenCVFisheyeCameraParam

OpenCVFisheyeCameraParam describes the parameters for an OpenCV fisheye camera model.

	Field	Type	Description
1	principal_point_x	`double`	The x-coordinate of the principal point in pixels.
2	principal_point_y	`double`	The y-coordinate of the principal point in pixels.
3	focal_length_x	`double`	The x-component of the focal length in pixels.
4	focal_length_y	`double`	The y-component of the focal length in pixels.
5	radial_coeffs	`repeated double`	Radial distortion coefficients.
6	max_angle	`double`	The maximum angle (in radians or degrees, as defined by the model).

message OpenCVPinholeCameraParam

OpenCVPinholeCameraParam describes the parameters for an OpenCV pinhole camera model.

	Field	Type	Description

message PosePair

PosePair represents the poses at the start and end of a frame.

	Field	Type	Description
1	start_pose	`common.Pose`	The pose at the start of the frame.
2	end_pose	`common.Pose`	The pose at the end of the frame.

message RGBRenderRequest

Deprecated: use CameraRenderRequest instead. RGBRenderRequest specifies the parameters for rendering an RGB image.

	Field	Type	Description
1	scene_id	`string`	The scene ID to render.
2	resolution_h	`uint32`	Image height in pixels.
3	resolution_w	`uint32`	Image width in pixels.
4	camera_intrinsics	`sensorsim.CameraSpec`	Camera intrinsic and extrinsic parameters.
5	frame_start_us	`fixed64`	Start time of the frame in microseconds.
6	frame_end_us	`fixed64`	End time of the frame in microseconds.
7	sensor_pose	`sensorsim.PosePair`	The pose of the sensor for this frame.
8	dynamic_objects	`repeated sensorsim.DynamicObject`	List of dynamic objects present in the scene.
9	image_format	`sensorsim.ImageFormat`	The image format to use for the output.
10	image_quality	`float`	The quality of the output image (0.0-1.0 for lossy formats).
11	insert_ego_mask	`bool`	If false, no ego mask will be applied.
12	ego_mask_id	`sensorsim.EgoMaskId`	If `insert_ego_mask` is true, this specifies which ego mask to use.
20	harmonizer_session_id	`optional string`	On a server with temporal Harmonizer enabled, this field is required when apply_harmonizer is true; the render uses the session's HarmonizerState. When apply_harmonizer is false or unset, supplying this field resets that session's history so a later opt-in starts a fresh coherent sequence rather than conditioning on stale frames. If the server is non-temporal, this field is silently ignored. The id must identify a single coherent render sequence (see create_harmonizer_session).
21	apply_harmonizer	`optional bool`	Request Harmonizer postprocessing for this render. Unset or false leaves the color output unharmonized. On a temporal server, apply_harmonizer=true requires harmonizer_session_id. When apply_harmonizer is false or unset, a supplied ID resets that session; without an ID, no temporal session state changes. If apply_harmonizer is true and the server was not started with a Harmonizer mode enabled, the request fails with FAILED_PRECONDITION.

message RGBRenderReturn

Deprecated: use CameraRenderReturn instead. RGBRenderReturn contains the rendered RGB image bytes.

	Field	Type	Description
1	image_bytes	`bytes`	The rendered image bytes (encoded as specified in the request).

message SemanticsClassEntry

Per-class metadata when render_product is CAMERA_RENDER_PRODUCT_SEMANTICS_COLORIZED (index, name, display color 0-255)

	Field	Type
1	class_index	`int32`
2	class_name	`string`
3	color_r	`uint32`
4	color_g	`uint32`
5	color_b	`uint32`

Enums

enum CameraRenderProduct

Which image product to render and return. COLOR is the zero/default value, so a default-constructed CameraRenderProduct is a legal (and natural) request.

Name	Number	Description
`CAMERA_RENDER_PRODUCT_COLOR`	0
`CAMERA_RENDER_PRODUCT_DISTANCE_COLORIZED`	1	metric ray distance (\|\|point - camera\|\| in meters), turbo colormap
`CAMERA_RENDER_PRODUCT_SEMANTICS_COLORIZED`	2	semantic class colors

enum ImageFormat

ImageFormat enumerates the supported image formats for RGB rendering.

Name	Number	Description
`UNDEFINED`	0
`PNG`	1
`JPEG`	2
`JPEG2000`	3	AlpaSim currently only supports PNG and JPEG.
`RGB_UINT8_PLANAR`	4
`AVC`	5
`AV1`	6

enum LidarDeviceType

LidarDeviceType enumerates the supported lidar device types.

Name	Number	Description
`PANDAR128`	0
`AT128`	1

enum OverlayBoundingBoxSource

Source of tracks for overlaying bounding boxes on output camera images UNSPECIFIED = no bbox overlay (default) MODEL = Learned/calibrated tracks in the checkpoint. ARTIFACT = sequence_tracks.json in the USDZ.

Name	Number	Description
`OVERLAY_BOUNDING_BOX_SOURCE_UNSPECIFIED`	0
`OVERLAY_BOUNDING_BOX_SOURCE_MODEL`	1
`OVERLAY_BOUNDING_BOX_SOURCE_ARTIFACT`	2

enum PolynomialType

The type of polynomial used for the F-theta model.

Name	Number	Description
`UNKNOWN`	0	Unknown polynomial type
`PIXELDIST_TO_ANGLE`	1	Polynomial mapping pixeldistances-to-angles ("backward" polynomial)
`ANGLE_TO_PIXELDIST`	2	Polynomial mapping angles-to-pixeldistances ("forward" polynomial)

enum ReferencePolynomial

The reference polynomial direction for the model.

Name	Number	Description
`FORWARD`	0
`BACKWARD`	1

enum ShutterType

ShutterType enumerates the possible shutter types for a camera sensor.

Name	Number	Description
`UNKNOWN`	0	Unknown shutter type
`ROLLING_TOP_TO_BOTTOM`	1	Rolling shutter from top to bottom of the imager
`ROLLING_LEFT_TO_RIGHT`	2	Rolling shutter from left to right of the imager
`ROLLING_BOTTOM_TO_TOP`	3	Rolling shutter from bottom to top of the imager
`ROLLING_RIGHT_TO_LEFT`	4	Rolling shutter from right to left of the imager
`GLOBAL`	5	Instantaneous global shutter (no rolling shutter)