> This page is for version Release 2.3. > For other versions, use one of these documentation indexes: > - Release 3.0 (default): https://docs.nvidia.com/isaaclab/3.0/llms.txt > - Release 2.3: https://docs.nvidia.com/isaaclab/2.3/llms.txt > - Release 2.2: https://docs.nvidia.com/isaaclab/2.2/llms.txt > - Release 2.1: https://docs.nvidia.com/isaaclab/2.1/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/isaaclab/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/isaaclab/_mcp/server. # Ray Caster ![A diagram outlining the basic geometry of frame transformations](/isaaclab/_fern-img/8bcea51655c8e4c16740901a721e01927bab0e70d31695460d908374e2d08ba8.webp) The Ray Caster sensor (and the ray caster camera) are similar to RTX based rendering in that they both involve casting rays. The difference here is that the rays cast by the Ray Caster sensor return strictly collision information along the cast, and the direction of each individual ray can be specified. They do not bounce, nor are they affected by things like materials or opacity. For each ray specified by the sensor, a line is traced along the path of the ray and the location of first collision with the specified mesh is returned. This is the method used by some of our quadruped examples to measure the local height field. To keep the sensor performant when there are many cloned environments, the line tracing is done directly in [Warp](https://nvidia.github.io/warp/). This is the reason why specific meshes need to be identified to cast against: that mesh data is loaded onto the device by warp when the sensor is initialized. As a consequence, the current iteration of this sensor only works for literally static meshes (meshes that *are not changed from the defaults specified in their USD file*). This constraint will be removed in future releases. Using a ray caster sensor requires a **pattern** and a parent xform to be attached to. The pattern defines how the rays are cast, while the prim properties defines the orientation and position of the sensor (additional offsets can be specified for more exact placement). Isaac Lab supports a number of ray casting pattern configurations, including a generic LIDAR and grid pattern. ```python @configclass class RaycasterSensorSceneCfg(InteractiveSceneCfg): """Design the scene with sensors on the robot.""" # ground plane ground = AssetBaseCfg( prim_path="/World/Ground", spawn=sim_utils.UsdFileCfg( usd_path=f"{ISAAC_NUCLEUS_DIR}/Environments/Terrains/rough_plane.usd", scale=(1, 1, 1), ), ) # lights dome_light = AssetBaseCfg( prim_path="/World/Light", spawn=sim_utils.DomeLightCfg(intensity=3000.0, color=(0.75, 0.75, 0.75)) ) # robot robot = ANYMAL_C_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot") ray_caster = RayCasterCfg( prim_path="{ENV_REGEX_NS}/Robot/base/lidar_cage", update_period=1 / 60, offset=RayCasterCfg.OffsetCfg(pos=(0, 0, 0.5)), mesh_prim_paths=["/World/Ground"], ray_alignment="yaw", pattern_cfg=patterns.LidarPatternCfg( channels=100, vertical_fov_range=[-90, 90], horizontal_fov_range=[-90, 90], horizontal_res=1.0 ), debug_vis=not args_cli.headless, ) ``` Notice that the units on the pattern config is in degrees! Also, we enable visualization here to explicitly show the pattern in the rendering, but this is not required and should be disabled for performance tuning. ![Lidar Pattern visualized](/isaaclab/_fern-img/f41e26ed780a6db71caf0deb543024f1c94e0ec8fa25a6f44b65910866c36cdc.webp) Querying the sensor for data can be done at simulation run time like any other sensor. ```python def run_simulator(sim: sim_utils.SimulationContext, scene: InteractiveScene): . . . # Simulate physics while simulation_app.is_running(): . . . # print information from the sensors print("-------------------------------") print(scene["ray_caster"]) print("Ray cast hit results: ", scene["ray_caster"].data.ray_hits_w) ``` ```bash ------------------------------- Ray-caster @ '/World/envs/env_.*/Robot/base/lidar_cage': view type : update period (s) : 0.016666666666666666 number of meshes : 1 number of sensors : 1 number of rays/sensor: 18000 total number of rays : 18000 Ray cast hit results: tensor([[[-0.3698, 0.0357, 0.0000], [-0.3698, 0.0357, 0.0000], [-0.3698, 0.0357, 0.0000], ..., [ inf, inf, inf], [ inf, inf, inf], [ inf, inf, inf]]], device='cuda:0') ------------------------------- ``` Here we can see the data returned by the sensor itself. Notice first that there are 3 closed brackets at the beginning and the end: this is because the data returned is batched by the number of sensors. The ray cast pattern itself has also been flattened, and so the dimensions of the array are `[N, B, 3]` where `N` is the number of sensors, `B` is the number of cast rays in the pattern, and 3 is the dimension of the casting space. Finally, notice that the first several values in this casting pattern are the same: this is because the lidar pattern is spherical and we have specified our FOV to be hemispherical, which includes the poles. In this configuration, the "flattening pattern" becomes apparent: the first 180 entries will be the same because it's the bottom pole of this hemisphere, and there will be 180 of them because our horizontal FOV is 180 degrees with a resolution of 1 degree. You can use this script to experiment with pattern configurations and build an intuition about how the data is stored by altering the `triggered` variable on line 81. ```python # Copyright (c) 2022-2025, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md). # All rights reserved. # # SPDX-License-Identifier: BSD-3-Clause import argparse from isaaclab.app import AppLauncher # add argparse arguments parser = argparse.ArgumentParser(description="Example on using the raycaster sensor.") parser.add_argument("--num_envs", type=int, default=1, help="Number of environments to spawn.") # append AppLauncher cli args AppLauncher.add_app_launcher_args(parser) # parse the arguments args_cli = parser.parse_args() # launch omniverse app app_launcher = AppLauncher(args_cli) simulation_app = app_launcher.app """Rest everything follows.""" import numpy as np import torch import isaaclab.sim as sim_utils from isaaclab.assets import AssetBaseCfg from isaaclab.scene import InteractiveScene, InteractiveSceneCfg from isaaclab.sensors.ray_caster import RayCasterCfg, patterns from isaaclab.utils import configclass from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR ## # Pre-defined configs ## from isaaclab_assets.robots.anymal import ANYMAL_C_CFG # isort: skip @configclass class RaycasterSensorSceneCfg(InteractiveSceneCfg): """Design the scene with sensors on the robot.""" # ground plane ground = AssetBaseCfg( prim_path="/World/Ground", spawn=sim_utils.UsdFileCfg( usd_path=f"{ISAAC_NUCLEUS_DIR}/Environments/Terrains/rough_plane.usd", scale=(1, 1, 1), ), ) # lights dome_light = AssetBaseCfg( prim_path="/World/Light", spawn=sim_utils.DomeLightCfg(intensity=3000.0, color=(0.75, 0.75, 0.75)) ) # robot robot = ANYMAL_C_CFG.replace(prim_path="{ENV_REGEX_NS}/Robot") ray_caster = RayCasterCfg( prim_path="{ENV_REGEX_NS}/Robot/base/lidar_cage", update_period=1 / 60, offset=RayCasterCfg.OffsetCfg(pos=(0, 0, 0.5)), mesh_prim_paths=["/World/Ground"], ray_alignment="yaw", pattern_cfg=patterns.LidarPatternCfg( channels=100, vertical_fov_range=[-90, 90], horizontal_fov_range=[-90, 90], horizontal_res=1.0 ), debug_vis=not args_cli.headless, ) def run_simulator(sim: sim_utils.SimulationContext, scene: InteractiveScene): """Run the simulator.""" # Define simulation stepping sim_dt = sim.get_physics_dt() sim_time = 0.0 count = 0 triggered = True countdown = 42 # Simulate physics while simulation_app.is_running(): if count % 500 == 0: # reset counter count = 0 # reset the scene entities # root state # we offset the root state by the origin since the states are written in simulation world frame # if this is not done, then the robots will be spawned at the (0, 0, 0) of the simulation world root_state = scene["robot"].data.default_root_state.clone() root_state[:, :3] += scene.env_origins scene["robot"].write_root_pose_to_sim(root_state[:, :7]) scene["robot"].write_root_velocity_to_sim(root_state[:, 7:]) # set joint positions with some noise joint_pos, joint_vel = ( scene["robot"].data.default_joint_pos.clone(), scene["robot"].data.default_joint_vel.clone(), ) joint_pos += torch.rand_like(joint_pos) * 0.1 scene["robot"].write_joint_state_to_sim(joint_pos, joint_vel) # clear internal buffers scene.reset() print("[INFO]: Resetting robot state...") # Apply default actions to the robot # -- generate actions/commands targets = scene["robot"].data.default_joint_pos # -- apply action to the robot scene["robot"].set_joint_position_target(targets) # -- write data to sim scene.write_data_to_sim() # perform step sim.step() # update sim-time sim_time += sim_dt count += 1 # update buffers scene.update(sim_dt) # print information from the sensors print("-------------------------------") print(scene["ray_caster"]) print("Ray cast hit results: ", scene["ray_caster"].data.ray_hits_w) if not triggered: if countdown > 0: countdown -= 1 continue data = scene["ray_caster"].data.ray_hits_w.cpu().numpy() np.save("cast_data.npy", data) triggered = True else: continue def main(): """Main function.""" # Initialize the simulation context sim_cfg = sim_utils.SimulationCfg(dt=0.005, device=args_cli.device) sim = sim_utils.SimulationContext(sim_cfg) # Set main camera sim.set_camera_view(eye=[3.5, 3.5, 3.5], target=[0.0, 0.0, 0.0]) # design scene scene_cfg = RaycasterSensorSceneCfg(num_envs=args_cli.num_envs, env_spacing=2.0) scene = InteractiveScene(scene_cfg) # Play the simulator sim.reset() # Now we are ready! print("[INFO]: Setup complete...") # Run the simulator run_simulator(sim, scene) if __name__ == "__main__": # run the main function main() # close sim app simulation_app.close() ```