Get Started with NuRec: For Autonomous Vehicles#

For developers working on autonomous vehicle software, NVIDIA Omniverse NuRec is integrated with a few different simulation platforms. To use NuRec in an AV simulation platform, you can get the pre-reconstructed scenes from the NVIDIA Physical AI dataset on HuggingFace and then use them on a simulation platform that has built a NuRec integration, like AlpaSim or CARLA.

Step 1: Set Up#

Check that you have the right hardware and software setup to run NuRec.

Step 2: Get Compatible Data#

The NCore data format is a standardized format for sensor data recordings from various sources, including camera and lidar sensors. NuRec requires input data to be in the NCore format.

You can get the pre-reconstructed scenes from the NVIDIA Nurec-trained Physical AI dataset on HuggingFace and then use them on a simulation platform that has built a NuRec integration, like AlpaSim or CARLA.

If you’d like to reconstruct the scenes yourself, you can use pre-converted sample data that is already in the NCore format from the NVIDIA NCore-converted Physical AI dataset on HuggingFace, then Generate Auxiliary Data required by NuRec, and use that to generate NuRec-trained scenes. For reference, this NCore-formatted sample data leverages sample data from the NVIDIA Physical AI raw dataset converted using the NVIDIA Physical AI Data Converter.

Step 3: Generate Reconstructed Scenes#

Reconstruction converts real-world data into a 3D scene, output as a USDZ file. To create these reconstructed scenes, use your NCore-formatted data and follow the steps outlined in Use the NuRec Model for Autonomous Vehicles.

If you’re using the pre-reconstructed dataset, skip to the rendering step.

Step 4: Render Reconstructions in a Simulation#

To render reconstructed scenes, you can use existing NuRec-powered simulation platforms like AlpaSim or CARLA:

You can also use the NuRec gRPC API in your own simulation platform.