NVIDIA Neural Reconstruction (NuRec)#
The NVIDIA Neural Reconstruction (NuRec) models and services support the seamless ingestion of real-world camera and lidar data to create a simulated 3D environment. These simulated scenarios can be used to train and test Physical AI Agents, including robotics and autonomous driving systems
How NuRec Works#
Reconstruction and rendering with NuRec follows a series of steps:
Converting real-world data to a NuRec-compatible format: NuRec requires a specific format, called the NCore data format, to ingest real-world data. Additionally, a set of auxiliary data generated from the NCore data sets serves as pre-processed input for NuRec. The NCore container includes data quality validation tools to ensure the data is ready for NuRec.
Reconstructing scenarios as 3D scenes: The reconstruction models and services for NuRec are available as a core NuRec container from NVIDIA. When you run NuRec, it converts real-world data into a 3D scene and saves the output as a USDZ file on the local host system. The USDZ file includes a Universal Scene Description (USD) file that defines the scene, a checkpoint file that includes the AI-trained reconstruction data, and a JSON file that includes the sequence track and rig trajectories. An OpenDrive (XODR) map might also be included in the USDZ package for use in simulation tasks like traffic modeling and scenario editing, but it is not required to run NuRec.
Rendering the 3D scenes: NuRec offers a gRPC API and is built into Omniverse Kit for seamless integration with simulation platforms. The NuRec gRPC API serves as a conduit of data and rendering between the simulation runtime and the NuRec container, where the scenes are reconstructed and rendered. Users of Omniverse Kit apps like Isaac Sim can load a compatible scene in the app and control rendering through the
OmniNuRecVolumeAPIproperties.Refining the rendered images: The Fixer model refines the rendered images to improve the quality of the rendered images during reconstruction and rendering.
Tip
To streamline the pipeline from real-world data to a NuRec-compatible rendered scene, NVIDIA offers a growing collection of pre-reconstructed scenes in the NVIDIA Physical AI dataset on HuggingFace.
How to Use NuRec#
To use NuRec in an AV or robotics simulation platform, you can get the pre-reconstructed scenes from the NVIDIA Physical AI dataset on HuggingFace and then use them on a simulation platform that has built a NuRec integration.
For developers working on autonomous robotics, rendering through NuRec is built into Omniverse,
so you can readily use Isaac Sim as a simulation platform. Isaac Sim supports rendering of Neural Radiance Fields (NeRFs),
3D Gaussian Splats (3DGS), and 3D Gaussian Unscented Transforms (3DGUT) converted to USDZ outputs using the open-source
3DGRUT project. Once you’ve generated the USDZ file, you can load
it and use the OmniNuRecVolumeAPI schema to render the scene.
For developers working on autonomous vehicle software, the open-source CARLA simulator supports NuRec. When you run the NuRec launcher script, it loads a curated set of pre-reconstructed scenes from the NVIDIA Physical AI dataset and renders them in the CARLA simulation replay through an integration with the NuRec gRPC API.
Learn More#
Read more about the technology powering NuRec in the relevant research papers fromm NVIDIA Research: