How NuRec Works#
NVIDIA Omniverse NuRec refers to the neural reconstruction and rendering models and services from NVIDIA that support the seamless ingestion of real-world camera and lidar data to create a simulated 3D environment suitable for training and testing Physical AI Agents, including robotics and autonomous driving systems.
Reconstruction and Rendering with NuRec#
NuRec encompasses multiple sub-components that work together to provide the core NuRec service.
NuRec overview
Data Compatibility with NuRec#
The NCore data format is a standardized format for sensor data recordings from various sources, including camera and lidar sensors. NuRec also requires an auxiliary dataset to reconstruct scenes when you convert and input your own real-world data. To use your own real-world data, you would need to convert it to the NCore format and then generate auxiliary data for NuRec.
If you want to use NuRec with a sample dataset, explore the options in Prepare Data for Use with NuRec. To learn more about NCore, see the NCore documentation.
Reconstruction#
Reconstruction converts real-world data into a 3D scene, output as a USDZ file. The reconstruction engine is inside the main NuRec container. The USDZ file is a zipped package that includes the following files:
XODR file: A driveable map for use in the simulation.
USDA files: Default file that defines the scene using Universal Scene Description (USD) and includes mapping, domelight, sequence tracks (cuboid tracks), and rig trajectories (logs of all trajectories).
Checkpoint: This is the actual AI-trained reconstruction data, including GS positions, auxiliary data.
JSON: Sequence track and rig trajectories are also available in this format.
Asset Harvester is a system of 5 models (Mask2Former, C-RADIO, Sana Multiview, LGM, Fixer) that converts actors and objects from the dataset to 3D assets. This is available as a model on HuggingFace.
Rendering#
Rendering leverages gsplat. Find the rendering engine inside the main NuRec container.
You can use the NuRec gRPC API to neurally render scenes in your simulation platform. If you use the NVIDIA Physical AI dataset, follow the instructions in Render the Physical AI dataset with NuRec.
Harmonizer is an online diffusion enhancer that runs downstream of reconstruction and rendering. It turns imperfect neural-reconstruction renderings into temporally consistent, more photorealistic simulation frames—targeting appearance harmonization for re-inserted dynamic objects, novel-view artifact correction, and lighting realism. Harmonizer distills a multi-step diffusion model into a single-step, temporally conditioned enhancer backed by Cosmos-Predict2-0.6B-Text2Image, building on the single-step paradigm from Difix3D+. Pretrained checkpoints are on Hugging Face (nvidia/Harmonizer). In NuRec, enable it via the gRPC server; see Use Harmonizer for Post-Processing and the Harmonizer repository.
Further Reading#
Learn more about the technology powering NuRec in these research papers from NVIDIA Research: