Panoptic 3D Reconstruction with NVIDIA TAO Toolkit

Overview

Panoptic 3D reconstruction combines panoptic segmentation—the joint understanding of semantic regions and individual object instances—with geometric 3D scene reconstruction from a single RGB image. The result is a structured, instance-aware 3D representation of a scene that enables downstream tasks requiring both appearance and spatial understanding.

TAO Toolkit provides NvPanoptix3D, a two-stage network that performs 2D panoptic segmentation followed by frustum-based 3D panoptic reconstruction. By processing a single RGB image, NvPanoptix3D produces depth map, per-pixel 2D semantic labels, instance labels, and 3D panoptic reconstruction in a single inference pass.

Supported Networks

NvPanoptix3D

NvPanoptix3D is a transformer-based panoptic 3D reconstruction network. Its two-stage architecture separates 2D perception from 3D lifting and reconstruction, making each stage independently trainable and interpretable:

• Stage 1: 2D stage: A MaskFormer-based network produces 2D panoptic segmentation masks with class and instance labels.

• Stage 2: 3D stage: A frustum projection module lifts 2D features into 3D space using depth cues and geometric constraints. Then, a sparse 3D convolutional frustum decoder decodes the 3D space features to reconstruct 3D scene geometry, voxel-based 3D instance and semantic segmentations.

Use cases:

• Indoor scene understanding and room-layout estimation

• Robotics and autonomous navigation in structured environments

• Augmented reality and scene composition

• 3D dataset generation and simulation

For detailed training, evaluation, and deployment instructions, refer to NvPanoptix3D.

• NvPanoptix3D
  • Pipeline Overview
  • Dataset Format
  • Creating a Configuration File
  • Training
  • Evaluation
  • Performance
  • Inference
  • Export
  • Inference with NVIDIA Triton Inference Server