Is this page helpful?

Overview#

NVIDIA Relighting NIM is an AI-powered service that applies real-time lighting effects to videos using HDR environment maps. The service processes input video and applies AI-based relighting to generate output video with customizable lighting conditions, enabling dynamic lighting changes on video subjects.

Key Features#

  • HDR Environment Lighting: Supports five built-in HDR presets (Lounge, Cobblestone Street Night, Glasshouse Interior, Little Paris Eiffel Tower, Wooden Studio) and custom HDR environment map files.

  • Background Modes: Three background compositing options: source video, custom image, and HDR projection. Each mode can be combined with configurable blur strength.

  • Effect Parameters: Fine-grained control over foreground gain, background gain, blur strength, and specular highlights.

  • Streaming and Transactional Modes: Streaming mode (fMP4) for progressive playback; transactional mode for non-streamable inputs.

  • Configurable Output: Customizable video bitrates, resolution, IDR interval, and encoding options.

Architecture#

The Relighting service is built on NVIDIA’s software platform:

  • CUDA for GPU-accelerated processing.

  • TensorRT for optimized neural network inference.

  • Triton Inference Server for efficient model serving and scaling.

The system uses a composite architecture with multiple deep learning models working together:

  1. Frame feature extraction and segmentation — Convolutional Neural Networks (CNNs) analyze each input video frame.

  2. Foreground/background separation — AI Green Screen (AIGS) segmentation models isolate the subject from the background.

  3. HDR environment lighting — Relighting projection models apply the selected HDR environment map to the separated foreground.

  4. Compositing — The relit foreground is combined with the selected background mode (source video, custom image, or HDR projection).

The service maintains frame-accurate processing while preserving video quality, making it ideal for content creation, virtual production, and video post-processing applications.

Input Modes#

The Relighting NIM supports two modes of operation:

  • Transactional Mode

    In this mode, the whole input video is uploaded to the NIM before the inference begins. The inference is run on the input file to obtain the output video, which is then sent back to the client. Suitable for small files, applications that can wait for complete processing, and non-streamable input videos.

  • Streaming Mode

    In this mode, the input video is streamed to the NIM in chunks. As the chunks arrive, NIM runs inference incrementally and streams the output back to client in chunks, even before the whole input file is received by the NIM. The server automatically selects streaming mode when the input video is streamable (indicated by MP4 metadata at the start of the file). Preferred for large files.

For more details, refer to the Input Modes section of Basic Inference.