Overview
NVIDIA LipSync NIM is an AI-powered service that synchronizes lip movements in videos with input speech, creating naturally synchronized speech animations. The service processes both video and speech inputs to generate a seamlessly synchronized output video in which the subject’s lip movements accurately match the speech.
This NIM is designed for live broadcast and production workflows that ingest and egress uncompressed video and audio over SMPTE ST 2110 using NVIDIA Rivermax. It is deployed with Helm charts on a Holoscan for Media cluster and can optionally be managed through a Kubernetes operator.
Lip-synchronized SMPTE ST 2110-20 video from the NIM is transmitted downstream to the receiver. In the receiver pipeline, the video is muxed with SMPTE ST 2110-30 audio received from the sender, producing the final output.
Architecture
The LipSync service is built on NVIDIA’s software platform:
- Holoscan for Media: Media platform for SMPTE ST 2110–oriented live pipelines and Kubernetes-native orchestration.
- NVIDIA Rivermax: GPU-accelerated SMPTE ST 2110 streaming.
- NVIDIA DeepStream: GStreamer-based media pipelines and accelerated video preprocessing.
- CUDA for GPU-accelerated processing.
- TensorRT for optimized neural network inference.
- Triton Inference Server for efficient model serving and scaling.
- NVIDIA Augmented Reality (AR) SDK for face detection, temporal sequence processing, frame feature extraction, and LipSync inference.
Processing Flow
Pipeline
The following diagram summarizes how media flows through the LipSync NIM on Holoscan for Media:
The NIM ingests a video stream along with a corresponding translated (target) audio stream and generates a lip-synchronized output video aligned to the provided audio.
The service supports the following operating modes:
-
Mode 1: Accepts video and audio inputs and produces a lip-synchronized video output.
-
Mode 2: Accepts video, audio, and ancillary data (Example: active-speaker bounding boxes for multi-speaker scenarios) and produces a lip-synchronized video output.
This document focuses on Mode 1 (single-speaker). Mode 2 requires integration with the NVIDIA Active Speaker Detection NIM to provide ancillary inputs such as speaker bounding boxes via the SMPTE ST 2110-40 ancillary data channel.
For the binary layout of optional ancillary data, including field offsets and sizes, refer to Ancillary Data Payload (SMPTE ST 2110-40).
Inputs and Outputs
Deployment
Three deployment patterns are supported:
For installation instructions, see Installation.
Supported Operating Modes
The solution supports the following industry-standard operating modes.
ST 2110 Static Mode
Media streams such as video, audio, and ancillary data are transported over IP networks using predefined multicast IP addresses and ports. Configuration is performed manually, and IP address and port settings must match between sending and receiving devices to ensure proper media flow.
NMOS Mode
Devices and media streams are automatically discovered and connected using AMWA NMOS standards. This mode simplifies system setup by removing the need for manual IP address and port configuration.
See Also
- Holoscan for Media documentation.
- System Requirements for Holoscan for Media.
- Rivermax and ST 2110 for Holoscan for Media.
- NVIDIA DeepStream
- NVIDIA Rivermax
- Limitations and Known Behaviors