Skip to navigation

NVIDIA LipSync NIM is an AI-powered service that synchronizes lip movements in videos with input speech, creating naturally synchronized speech animations. The service processes both video and speech inputs to generate a seamlessly synchronized output video in which the subject’s lip movements accurately match the speech.

This NIM is designed for live broadcast and production workflows that ingest and egress uncompressed video and audio over SMPTE ST 2110 using NVIDIA Rivermax. It is deployed with Helm charts on a Holoscan for Media cluster and can optionally be managed through a Kubernetes operator.

Lip-synchronized SMPTE ST 2110-20 video from the NIM is transmitted downstream to the receiver. In the receiver pipeline, the video is muxed with SMPTE ST 2110-30 audio received from the sender, producing the final output.

Architecture

The LipSync service is built on NVIDIA’s software platform:

  • Holoscan for Media: Media platform for SMPTE ST 2110–oriented live pipelines and Kubernetes-native orchestration.
  • NVIDIA Rivermax: GPU-accelerated SMPTE ST 2110 streaming.
  • NVIDIA DeepStream: GStreamer-based media pipelines and accelerated video preprocessing.
  • CUDA for GPU-accelerated processing.
  • TensorRT for optimized neural network inference.
  • Triton Inference Server for efficient model serving and scaling.
  • NVIDIA Augmented Reality (AR) SDK for face detection, temporal sequence processing, frame feature extraction, and LipSync inference.

Processing Flow

Pipeline

The following diagram summarizes how media flows through the LipSync NIM on Holoscan for Media:

LipSync Holoscan for Media architecture and processing flow

The NIM ingests a video stream along with a corresponding translated (target) audio stream and generates a lip-synchronized output video aligned to the provided audio.

The service supports the following operating modes:

  • Mode 1: Accepts video and audio inputs and produces a lip-synchronized video output.

  • Mode 2: Accepts video, audio, and ancillary data (Example: active-speaker bounding boxes for multi-speaker scenarios) and produces a lip-synchronized video output.

This document focuses on Mode 1 (single-speaker). Mode 2 requires integration with the NVIDIA Active Speaker Detection NIM to provide ancillary inputs such as speaker bounding boxes via the SMPTE ST 2110-40 ancillary data channel.

For the binary layout of optional ancillary data, including field offsets and sizes, refer to Ancillary Data Payload (SMPTE ST 2110-40).

Inputs and Outputs

DirectionStream TypeTransportFormatSupported Configurations
InputVideoSMPTE ST 2110-20Uncompressed YCbCr 4:2:2 10-bitResolutions: 720p, 1080p, 4K UHD. Frame rates: 23.97 (24000/1001), 25, 29.97 (30000/1001), 30, 50, 59.94 (60000/1001), and 60 fps.
InputAudioSMPTE ST 2110-30Mono PCM L24, 48 kHzOne stream.
Input (optional)Ancillary dataSMPTE ST 2110-40SMPTE 291MPer-frame bounding boxes, speaker IDs, and confidence scores (required for multi-speaker).
OutputVideoSMPTE ST 2110-20Matches input rasterLip-synchronized video output.

Deployment

Three deployment patterns are supported:

Deployment OptionDescription
NIM Service ChartDeploys the LipSync NIM as a standalone Kubernetes deployment for integration into a custom media pipeline.
Kubernetes OperatorManages LipSync NIM workloads through a NvidiaLipsyncMediaFunction custom resource. Suitable for declarative, production-grade management.
End-to-End Demo ChartDeploys a complete demo pipeline (sender, LipSync NIM service, and receiver) as a single Helm release. Demonstrates the full sender-to-receiver flow and is recommended for initial evaluation.

For installation instructions, see Installation.

Supported Operating Modes

The solution supports the following industry-standard operating modes.

ST 2110 Static Mode

Media streams such as video, audio, and ancillary data are transported over IP networks using predefined multicast IP addresses and ports. Configuration is performed manually, and IP address and port settings must match between sending and receiving devices to ensure proper media flow.

NMOS Mode

Devices and media streams are automatically discovered and connected using AMWA NMOS standards. This mode simplifies system setup by removing the need for manual IP address and port configuration.

See Also