> This page is for NVIDIA LipSync on Holoscan for Media, version 1.0.0 (Latest) (default).
> For other versions, use one of these documentation indexes:
> - 1.0.0 (Latest) (default): https://docs.nvidia.com/nim/maxine/lipsync-h4m/1.0.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nim/maxine/llms.txt.

# Overview

NVIDIA LipSync NIM is an AI-powered service that synchronizes lip movements in videos with input speech, creating naturally synchronized speech animations. The service processes both video and speech inputs to generate a seamlessly synchronized output video in which the subject's lip movements accurately match the speech.

This NIM is designed for live broadcast and production workflows that ingest and egress uncompressed video and audio over SMPTE ST 2110 using [NVIDIA Rivermax](https://developer.nvidia.com/networking/rivermax-getting-started). It is deployed with Helm charts on a [Holoscan for Media](https://developer.nvidia.com/holoscan-for-media) cluster and can optionally be managed through a Kubernetes operator.

Lip-synchronized SMPTE ST 2110-20 video from the NIM is transmitted downstream to the receiver. In the receiver pipeline, the video is muxed with SMPTE ST 2110-30 audio received from the sender, producing the final output.

## Architecture

The LipSync service is built on NVIDIA's software platform:

- [**Holoscan for Media**](https://developer.nvidia.com/holoscan-for-media): Media platform for SMPTE ST 2110–oriented live pipelines and Kubernetes-native orchestration.
- [**NVIDIA Rivermax**](https://developer.nvidia.com/networking/rivermax-getting-started): GPU-accelerated SMPTE ST 2110 streaming.
- [**NVIDIA DeepStream**](https://developer.nvidia.com/deepstream-sdk): GStreamer-based media pipelines and accelerated video preprocessing.
- **CUDA** for GPU-accelerated processing.
- **TensorRT** for optimized neural network inference.
- **Triton Inference Server** for efficient model serving and scaling.
- [**NVIDIA Augmented Reality (AR) SDK**](https://docs.nvidia.com/maxine/ar/latest/index.html) for face detection, temporal sequence processing, frame feature extraction, and LipSync inference.

## Processing Flow

### Pipeline
The following diagram summarizes how media flows through the LipSync NIM on Holoscan for Media:

![LipSync Holoscan for Media architecture and processing flow](/nim/maxine/_fern-files/maxine-nim.docs.buildwithfern.com/c17e6a2cc669f20437de8e7571c1f1248a672d5669951fe0c1e47f6a4de202ca/pages/lipsync-h4m/images/Lipsync_arch.svg)

The NIM ingests a video stream along with a corresponding translated (target) audio stream and generates a lip-synchronized output video aligned to the provided audio.

The service supports the following operating modes:

- Mode 1: Accepts video and audio inputs and produces a lip-synchronized video output.

- Mode 2: Accepts video, audio, and ancillary data (Example: active-speaker bounding boxes for multi-speaker scenarios) and produces a lip-synchronized video output.

This document focuses on Mode 1 (single-speaker).
Mode 2 requires integration with the NVIDIA Active Speaker Detection NIM to provide ancillary inputs such as speaker bounding boxes via the SMPTE ST 2110-40 ancillary data channel.

For the binary layout of optional ancillary data, including field offsets and sizes, refer to [Ancillary Data Payload (SMPTE ST 2110-40)](/nim/maxine/lipsync-h4m/advanced-usage#ancillary-data-payload-smpte-st-2110-40).

## Inputs and Outputs

| Direction | Stream Type | Transport | Format | Supported Configurations |
|-----------|-------------|-----------|--------|--------------------------|
| Input | Video | [SMPTE ST 2110-20](https://docs.nvidia.com/holoscan-for-media/latest/user-guide/getting-help/rivermax-st-2110.html) | Uncompressed [YCbCr 4:2:2 10-bit](https://docs.nvidia.com/holoscan-for-media/latest/user-guide/app-development/index.html#gpu-encoding-and-decoding) | Resolutions: 720p, 1080p, 4K UHD. Frame rates: 23.97 (24000/1001), 25, 29.97 (30000/1001), 30, 50, 59.94 (60000/1001), and 60 fps. |
| Input | Audio | [SMPTE ST 2110-30](https://docs.nvidia.com/holoscan-for-media/latest/user-guide/getting-help/rivermax-st-2110.html) | Mono [PCM L24](https://docs.nvidia.com/holoscan-for-media/latest/user-guide/getting-help/rivermax-st-2110.html#sdp), 48 kHz | One stream. |
| Input (optional) | Ancillary data | [SMPTE ST 2110-40](https://docs.nvidia.com/holoscan-for-media/latest/user-guide/getting-help/rivermax-st-2110.html) | SMPTE 291M | Per-frame bounding boxes, speaker IDs, and confidence scores (required for multi-speaker). |
| Output | Video | [SMPTE ST 2110-20](https://docs.nvidia.com/holoscan-for-media/latest/user-guide/getting-help/rivermax-st-2110.html) | Matches input raster | Lip-synchronized video output. |

## Deployment

Three deployment patterns are supported:

| Deployment Option | Description |
|-------------------|-------------|
| **NIM Service Chart** | Deploys the LipSync NIM as a standalone Kubernetes deployment for integration into a custom media pipeline. |
| **Kubernetes Operator** | Manages LipSync NIM workloads through a `NvidiaLipsyncMediaFunction` custom resource. Suitable for declarative, production-grade management. |
| **End-to-End Demo Chart** | Deploys a complete demo pipeline (sender, LipSync NIM service, and receiver) as a single Helm release. Demonstrates the full sender-to-receiver flow and is recommended for initial evaluation. |

For installation instructions, see [Installation](/nim/maxine/lipsync-h4m/installation/index).

## Supported Operating Modes

The solution supports the following industry-standard operating modes.

### ST 2110 Static Mode

Media streams such as video, audio, and ancillary data are transported over IP networks using predefined multicast IP addresses and ports. Configuration is performed manually, and IP address and port settings must match between sending and receiving devices to ensure proper media flow.

### NMOS Mode

Devices and media streams are automatically discovered and connected using AMWA NMOS standards. This mode simplifies system setup by removing the need for manual IP address and port configuration.

## See Also

- [Holoscan for Media](https://docs.nvidia.com/holoscan-for-media/latest/index.html) documentation.
- [System Requirements](https://docs.nvidia.com/holoscan-for-media/latest/user-guide/requirements/index.html) for Holoscan for Media.
- [Rivermax](https://docs.nvidia.com/holoscan-for-media/latest/user-guide/getting-help/rivermax-st-2110.html) and ST 2110 for Holoscan for Media.
- [NVIDIA DeepStream](https://developer.nvidia.com/deepstream-sdk)
- [NVIDIA Rivermax](https://developer.nvidia.com/networking/rivermax-getting-started)
- [Limitations and Known Behaviors](/nim/maxine/lipsync-h4m/limitations-and-known-behaviors)