> This page is for NVIDIA LipSync NIM, version 1.2.0.
> For other versions, use one of these documentation indexes:
> - 1.4.0 (Latest) (default): https://docs.nvidia.com/nim/maxine/lipsync/1.4.0/llms.txt
> - 1.3.0: https://docs.nvidia.com/nim/maxine/lipsync/1.3.0/llms.txt
> - 1.2.0: https://docs.nvidia.com/nim/maxine/lipsync/1.2.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nim/maxine/llms.txt.

# Advanced Usage

## Model Caching

When the container launches for the first time, it downloads the required models from NGC. To avoid downloading the models on subsequent runs, you can cache them locally by using a cache directory:

```bash
# Create the cache directory on the host machine
export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p "$LOCAL_NIM_CACHE"
chmod 777 $LOCAL_NIM_CACHE

# Choose manifest profile ID based on target architecture.
export MANIFEST_PROFILE_ID=<enter_valid_manifest_profile_id>

# Run the container with the cache directory mounted in the appropriate location
docker run -it --rm --name=lipsync-nim \
  --runtime=nvidia \
  --gpus all \
  --shm-size=8GB \
  -e NGC_API_KEY=$NGC_API_KEY \
  -e NIM_MANIFEST_PROFILE=$MANIFEST_PROFILE_ID \
  -e NIM_HTTP_API_PORT=8000 \
  -p 8000:8000 \
  -p 8001:8001 \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  nvcr.io/nim/nvidia/lipsync:latest
```

For more information about `MANIFEST_PROFILE_ID`, refer to [Model Manifest Profiles](/nim/maxine/lipsync/1.2.0/getting-started#model-manifest-profiles).

## SSL enablement 

Lipsync NIM provides an SSL mode to ensure secure communication between clients and the server by encrypting data in transit.
To enable SSL, you must provide the path to the SSL certificate and key files in the container. The following example shows how to do this:

```bash
export NGC_API_KEY=<add-your-api-key>
SSL_CERT=path/to/ssl_key

docker run -it --rm --name=lipsync-nim \
  --runtime=nvidia \
  --gpus all \
  --shm-size=8GB \
  -v $SSL_CERT:/opt/nim/crt/:ro \
  -e NGC_API_KEY=$NGC_API_KEY \
  -p 8000:8000 \
  -p 8001:8001 \
  -e NIM_SSL_MODE="mtls" \
  -e NIM_SSL_CA_CERTS_PATH="/opt/nim/crt/ssl_ca.pem" \
  -e NIM_SSL_CERT_PATH="/opt/nim/crt/ssl_cert_server.pem" \
  -e NIM_SSL_KEY_PATH="/opt/nim/crt/ssl_key_server.pem" \
  nvcr.io/nim/nvidia/lipsync:latest
```

`NIM_SSL_MODE` can be set to `mtls`, `tls`, or `disabled`. If set to `mtls`, the container uses mutual TLS authentication. If set to `tls`, the container uses TLS authentication.
For more information, refer to [Environment Variables](/nim/maxine/lipsync/1.2.0/getting-started#environment-variables).

Be sure to verify the permissions of the SSL certificate and key files on the host machine. The container cannot access the files if they are not readable by the user running the container.

## NIM Service Configuration Parameters via Client

The following options can be configured by each request to the Lipsync NIM:

### Video-Audio Alignment Options

- `extend_audio`: Controls how to handle cases where video is longer than audio. This parameter is useful when working with content where the video track extends beyond the audio duration, such as when processing silent video segments, incomplete audio recordings, or when synchronizing content with varying durations.
  - `EXTEND_AUDIO_UNSPECIFIED` (default): Truncates video to match audio length.
  - `EXTEND_AUDIO_SILENCE`: Adds silent audio padding to keep audio and video synchronized.

  This option can be configured by setting the `extend_audio` parameter to `EXTEND_AUDIO_SILENCE` in the configuration message when making requests to the NIM.

  The following example uses the sample Python client to extend audio with custom options for handling mismatched input video and audio durations:
  ```bash
  python lipsync.py --target 127.0.0.1:8001 --video-input /path/to/video.mp4 --audio-input /path/to/audio.wav --extend-audio silence
  ```
  
- `extend_video`: Controls how to handle cases where audio is longer than video. This parameter is useful when working with content where the audio track extends beyond the video duration, such as when adding voiceovers, dubbing content etc.
  - `EXTEND_VIDEO_UNSPECIFIED` (default): Keeps original video length and truncates longer audio.
  - `EXTEND_VIDEO_FORWARD`: Extends video duration by repeating the final 5 seconds of frames in forward order until matching audio length.
  - `EXTEND_VIDEO_REVERSE`: Extends video duration by repeating the final 5 seconds of frames in reverse order until matching audio length.

  This option can be configured by setting the `extend_video` to `EXTEND_VIDEO_FORWARD` or `EXTEND_VIDEO_REVERSE` in the configuration message when making requests to the NIM.

  The following example uses the sample Python client to extend video with custom options for handling mismatched input video and audio durations:
  ```bash
  python lipsync.py --target 127.0.0.1:8001 --video-input /path/to/video.mp4 --audio-input /path/to/audio.wav --extend-video reverse
  ```

### Encoding Options
The output video encoding quality can be configured through the `output_video_encoding` parameter in the following ways:

- **Lossless encoding:** Provides maximum quality with no compression artifacts, preserving the original video quality pixel-for-pixel. This mode maintains the highest fidelity but results in significantly larger file sizes. Use this mode when quality is the top priority.

  - To configure this option in the configuration message when making requests to the NIM, set `output_video_encoding` to `VideoEncoding(lossless=True)`.

  - To run LipSync with lossless encoding (which overrides the bitrate setting) via the sample client, use the `--lossless` option:
    ```bash
    python lipsync.py --target 127.0.0.1:8001 --video-input /path/to/video.mp4 --audio-input /path/to/audio.wav --lossless
    ```

- **Bitrate control:** Allows balancing quality and file size by specifying the video bitrate in megabits per second (Mbps). The default is 30 Mbps. Higher bitrates produce better quality but larger file sizes; lower bitrates reduce file size at the cost of quality.

  - To configure this option in the configuration message when making requests to the NIM, set `output_video_encoding` to `VideoEncoding(lossy=LossyEncoding(bitrate_mbps=<desired-bitrate-in-mbps>))`.

  - To run LipSync with a desired output bitrate via the sample client, use the `--bitrate` option:
    ```bash
    python lipsync.py --target 127.0.0.1:8001 --video-input /path/to/video.mp4 --audio-input /path/to/audio.wav --bitrate 20
    ```

- **IDR interval control:**: Sets the interval between Instantaneous Decoder Refresh (IDR) frames (default: 8) for seeking and random-access capabilities. Lower values improve seeking accuracy, random access, and overall encoding quality but increase file size; higher values reduce file size but can impact seeking performance and quality.
     
  - To configure this option in the configuration message when making requests to the NIM, set `output_video_encoding` to `VideoEncoding(lossy=LossyEncoding(idr_interval=<desired_idr_interval>))`.

  - To run LipSync with a desired output bitrate via the sample client, use the `--idr_interval` option:
    ```bash
    python lipsync.py --target 127.0.0.1:8001 --video-input /path/to/video.mp4 --audio-input /path/to/audio.wav --idr-interval 10
    ```

- **Custom encoding parameters:** Provides fine-grained control for expert users via JSON configuration. These parameters configure properties of the [DeepStream H264 encoder](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvvideo4linux2.html#encoder).

  To specify custom encoding parameters, set `output_video_encoding` to `custom_encoding` in the configuration message sent to the NIM in every request:

  ```python
  custom_encoding = custom {
    bitrate:5000000,
    idr_interval:16,
    maxbitrate: 6000000
  }
  ```
  <Note>
  Custom encoding parameters override standard bitrate and IDR interval settings. Use with caution because incorrect values can affect output quality or encoding stability.
  </Note>
     
### Speaker Data Option

The `speaker_data_input` option specifies the path to a JSON file that defines per-frame speaker bounding box information for targeted lip synchronization. This option is specifically designed to support multi-speaker scenarios by letting you specify the active speaker to lip-sync on a per-frame basis. The file contains bounding-box coordinates (`x`, `y`, `width`, `height`), speaker identity, and speaking status for each frame. If not provided, the LipSync NIM automatically detects faces in each frame using its built-in face detection capabilities.

When providing speaker data, set `is_speaker_info_provided` to `True` in the configuration message when making requests to the NIM.

The `SpeakerInfo` structure contains the following fields:

| Field | Type | Description |
|-------|------|-------------|
| `speaker_bbox` | BoundingBox | Defines the bounding box coordinates for the speaker's face region. |
| `speaker_id` | int32 | Unique identifier for this speaker across frames. |
| `is_speaking` | bool | Flag indicating whether the speaker is currently speaking. |

The `BoundingBox` structure contains the following fields:

| Field | Type | Description |
|-------|------|-------------|
| `x` | float | X-coordinate of the top-left corner of the bounding box. |
| `y` | float | Y-coordinate of the top-left corner of the bounding box. |
| `width` | float | Width of the bounding box in pixels. |
| `height` | float | Height of the bounding box in pixels. |

The `SpeakerInfoPerFrame` structure wraps per-frame data:

| Field | Type | Description |
|-------|------|-------------|
| `frame_id` | uint32 | Frame index in the video. |
| `speaker_infos` | repeated SpeakerInfo | List of all speakers in this frame. |
| `bypass` | bool (optional) | If set to `true`, LipSync processing is bypassed for this frame and the original frame is returned unchanged. If `false` or unset, the frame is processed normally. This field is useful for cutscenes or frames that contain no subject. |

When providing speaker data, you must send a `SpeakerInfoPerFrame` object for every frame in the input video. For multi-speaker scenarios, each frame can contain multiple speakers.

**JSON File Format for Speaker Data**

When creating a JSON file for speaker data, use the following format. The file must contain a top-level `frames` array with one entry per frame, as in the following example:

```json
{
  "frames": [
    {
      "speakers": [
        {
          "bbox": [186, 191, 175, 254],
          "speaker_id": 1,
          "is_speaking": false
        },
        {
          "bbox": [815, 188, 263, 357],
          "speaker_id": 2,
          "is_speaking": true
        }
      ]
    },
    {
      "speakers": [
        {
          "bbox": [188, 191, 174, 254],
          "speaker_id": 1,
          "is_speaking": false
        }
      ]
    }
  ]
}
```

Each frame entry contains a `speakers` array with the following values:
- `bbox`: Array of four values `[x, y, width, height]` defining the bounding box of the speaker's face in pixel coordinates.
- `speaker_id`: Integer identifier for tracking this face across frames.
- `is_speaking`: Boolean indicating whether the speaker is currently speaking.

If the `speakers` array is empty or absent for a given frame, the server auto-detects faces for that frame.

**Obtaining Bounding Box Coordinates**

To create the speaker data JSON file, you need to obtain bounding-box coordinates for each frame. You can use an external face-detection system to generate these coordinates.

<Note>
The quality and accuracy of your face detection directly affects the LipSync results. For optimal lip-synchronization performance, ensure that your bounding boxes accurately encompass the facial regions.
</Note>

To run LipSync NIM with a speaker data file via the sample client, use the `--speaker-data-input` option:
```bash
python lipsync.py --target 127.0.0.1:8001 --video-input /path/to/video.mp4 --audio-input /path/to/audio.wav --speaker-data-input /path/to/speaker_data.json
```

### Head Movement Speed

The `head_movement_speed` parameter controls the expected speed of head movement in the input video:
- `0`: Static or slow-moving head (default).
- `1`: Fast-moving head.

This parameter helps the model optimize lip synchronization based on the dynamics of the subject's head movement.

To run LipSync with head movement speed via the sample client, use the `--head-movement-speed` option:
```bash
python lipsync.py --target 127.0.0.1:8001 --video-input /path/to/video.mp4 --audio-input /path/to/audio.wav --head-movement-speed 1
```

### Output Audio Codec

The `output_audio_codec` parameter specifies the audio codec used in the output video file:
- `opus`: Opus audio codec (default).
- `mp3`: MP3 audio codec.

To run LipSync with a specific output audio codec via the sample client, use the `--output-audio-codec` option:
```bash
python lipsync.py --target 127.0.0.1:8001 --video-input /path/to/video.mp4 --audio-input /path/to/audio.wav --output-audio-codec mp3
```

### Background Audio Mixing

The LipSync NIM supports mixing background audio with the output to preserve ambient sounds. This is useful when the original video contains background music or environmental sounds that should be retained in the output.

The background audio configuration includes the following options:
- `--mix-background-audio`: Flag to enable background audio mixing.
- `--background-audio-input`: Path to the background audio file (WAV or MP3).
- `--background-audio-volume`: Volume level for the background audio (0.0 to 1.0; default: 0.5).
  - `0.0`: Background audio is not included.
  - `1.0`: Background audio is included at its full, original volume.
  - Values between 0.0 and 1.0: Background volume is reduced proportionally.

<Note>
The background audio and the speech audio must have the same sample rate.
</Note>

To run LipSync with background audio mixing via the sample client:
```bash
python lipsync.py --target 127.0.0.1:8001 --video-input /path/to/video.mp4 --audio-input /path/to/audio.wav --mix-background-audio --background-audio-input /path/to/background.wav --background-audio-volume 0.3
```

### Python Configuration Example

The following example shows how to set configuration parameters while sending an inference request from a Python client to the LipSync NIM:
```python
import nvidia.ai4m.lipsync.v1.lipsync_pb2 as lipsync_pb2
import nvidia.ai4m.video.v1.video_pb2 as video_pb2
import nvidia.ai4m.audio.v1.audio_pb2 as audio_pb2

params = {
    "input_audio_codec": audio_pb2.AudioCodec.AUDIO_CODEC_WAV,
    "extend_audio": lipsync_pb2.ExtendAudio.EXTEND_AUDIO_SILENCE,
    "extend_video": lipsync_pb2.ExtendVideo.EXTEND_VIDEO_REVERSE,
    "output_video_encoding": video_pb2.VideoEncoding(lossless=True),
    "is_speaker_info_provided": True,
    "output_audio_codec": audio_pb2.AudioCodec.AUDIO_CODEC_OPUS,
    "head_movement_speed": 0,
}

yield lipsync_pb2.LipsyncRequest(config=lipsync_pb2.LipsyncConfig(**params))
```

### Debug Mode

The LipSync NIM includes a debug mode that provides visual feedback during processing. When enabled, diagnostic overlays are rendered directly onto each output video frame, making it easier to verify effect behavior and troubleshoot issues.

To enable debug mode, set the environment variable `LIPSYNC_DEBUG_MODE=1` when launching the NIM container:

```bash
docker run -it --rm --name=lipsync-nim \
  --runtime=nvidia \
  --gpus all \
  --shm-size=8GB \
  -e NGC_API_KEY=$NGC_API_KEY \
  -e LIPSYNC_DEBUG_MODE=1 \
  -p 8000:8000 \
  -p 8001:8001 \
  nvcr.io/nim/nvidia/lipsync:latest
```

When debug mode is enabled, the following overlays appear on each frame:

| Overlay | Description |
|---------|-------------|
| **Frame number** | Displayed at the top-center of each frame with a white background. |
| **LipSync effect status** | A `LIPSYNC ON` or `LIPSYNC OFF` indicator below the frame number. The background is green when the effect is active and red when it is bypassed. |
| **Activation bounding box** | A square bounding box showing where the LipSync effect is being applied. The box is green when the effect is active (strength > 0) and red when bypassed (strength = 0). Box coordinates and dimensions are labeled above the box. |
| **Speaker bounding boxes** | White bounding boxes for each speaker, shown only when speaker data is provided. Each box is labeled with a speaker identifier (e.g., `S0`, `S1`) and shows `[speaking]` when the speaker is actively speaking. |

Debug mode is useful for:
- Verifying that speaker bounding boxes are correctly positioned.
- Confirming that the LipSync effect is being applied to the intended regions.
- Troubleshooting cases where the effect appears inactive or misaligned.

For more information, refer to [Runtime Parameters for the Container](/nim/maxine/lipsync/1.2.0/getting-started#runtime-parameters-for-the-container).

For more information about AI for Media NIM clients, refer to the GitHub repository [NVIDIA-Maxine/nim-clients](https://github.com/NVIDIA-Maxine/nim-clients).