Skip to navigation

Getting Started

View as MarkdownOpen in Claude

Prerequisites

To ensure that you have the supported hardware and software stack, check the Support Matrix.

NGC Authentication

Generate an API Key

LipSync NIM is available only through the AI for Media Private Access Program. Joining the Private Access Program gives you an NGC API key with the permissions required for this NIM.

You can generate a key at https://org.ngc.nvidia.com/setup/api-keys after joining the Private Access Program.

When creating an NGC API Personal key, ensure that at least NGC Catalog is selected from the Services Included dropdown. You can include more services if this key is to be reused for other purposes.

Personal keys allow you to configure an expiration date, revoke or delete the key using an action button, and rotate the key as needed. For more information about key types, refer to NGC API Keys in the NGC User Guide.

Export the NGC API Key

Pass the value of the API key to the docker run command in the next section as the NGC_API_KEY environment variable to download the appropriate models and resources when starting the NIM.

If you are not familiar with how to create the NGC_API_KEY environment variable, the simplest way is to export it in your terminal:

export NGC_API_KEY=<value>

Run one of the following commands to make the key available at startup:

# If using bash
echo "export NGC_API_KEY=<value>" >> ~/.bashrc
# If using zsh
echo "export NGC_API_KEY=<value>" >> ~/.zshrc

Other, more secure options include saving the value in a file, so that you can retrieve with cat $NGC_API_KEY_FILE, or using a password manager.

Docker Login to NGC

To pull the NIM container image from NGC, first authenticate with the NVIDIA Container Registry with the following command:

echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin

Use $oauthtoken as the username and NGC_API_KEY as the password. The $oauthtoken username is a special name that indicates that you will authenticate with an API key and not a user name and password.

Before pulling the container image, you must accept the Governing Terms on the LipSync NIM container page in NGC. This page is accessible only after enrollment in the AI for Media Private Access Program.

Launching the NIM Container

The following command launches the LipSync NIM container with the gRPC service. (For a list of parameters, see Runtime Parameters for the Container.)

# Choose manifest profile ID based on target architecture.
export MANIFEST_PROFILE_ID=<enter_valid_manifest_profile_id>

Then run the NIM launch command:

docker run -it --rm --name=lipsync-nim \
--runtime=nvidia \
--gpus all \
--shm-size=8GB \
-e NGC_API_KEY=$NGC_API_KEY \
-e NIM_MANIFEST_PROFILE=$MANIFEST_PROFILE_ID \
-e NIM_MAX_CONCURRENCY_PER_GPU=1 \
-e NIM_HTTP_API_PORT=8000 \
-e NIM_GRPC_API_PORT=8001 \
-p 8000:8000 \
-p 8001:8001 \
nvcr.io/nim/nvidia/lipsync:latest

The flag --gpus all is used to assign all available GPUs to the NIM container. To assign specific GPUs to the NIM container (in case of multiple GPUs available in your machine), use --gpus '"device=0,1,2..."'.

Model Manifest Profiles

The following table lists the manifest profile IDs for the generic, language-agnostic model that you can specify in MANIFEST_PROFILE_ID.

GPU Architecture (compute capability)Manifest Profile ID
Blackwell (cc 12.0)633d503ee0efadac22f33b0d81b4cd168d58452233b3a0575e68c933edc9d5f5
Ada (cc 8.9)282341d1997797fd6cf7ddc22aacf83d55a049db79193db0128ded1bce0e0610
Ampere (cc 8.6)9c3bcdea7d1d83e7d5956fd6fd3b5798eaaab67b13c0f281485585f34232934e
Turing (cc 7.5)01c65463c5f01ab06b7a8ff3fc62929bfee2c3003ef7af0a9eb1227ec874f749

For language-specific models, do not set MANIFEST_PROFILE_ID. Instead, use NIM_TAGS_SELECTOR (refer to Selecting a Language-Specific Model), which automatically selects the profile that matches both your GPU architecture and the requested language.

MANIFEST_PROFILE_ID is an optional parameter. If the manifest profile ID is not supplied, the NIM automatically selects a matching profile ID based on the target hardware architecture.

However, if MANIFEST_PROFILE_ID is used, ensure that the associated GPU architecture is compatible with the target hardware. If an incorrect manifest profile ID is used, a deserialization error occurs on inference.

Selecting a Language-Specific Model

The LipSync NIM includes a generic, language-agnostic model (used by default) and fine-tuned models for specific languages. You select a language-specific model at NIM container launch time by using the NIM_TAGS_SELECTOR environment variable. Each container loads exactly one model.

NIM_TAGS_SELECTORModel used
(not set)Generic, language-agnostic model (default)
language=deGerman
language=esSpanish
language=frFrench

Only de, es, and fr are supported. Any other value causes the container to fail at startup; omit the variable to use the generic model.

The following example launches the NIM with the German model:

docker run -it --rm --name=lipsync-nim \
--runtime=nvidia \
--gpus all \
--shm-size=8GB \
-e NGC_API_KEY=$NGC_API_KEY \
-e NIM_TAGS_SELECTOR="language=de" \
-e NIM_HTTP_API_PORT=8000 \
-e NIM_GRPC_API_PORT=8001 \
-p 8000:8000 \
-p 8001:8001 \
nvcr.io/nim/nvidia/lipsync:latest

When you select a language with NIM_TAGS_SELECTOR, leave NIM_MANIFEST_PROFILE unset. The NIM automatically selects the profile that matches both your GPU architecture and the requested language.

If the NIM launch is successful, you get a response similar to the following.

I1027 22:31:44.952125 123 grpc_server.cc:2560] "Started GRPCInferenceService at 127.0.0.1:9001"
I1027 22:31:44.952247 123 http_server.cc:4755] "Started HTTPService at 127.0.0.1:9000"
I1027 22:31:44.993329 123 http_server.cc:358] "Started Metrics Service at 127.0.0.1:9002"
Triton server is ready
[INFO AI4M BASE LOGGER 2025-10-27 22:31:46.097 PID:207] Using threading mode for gRPC service
[INFO AI4M BASE LOGGER 2025-10-27 22:31:46.097 PID:207] Starting threading gRPC service with 1 threads
[INFO AI4M BASE LOGGER 2025-10-27 22:31:46.105 PID:207] Using Insecure Server Credentials
[INFO AI4M BASE LOGGER 2025-10-27 22:31:46.107 PID:207] Listening to 0.0.0.0:8001

By default, the LipSync NIM gRPC service is hosted on port 8001. You must use this port for inferencing requests. The port is configurable via the NIM_GRPC_API_PORT environment variable.

Environment Variables

The following table describes the environment variables that can be passed into a NIM as a -e argument added to a docker run command:

ENVRequired?DefaultNotes
NGC_API_KEYYesNoneYou must set this variable to the value of your personal NGC API key.
NIM_CACHE_PATHOptional/opt/nim/.cacheLocation (in container) where the container caches model artifacts.
NIM_GRPC_API_PORTNo8001Publish the NIM service to the prescribed port inside the container. Be sure to adjust the port passed to the -p/--publish flag of docker run to reflect that (for example, -p $NIM_GRPC_API_PORT:$NIM_GRPC_API_PORT). The left side of the colon (:) is the port of your host address and does not need to match $NIM_GRPC_API_PORT. The right side of the colon (:) is the port inside the container, which must match NIM_GRPC_API_PORT (or 8001 if not set). Supported endpoints are /v1/license (returns the license information), /v1/metadata (returns metadata including asset information, license information, model information, and version), and /v1/metrics (exposes Prometheus metrics via an ASGI app endpoint).
NIM_MANIFEST_PROFILEOptionalNoneYou must set this model profile to be able to download the specific model type supported on your GPU. For more about NIM_MANIFEST_PROFILE, refer to Model Manifest Profiles.
NIM_TAGS_SELECTOROptionalNoneSelects the language-specific model to load, in the form language=<code>; <code> is de, es, or fr. If not set, the generic, language-agnostic model is used. For details, refer to Selecting a Language-Specific Model.
NIM_SSL_MODENodisabledSet SSL security on the endpoints to tls or mtls. Defaults to unsecured endpoint.
NIM_SSL_CA_PATHNoNoneSet the path to CA root certificate inside the NIM. This is required only when NIM_SSL_MODE is mtls. For example, if the SSL certificates are mounted at /opt/nim/crt in the NIM, NIM_SSL_CA_PATH can be set to /opt/nim/crt/ssl_ca_cert.pem.
NIM_SSL_CERT_PATHNoNoneSet the path to the server’s public SSL certificate inside the NIM. This is required only when an SSL mode is enabled. For example, if the SSL certificates are mounted at /opt/nim/crt in the NIM, NIM_SSL_CERT_PATH can be set to /opt/nim/crt/ssl_cert_server.pem.
NIM_SSL_KEY_PATHNoNoneSet the path to the server’s private key inside the NIM. This is required only when an SSL mode is enabled. For example, if the SSL certificates are mounted at /opt/nim/crt in the NIM,NIM_SSL_KEY_PATH can be set to /opt/nim/crt/ssl_key_server.pem.
NV_AI4M_LS_INPUT_QUEUE_TIMEOUT_SNo30Maximum number of seconds the service will wait for the arrival of audio samples and speaker information for a decoded video frame before raising a TimeoutException. If the required audio or speaker data for a frame does not arrive within this period, the request fails.

Runtime Parameters for the Container

FlagsDescription
-it--interactive + --tty (see docker container run).
--rmDelete the container after it stops (see docker container run).
--name=container-nameGive a name to the NIM container. Use any preferred value.
--runtime=nvidiaEnsure NVIDIA drivers are accessible in the container.
--gpus allExpose NVIDIA GPUs inside the container. If you are running on a host with multiple GPUs, you need to specify which GPU to use. You can also specify multiple GPUs. For more information about mounting specific GPUs, see GPU Enumeration.
--shm-size=8GBAllocate host memory for multi-process communication.
-e NIM_MAX_CONCURRENCY_PER_GPUNumber of concurrent inference requests to be supported by the NIM server per GPU (default: 1). Higher values consume more GPU memory and can cause out-of-memory errors.
-e NGC_API_KEY=$NGC_API_KEYProvide the container with the token necessary to download adequate models and resources from NGC. See NGC Authentication.
-p <host_port>:<container_port>Ports published by the container are directly accessible on the host port.
-e LIPSYNC_DEBUG_MODEEnvironment variable to enable debug mode that overlays frame number, lipsync effect status and bounding boxes for each frame. Enable by setting it to 1. (default: 0).

Stopping the Container

The following commands can be used to stop the container.

docker stop $CONTAINER_NAME
docker rm $CONTAINER_NAME