Is this page helpful?

Advanced Usage#

This page describes deployment options, request parameters, and runtime behavior for the Active Speaker Detection NIM.

Model Profiles and Selection#

A NIM model profile defines which model engines the NIM can use and the criteria it uses to choose them. Each profile is identified by a unique hash of its contents.

You can select a profile at deployment time by following the Profile Selection steps. If you do not set a profile, the NIM chooses one that matches the GPU compute capability.

Model profiles are embedded within the NIM container in a Model Manifest file, which is by default placed at /etc/nim/config/model_manifest.yaml within the container filesystem.

The following table lists model profile IDs that you can specify in NIM_MODEL_PROFILE.

GPU Architecture (compute capability)

Model Profile ID

Blackwell (cc 12.0)

eb02786152e604d9495f891e38b2da550db36fa2c9e91d25211b0c1c8c9d6175

Blackwell (cc 10.0)

f1d12a64b41e0a9972ba7b361437c0367e08c5df79a04ca01d8af6581deda1a0

Hopper (cc 9.0)

d8e49a40fcf78248e355a542405cbd9ccb975ac79f5c07bb2775975017c979c7

Ada (cc 8.9)

85f1043c43b43b954053946f6649e6a24f57a125e74ee656668d7889d0f9a186

Ampere (cc 8.6)

d3c85f2b926806c6a307b3ed08421a315658b4d9ce3dea73ad47ad1ab055d9a9

Ampere (cc 8.0)

f4a75f436f19b3e4728c334240d5b2ff15b102ec953ee7d1cb231c9fc93dd3d3

Turing (cc 7.5)

fabc73f93374ed9ad5069e6e05a0a95ad9e28e1ab65b3a235199b53320c77a3c

Profile Selection#

To select a model profile for deployment, set a specific profile ID with -e NIM_MODEL_PROFILE=<value>. You can find the valid profile IDs by using the list-model-profiles utility:

docker run --rm --runtime=nvidia -e NGC_API_KEY=$NGC_API_KEY --gpus=all \
  nvcr.io/nim/nvidia/active-speaker-detection:latest list-model-profiles

The utility prints the detected system information followed by the profiles that are compatible and incompatible with that hardware. Each entry lists the profile ID followed by its tags. The following example shows the output on an Ada (compute capability 8.9) system, where only the matching profile is compatible:

SYSTEM INFO
- Free GPUs:
  -  [26b9:10de] (0) NVIDIA L40S [current utilization: 0%]
MODEL PROFILES
- Compatible with system:
    - 85f1043c43b43b954053946f6649e6a24f57a125e74ee656668d7889d0f9a186 - backend:trt|compute_capability:8.9|model_type:trt|pp:1|precision:fp32|tp:1
- Incompatible with system:
    - eb02786152e604d9495f891e38b2da550db36fa2c9e91d25211b0c1c8c9d6175 - backend:trt|compute_capability:12.0|model_type:trt|pp:1|precision:fp32|tp:1
    - f1d12a64b41e0a9972ba7b361437c0367e08c5df79a04ca01d8af6581deda1a0 - backend:trt|compute_capability:10.0|model_type:trt|pp:1|precision:fp32|tp:1
    - d8e49a40fcf78248e355a542405cbd9ccb975ac79f5c07bb2775975017c979c7 - backend:trt|compute_capability:9.0|model_type:trt|pp:1|precision:fp32|tp:1
    - d3c85f2b926806c6a307b3ed08421a315658b4d9ce3dea73ad47ad1ab055d9a9 - backend:trt|compute_capability:8.6|pp:1|precision:fp32|tp:1
    - f4a75f436f19b3e4728c334240d5b2ff15b102ec953ee7d1cb231c9fc93dd3d3 - backend:trt|compute_capability:8.0|model_type:trt|pp:1|precision:fp32|tp:1
    - fabc73f93374ed9ad5069e6e05a0a95ad9e28e1ab65b3a235199b53320c77a3c - backend:trt|compute_capability:7.5|model_type:trt|pp:1|precision:fp32|tp:1

Then run the NIM launch command:

docker run -it --rm --name=active-speaker-detection-nim \
  --runtime=nvidia \
  --gpus all \
  --shm-size=8GB \
  -e NGC_API_KEY=$NGC_API_KEY \
  -e NIM_MODEL_PROFILE=$MODEL_PROFILE_ID \
  -e NV_AI4M_MAX_CONCURRENCY_PER_GPU=1 \
  -e NIM_HTTP_API_PORT=8000 \
  -e NIM_GRPC_API_PORT=8001 \
  -p 8000:8000 \
  -p 8001:8001 \
  nvcr.io/nim/nvidia/active-speaker-detection:latest

Note

NIM_MODEL_PROFILE is an optional parameter. If the model profile ID is not supplied, the NIM automatically selects a matching profile ID based on the target hardware architecture.

However, if NIM_MODEL_PROFILE is used, ensure that the associated GPU architecture is compatible with the target hardware. If an incorrect model profile ID is used, a deserialization error occurs on inference.

Model Caching#

When the container launches for the first time, it downloads the required models from NGC. To avoid downloading the models on subsequent runs, you can cache them locally by using a cache directory:

# Create the cache directory on the host machine.
export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p "$LOCAL_NIM_CACHE"
chmod 777 $LOCAL_NIM_CACHE

# Choose model profile ID based on target architecture.
export MODEL_PROFILE_ID=<enter_valid_model_profile_id>

# Run the container with the cache directory mounted in the appropriate location.
docker run -it --rm --name=active-speaker-detection-nim \
  --runtime=nvidia \
  --gpus all \
  --shm-size=8GB \
  -e NGC_API_KEY=$NGC_API_KEY \
  -e NIM_MODEL_PROFILE=$MODEL_PROFILE_ID \
  -e NIM_HTTP_API_PORT=8000 \
  -e NIM_GRPC_API_PORT=8001 \
  -p 8000:8000 \
  -p 8001:8001 \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  nvcr.io/nim/nvidia/active-speaker-detection:latest

For more information about MODEL_PROFILE_ID, refer to Model Profiles and Selection.

SSL Enablement#

Active Speaker Detection NIM provides an SSL mode to ensure secure communication between clients and the server by encrypting data in transit. To enable SSL, you must provide the path to the SSL certificate and key files in the container. The following example shows how to do this:

export NGC_API_KEY=<add-your-api-key>
export SSL_CERT=path/to/ssl_key

docker run -it --rm --name=active-speaker-detection-nim \
  --runtime=nvidia \
  --gpus all \
  --shm-size=8GB \
  -v $SSL_CERT:/opt/nim/crt/:ro \
  -e NGC_API_KEY=$NGC_API_KEY \
  -p 8000:8000 \
  -p 8001:8001 \
  -e NIM_SSL_MODE="mtls" \
  -e NIM_SSL_CA_CERTS_PATH="/opt/nim/crt/ssl_ca.pem" \
  -e NIM_SSL_CERT_PATH="/opt/nim/crt/ssl_cert_server.pem" \
  -e NIM_SSL_KEY_PATH="/opt/nim/crt/ssl_key_server.pem" \
  nvcr.io/nim/nvidia/active-speaker-detection:latest

NIM_SSL_MODE can be set to mtls, tls, or disabled. If set to mtls, the container uses mutual TLS authentication. If set to tls, the container uses TLS authentication. For more information, refer to Environment Variables.

Be sure to verify the permissions of the SSL certificate and key files on the host machine. The container cannot access the files if they are not readable by the user running the container.

Multiple Concurrent Inputs#

To run the server in multi-input concurrent mode, set the environment variable NV_AI4M_MAX_CONCURRENCY_PER_GPU to an integer greater than 1 in the server container. The server then accepts as many concurrent inputs per GPU as specified by the NV_AI4M_MAX_CONCURRENCY_PER_GPU variable.

Because Triton distributes the workload equally across all GPUs, the total number of concurrent inputs supported by the server is the number of GPUs multiplied by NV_AI4M_MAX_CONCURRENCY_PER_GPU.

Note

On bare-metal Linux systems with PCIe topology, refer to PCIe Multi-GPU Systems for the recommended IOMMU configuration.

The Active Speaker Detection NIM uses NVENC/NVDEC hardware acceleration for video decoding.

  • GPUs without NVDEC hardware support might not be supported.

  • Some GPUs support only a limited number of concurrent NVDEC sessions, which means the NIM can process only that same number of concurrent inputs on those GPUs.

For details, refer to the Video Encode and Decode Support Matrix.

Note

If incoming requests to the NIM exceed the GPU’s maximum concurrent decode limit, the processing might fail.

Maximum Speakers Configuration#

The NV_AI4M_ASD_MAX_SPEAKERS environment variable controls the maximum number of audio speaker streams that the Active Speaker Detection NIM can handle simultaneously. This setting affects both the gRPC service and the Triton backend.

  • Default: 5

  • Range: 1 to 5

To configure the maximum number of speakers:

docker run -it --rm --name=active-speaker-detection-nim \
  --runtime=nvidia \
  --gpus all \
  --shm-size=8GB \
  -e NGC_API_KEY=$NGC_API_KEY \
  -e NV_AI4M_ASD_MAX_SPEAKERS=2 \
  -e NIM_HTTP_API_PORT=8000 \
  -e NIM_GRPC_API_PORT=8001 \
  -p 8000:8000 \
  -p 8001:8001 \
  nvcr.io/nim/nvidia/active-speaker-detection:latest

Note

The NV_AI4M_ASD_MAX_SPEAKERS value must match the number of speaker streams expected in your use case.

Service Information in the Response#

Each response stream opens with a one-shot ServiceInfo message on the service_info field of DetectActiveSpeakerResponse, before any detection results. It reports feature_name, feature_version, model_info, server_request_id, and client_session_id. Quote server_request_id when reporting an issue — it locates the request in the server logs.

The following example prints the server request ID, then detection results:

for response in response_iterator:
    if response.HasField("service_info"):
        print(response.service_info.server_request_id)
    elif response.HasField("active_speaker_detection_result"):
        result = response.active_speaker_detection_result
        print(result.frame_id, len(result.speaker_data))

Audio Source Configuration#

The Active Speaker Detection NIM supports two modes for audio input feed, configured by the client at request time:

  • Separate Stream (AUDIO_SOURCE_CONFIG_SEPARATE_STREAM): The client provides a separate audio file alongside the video. This is the default behavior. Use the --audio-input argument in the client to specify the audio file path.

  • Embedded in Video (AUDIO_SOURCE_CONFIG_EMBEDDED_IN_VIDEO): Audio is demuxed from the supplied video container. No separate audio file is needed. Use the --skip-audio flag in the client.

Separate Audio Stream#

The following example sends a separate audio file with the video:

python active_speaker_detection.py --target 127.0.0.1:8001 \
   --video-input /path/to/video.mp4 \
   --audio-input /path/to/audio.wav \
   --diarization-input /path/to/diarization.json

Embedded Audio#

The following example uses the audio embedded in the video container:

python active_speaker_detection.py --target 127.0.0.1:8001 \
   --video-input /path/to/video_with_audio.mp4 \
   --diarization-input /path/to/diarization.json \
   --skip-audio

Speaker Detection Threshold#

The speaker_detection_threshold field in the ActiveSpeakerDetectionConfig proto controls the minimum confidence score required to classify a detected face as actively speaking.

  • Valid range: (0, 1) open interval (exclusive of both endpoints).

  • Proto field: optional float speaker_detection_threshold in ActiveSpeakerDetectionConfig.

Note

The sample client (active_speaker_detection.py) exposes this field through the --speaker-detection-threshold command-line flag. You can also set speaker_detection_threshold on the ActiveSpeakerDetectionConfig proto directly in your own client code, as shown below.

A lower threshold is more permissive, resulting in more faces being classified as speaking. A higher threshold is more strict, resulting in fewer faces classified as speaking. The face-detection bounding boxes themselves are not affected by this threshold—only the is_speaking classification changes.

To set the threshold, include the speaker_detection_threshold field in the ActiveSpeakerDetectionConfig message sent as the first request in the gRPC stream:

detection_config = ActiveSpeakerDetectionConfig(
    input_video_config=video_config,
    input_audio_config=audio_config,
    audio_source_config=AUDIO_SOURCE_CONFIG_SEPARATE_STREAM,
    speaker_detection_threshold=0.75,
)

The value must be in the open interval (0, 1). Omitting the field or setting it to 0 uses the configured default. Any other value outside the range (< 0 or >= 1, such as -0.5 or 1.5) is rejected with an INVALID_ARGUMENT gRPC error.

Voice Activity Smoothing for Diarization Workflows#

The speaking state produced for diarized tracks can be temporally smoothed so that a track does not flicker between speaking and not-speaking on a frame-to-frame basis. Smoothing is especially useful when diarization arrives incrementally, such as streaming or word-by-word diarization. Each packet covers only a short slice of time, so there are often gaps at word or segment boundaries, including brief silences and inter-word pauses. Without smoothing, those uncovered frames are treated as not speaking, and a continuous speaker flickers off and on. Temporal smoothing holds the speaking state across those short gaps.

Smoothing supports three levels:

  • off — No smoothing. The raw per-frame speaking decision is returned.

  • low — Light smoothing that removes brief single-frame flips.

  • high (default) — Stronger smoothing that keeps a diarized speaker’s state stable across short pauses and detection noise.

Smoothing is a deploy-time setting that can be overridden per request.

Deploy-Time Configuration#

Set the NV_AI4M_ASD_VA_SMOOTHING environment variable when launching the container to select the default smoothing level for all requests:

docker run -it --rm --name=active-speaker-detection-nim \
  --runtime=nvidia \
  --gpus all \
  --shm-size=8GB \
  -e NGC_API_KEY=$NGC_API_KEY \
  -e NV_AI4M_ASD_VA_SMOOTHING=high \
  -e NIM_HTTP_API_PORT=8000 \
  -e NIM_GRPC_API_PORT=8001 \
  -p 8000:8000 \
  -p 8001:8001 \
  nvcr.io/nim/nvidia/active-speaker-detection:latest

Valid values are off, low, and high (case-insensitive). If the variable is unset or invalid, the server falls back to high.

Per-Request Override#

An individual client request can override the deployment default with the voice_activity_smoothing field in the ActiveSpeakerDetectionConfig proto. The field uses the VoiceActivitySmoothing enum:

Enum value

Effect

VOICE_ACTIVITY_SMOOTHING_UNSPECIFIED

Use the deployment default

VOICE_ACTIVITY_SMOOTHING_OFF

Disable smoothing for this request.

VOICE_ACTIVITY_SMOOTHING_LOW

Low smoothing for this request.

VOICE_ACTIVITY_SMOOTHING_HIGH

High smoothing for this request.

The following example overrides the deployment default:

detection_config = ActiveSpeakerDetectionConfig(
    input_video_config=video_config,
    input_audio_config=audio_config,
    audio_source_config=AUDIO_SOURCE_CONFIG_SEPARATE_STREAM,
    voice_activity_smoothing=VOICE_ACTIVITY_SMOOTHING_HIGH,
)

When the field is omitted or set to VOICE_ACTIVITY_SMOOTHING_UNSPECIFIED, the server applies the deploy-time default.

Multi-View Speaker Identification#

By default, each video stream is processed as an isolated session, so the same physical person appearing in two concurrent camera streams is not correlated across those streams. Enabling the multi-view model correlates the same people across concurrent camera streams, so tracking IDs stay consistent and a speaker can be followed as they move between views.

Note

Multi-view is a deploy-time selection. When it is disabled (the default), each stream is treated as an independent session.

To enable cross-stream identification, set NV_AI4M_ASD_MULTIVIEW=true when launching the container. This selects the multi-view inference model, which shares the same input and output tensors as the single-view model:

docker run -it --rm --name=active-speaker-detection-nim \
  --runtime=nvidia \
  --gpus all \
  --shm-size=8GB \
  -e NGC_API_KEY=$NGC_API_KEY \
  -e NV_AI4M_ASD_MULTIVIEW=true \
  -e NIM_HTTP_API_PORT=8000 \
  -e NIM_GRPC_API_PORT=8001 \
  -p 8000:8000 \
  -p 8001:8001 \
  nvcr.io/nim/nvidia/active-speaker-detection:latest

Note

Streams that need to share tracking IDs must be submitted concurrently within the same session. Only concurrent streams are batched and processed together as a single session, which is what enables ID sharing.

Output Semantics in Multi-View Mode#

Multi-view mode changes only the meaning of tracking_id:

  • Default model: IDs are local to each stream.

  • Multi-view mode: IDs are assigned from a shared face gallery across streams in the Triton backend process.

The output shape is unchanged.

The speaking-state fields remain stream-local. The same global tracking_id can have different is_speaking, audio_id, and confidence values on different streams when the views differ in face angle, size, occlusion, or audio-visual sync evidence.

Input Stream Handling#

The Active Speaker Detection NIM ingests video, audio, and diarization streams on independent feeder threads. Each input stream is decoupled from the request thread, so a stalled or late stream does not block the request thread or starve the other input streams. This behavior is governed by per-unit coverage timeouts, buffer caps, and an explicit diarization miss policy.

Per-Unit Coverage Timeouts#

Each input stream has its own per-unit wait budget. The budget is applied to each unit of work (for example, each decoded video frame), not a single time for the whole request.

Environment Variable

Default

Description

NV_AI4M_ASD_VIDEO_TIMEOUT_S

30.0

Per-unit budget for the video stream to cover a frame in seconds. The same budget bounds the wait for the first decoded frame.

NV_AI4M_ASD_AUDIO_TIMEOUT_S

30.0

Per-unit budget waiting for audio samples before the audio stream times out in seconds.

NV_AI4M_ASD_DIARIZATION_TIMEOUT_S

30.0

Per-unit wait budget for diarization coverage in seconds. Applies only when diarization is included in the request.

Each budget must be greater than zero. A 0, negative, or non-numeric value is rejected at startup: the NIM logs a warning and uses the 30.0 second default instead. A literal zero second budget would expire before any producer could deliver the first unit, failing every request.

Buffer Caps#

Each feeder bounds how much input data it will hold per stream while its queued for processing. These caps are an out-of-memory backstop, not flow control: a stream only accumulates when its reader has stopped consuming, and exceeding the cap fails the request with a RESOURCE_EXHAUSTED gRPC error rather than blocking the request thread (which would stall every other stream).

Note

Exercise caution when raising concurrency together with these buffer caps. Caps apply per request, so total host memory scales with NV_AI4M_MAX_CONCURRENCY_PER_GPU and the number of GPUs. At high concurrency, lower the caps so a stalled reader cannot exhaust host memory, and keep timeout budgets consistent with the reduced buffers. Raising both concurrency and cap values without reviewing memory headroom is not recommended.

Environment Variable

Default

Description

NV_AI4M_ASD_MAX_VIDEO_INPUT_BUFFER_MB

2048

Maximum encoded video, in MB, that the NIM queues when it cannot decode video as fast as it arrives. Exceeding this cap fails the request with RESOURCE_EXHAUSTED.

NV_AI4M_ASD_MAX_AUDIO_INPUT_BUFFER_MB

512

Maximum encoded audio, in MB, that the NIM queues when it cannot consume audio as fast as it arrives. Exceeding this cap fails the request with RESOURCE_EXHAUSTED.

NV_AI4M_ASD_MAX_DIARIZATION_INPUT_BUFFER_SEGMENTS

1000000

Maximum number of diarization segments the NIM queues when they arrive faster than they are consumed. Exceeding this cap fails the request with RESOURCE_EXHAUSTED.

NV_AI4M_ASD_MAX_VIDEO_QUEUE_SIZE

60

Maximum number of decoded video frames that can be queued. If the queue is full and diarization is still incomplete, the NIM continues with the available data (refer to Diarization Miss Policy) instead of blocking decode.

Diarization Miss Policy#

When diarization coverage is not reached within the budget, the NV_AI4M_ASD_DIARIZATION_TIMEOUT_MODE environment variable controls the behavior:

Mode

Behavior

adaptive (default)

Never latch single-channel on a miss. Keep using whatever diarization is buffered and re-adopt late segments; only a genuine absence of diarization uses the single-channel floor.

single_speaker

Fall back to a single active speaker and continue processing with the available speaker data.

terminate

Abort the request with an error.

For low-latency adaptive streams, you can set a one-time warmup barrier with NV_AI4M_ASD_DIARIZATION_FIRST_PACKET_TIMEOUT_S (default 0.0, disabled). This waits for the first diarization segment before the inference loop starts, which absorbs upstream warmup and avoids a frame-0 race that would mislatch the single-channel fallback.

For more information about Maxine NIM clients, refer to the GitHub repository NVIDIA-Maxine/nim-clients.