> This page is for NVIDIA Active Speaker Detection NIM, version 1.2.0 (Latest) (default).
> For other versions, use one of these documentation indexes:
> - 1.2.0 (Latest) (default): https://docs.nvidia.com/nim/maxine/active-speaker-detection/1.2.0/llms.txt
> - 1.1.0: https://docs.nvidia.com/nim/maxine/active-speaker-detection/1.1.0/llms.txt
> - 1.0.0: https://docs.nvidia.com/nim/maxine/active-speaker-detection/1.0.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nim/maxine/llms.txt.

# Advanced Usage

This page describes deployment options, request parameters, and runtime behavior for the Active Speaker Detection NIM.

## Model Profiles and Selection

A NIM model profile defines which model engines the NIM can use and the criteria it uses to choose them. Each profile is identified by a unique hash of its contents.

You can select a profile at deployment time by following the [Profile Selection](#profile-selection) steps. If you do not set a profile, the NIM chooses one that matches the GPU compute capability.

Model profiles are embedded within the NIM container in a Model Manifest file, which is by default placed at `/etc/nim/config/model_manifest.yaml` within the container filesystem.

The following table lists model profile IDs that you can specify in `NIM_MODEL_PROFILE`.

| GPU Architecture (compute capability) | Model Profile ID                                              |
| ------------------------------------- | ---------------------------------------------------------------- |
| Blackwell (cc 12.0)                   | eb02786152e604d9495f891e38b2da550db36fa2c9e91d25211b0c1c8c9d6175 |
| Blackwell (cc 10.0)                   | f1d12a64b41e0a9972ba7b361437c0367e08c5df79a04ca01d8af6581deda1a0 |
| Hopper (cc 9.0)                       | d8e49a40fcf78248e355a542405cbd9ccb975ac79f5c07bb2775975017c979c7 |
| Ada (cc 8.9)                          | 85f1043c43b43b954053946f6649e6a24f57a125e74ee656668d7889d0f9a186 |
| Ampere (cc 8.6)                       | d3c85f2b926806c6a307b3ed08421a315658b4d9ce3dea73ad47ad1ab055d9a9 |
| Ampere (cc 8.0)                       | f4a75f436f19b3e4728c334240d5b2ff15b102ec953ee7d1cb231c9fc93dd3d3 |
| Turing (cc 7.5)                       | fabc73f93374ed9ad5069e6e05a0a95ad9e28e1ab65b3a235199b53320c77a3c |

### Profile Selection

To select a model profile for deployment, set a specific profile ID with `-e NIM_MODEL_PROFILE=<value>`. You can find the valid profile IDs by using the `list-model-profiles` utility:

```bash
docker run --rm --runtime=nvidia -e NGC_API_KEY=$NGC_API_KEY --gpus=all \
  nvcr.io/nim/nvidia/active-speaker-detection:latest list-model-profiles
```

The utility prints the detected system information followed by the profiles that are compatible and incompatible with that hardware. Each entry lists the profile ID followed by its tags. The following example shows the output on an Ada (compute capability 8.9) system, where only the matching profile is compatible:

```output
SYSTEM INFO
- Free GPUs:
  -  [26b9:10de] (0) NVIDIA L40S [current utilization: 0%]
MODEL PROFILES
- Compatible with system:
    - 85f1043c43b43b954053946f6649e6a24f57a125e74ee656668d7889d0f9a186 - backend:trt|compute_capability:8.9|model_type:trt|pp:1|precision:fp32|tp:1
- Incompatible with system:
    - eb02786152e604d9495f891e38b2da550db36fa2c9e91d25211b0c1c8c9d6175 - backend:trt|compute_capability:12.0|model_type:trt|pp:1|precision:fp32|tp:1
    - f1d12a64b41e0a9972ba7b361437c0367e08c5df79a04ca01d8af6581deda1a0 - backend:trt|compute_capability:10.0|model_type:trt|pp:1|precision:fp32|tp:1
    - d8e49a40fcf78248e355a542405cbd9ccb975ac79f5c07bb2775975017c979c7 - backend:trt|compute_capability:9.0|model_type:trt|pp:1|precision:fp32|tp:1
    - d3c85f2b926806c6a307b3ed08421a315658b4d9ce3dea73ad47ad1ab055d9a9 - backend:trt|compute_capability:8.6|pp:1|precision:fp32|tp:1
    - f4a75f436f19b3e4728c334240d5b2ff15b102ec953ee7d1cb231c9fc93dd3d3 - backend:trt|compute_capability:8.0|model_type:trt|pp:1|precision:fp32|tp:1
    - fabc73f93374ed9ad5069e6e05a0a95ad9e28e1ab65b3a235199b53320c77a3c - backend:trt|compute_capability:7.5|model_type:trt|pp:1|precision:fp32|tp:1
```

Then run the NIM launch command:

```bash
docker run -it --rm --name=active-speaker-detection-nim \
  --runtime=nvidia \
  --gpus all \
  --shm-size=8GB \
  -e NGC_API_KEY=$NGC_API_KEY \
  -e NIM_MODEL_PROFILE=$MODEL_PROFILE_ID \
  -e NV_AI4M_MAX_CONCURRENCY_PER_GPU=1 \
  -e NIM_HTTP_API_PORT=8000 \
  -e NIM_GRPC_API_PORT=8001 \
  -p 8000:8000 \
  -p 8001:8001 \
  nvcr.io/nim/nvidia/active-speaker-detection:latest
```

<Note>
`NIM_MODEL_PROFILE` is an optional parameter. If the model profile ID is not supplied, the NIM automatically selects a matching profile ID based on the target hardware architecture.

However, if `NIM_MODEL_PROFILE` is used, ensure that the associated GPU architecture is compatible with the target hardware. If an incorrect model profile ID is used, a deserialization error occurs on inference.
</Note>

## Model Caching

When the container launches for the first time, it downloads the required models from NGC. To avoid downloading the models on subsequent runs, you can cache them locally by using a cache directory:

```bash
# Create the cache directory on the host machine.
export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p "$LOCAL_NIM_CACHE"
chmod 777 $LOCAL_NIM_CACHE

# Choose model profile ID based on target architecture.
export MODEL_PROFILE_ID=<enter_valid_model_profile_id>

# Run the container with the cache directory mounted in the appropriate location.
docker run -it --rm --name=active-speaker-detection-nim \
  --runtime=nvidia \
  --gpus all \
  --shm-size=8GB \
  -e NGC_API_KEY=$NGC_API_KEY \
  -e NIM_MODEL_PROFILE=$MODEL_PROFILE_ID \
  -e NIM_HTTP_API_PORT=8000 \
  -e NIM_GRPC_API_PORT=8001 \
  -p 8000:8000 \
  -p 8001:8001 \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  nvcr.io/nim/nvidia/active-speaker-detection:latest
```

For more information about `MODEL_PROFILE_ID`, refer to [Model Profiles and Selection](#model-profiles-and-selection).

## SSL Enablement

Active Speaker Detection NIM provides an SSL mode to ensure secure communication between clients and the server by encrypting data in transit.
To enable SSL, you must provide the path to the SSL certificate and key files in the container. The following example shows how to do this:

```bash
export NGC_API_KEY=<add-your-api-key>
export SSL_CERT=path/to/ssl_key

docker run -it --rm --name=active-speaker-detection-nim \
  --runtime=nvidia \
  --gpus all \
  --shm-size=8GB \
  -v $SSL_CERT:/opt/nim/crt/:ro \
  -e NGC_API_KEY=$NGC_API_KEY \
  -p 8000:8000 \
  -p 8001:8001 \
  -e NIM_SSL_MODE="mtls" \
  -e NIM_SSL_CA_CERTS_PATH="/opt/nim/crt/ssl_ca.pem" \
  -e NIM_SSL_CERT_PATH="/opt/nim/crt/ssl_cert_server.pem" \
  -e NIM_SSL_KEY_PATH="/opt/nim/crt/ssl_key_server.pem" \
  nvcr.io/nim/nvidia/active-speaker-detection:latest
```

`NIM_SSL_MODE` can be set to `mtls`, `tls`, or `disabled`. If set to `mtls`, the container uses mutual TLS authentication. If set to `tls`, the container uses TLS authentication.
For more information, refer to [Environment Variables](/nim/maxine/active-speaker-detection/getting-started#environment-variables).

Be sure to verify the permissions of the SSL certificate and key files on the host machine. The container cannot access the files if they are not readable by the user running the container.

## Multiple Concurrent Inputs

To run the server in multi-input concurrent mode, set the environment variable `NV_AI4M_MAX_CONCURRENCY_PER_GPU` to an integer greater than 1 in the server container. The server then accepts as many concurrent inputs per GPU as specified by the `NV_AI4M_MAX_CONCURRENCY_PER_GPU` variable.

Because Triton distributes the workload equally across all GPUs, the total number of concurrent inputs supported by the server is the number of GPUs multiplied by `NV_AI4M_MAX_CONCURRENCY_PER_GPU`.

<Note>
On bare-metal Linux systems with PCIe topology, refer to [PCIe Multi-GPU Systems](/nim/maxine/active-speaker-detection/advanced-usage/multi-gpu) for the recommended IOMMU configuration.
</Note>
The Active Speaker Detection NIM uses NVENC/NVDEC hardware acceleration for video decoding.

- GPUs without NVDEC hardware support might not be supported.
- Some GPUs support only a limited number of concurrent NVDEC sessions, which means the NIM can process only that same number of concurrent inputs on those GPUs.

For details, refer to the [Video Encode and Decode Support Matrix](https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new).

<Note>
If incoming requests to the NIM exceed the GPU's maximum concurrent decode limit, the processing might fail.
</Note>

## Maximum Speakers Configuration

The `NV_AI4M_ASD_MAX_SPEAKERS` environment variable controls the maximum number of audio speaker streams that the Active Speaker Detection NIM can handle simultaneously. This setting affects both the gRPC service and the Triton backend.

- **Default:** 5
- **Range:** 1 to 5

To configure the maximum number of speakers:

```bash
docker run -it --rm --name=active-speaker-detection-nim \
  --runtime=nvidia \
  --gpus all \
  --shm-size=8GB \
  -e NGC_API_KEY=$NGC_API_KEY \
  -e NV_AI4M_ASD_MAX_SPEAKERS=2 \
  -e NIM_HTTP_API_PORT=8000 \
  -e NIM_GRPC_API_PORT=8001 \
  -p 8000:8000 \
  -p 8001:8001 \
  nvcr.io/nim/nvidia/active-speaker-detection:latest
```

<Note>
The `NV_AI4M_ASD_MAX_SPEAKERS` value must match the number of speaker streams expected in your use case.
</Note>
## Service Information in the Response

Each response stream opens with a one-shot `ServiceInfo` message on the `service_info` field of
`DetectActiveSpeakerResponse`, before any detection results. It reports `feature_name`,
`feature_version`, `model_info`, `server_request_id`, and `client_session_id`. Quote
`server_request_id` when reporting an issue — it locates the request in the server logs.

The following example prints the server request ID, then detection results:

```python
for response in response_iterator:
    if response.HasField("service_info"):
        print(response.service_info.server_request_id)
    elif response.HasField("active_speaker_detection_result"):
        result = response.active_speaker_detection_result
        print(result.frame_id, len(result.speaker_data))
```

## Audio Source Configuration

The Active Speaker Detection NIM supports two modes for audio input feed, configured by the client at request time:

- **Separate Stream** (`AUDIO_SOURCE_CONFIG_SEPARATE_STREAM`): The client provides a separate audio file alongside the video. This is the default behavior. Use the `--audio-input` argument in the client to specify the audio file path.
- **Embedded in Video** (`AUDIO_SOURCE_CONFIG_EMBEDDED_IN_VIDEO`): Audio is demuxed from the supplied video container. No separate audio file is needed. Use the `--skip-audio` flag in the client.

### Separate Audio Stream

The following example sends a separate audio file with the video:

```bash
python active_speaker_detection.py --target 127.0.0.1:8001 \
   --video-input /path/to/video.mp4 \
   --audio-input /path/to/audio.wav \
   --diarization-input /path/to/diarization.json
```

### Embedded Audio

The following example uses the audio embedded in the video container:

```bash
python active_speaker_detection.py --target 127.0.0.1:8001 \
   --video-input /path/to/video_with_audio.mp4 \
   --diarization-input /path/to/diarization.json \
   --skip-audio
```

## Speaker Detection Threshold

The `speaker_detection_threshold` field in the `ActiveSpeakerDetectionConfig` proto controls the minimum confidence score required to classify a detected face as actively speaking.

- **Valid range:** `(0, 1)` open interval (exclusive of both endpoints).
- **Proto field:** `optional float speaker_detection_threshold` in `ActiveSpeakerDetectionConfig`.

<Note>
The sample client (`active_speaker_detection.py`) exposes this field through the
`--speaker-detection-threshold` command-line flag. You can also set
`speaker_detection_threshold` on the `ActiveSpeakerDetectionConfig` proto directly in your own
client code, as shown below.
</Note>

A lower threshold is more permissive, resulting in more faces being classified as speaking. A higher threshold is more strict, resulting in fewer faces classified as speaking. The face-detection bounding boxes themselves are not affected by this threshold—only the `is_speaking` classification changes.

To set the threshold, include the `speaker_detection_threshold` field in the `ActiveSpeakerDetectionConfig` message sent as the first request in the gRPC stream:

```python
detection_config = ActiveSpeakerDetectionConfig(
    input_video_config=video_config,
    input_audio_config=audio_config,
    audio_source_config=AUDIO_SOURCE_CONFIG_SEPARATE_STREAM,
    speaker_detection_threshold=0.75,
)
```

The value must be in the open interval `(0, 1)`. Omitting the field or setting it to `0` uses the configured default. Any other value outside the range (`< 0` or `>= 1`, such as `-0.5` or `1.5`) is rejected with an `INVALID_ARGUMENT` gRPC error.

## Voice Activity Smoothing for Diarization Workflows

The speaking state produced for diarized tracks can be temporally smoothed so that a track does not flicker between speaking and not-speaking on a frame-to-frame basis.
Smoothing is especially useful when diarization arrives incrementally, such as streaming or word-by-word diarization. Each packet covers only a short slice of time, so there are often gaps at word or segment boundaries, including brief silences and inter-word pauses. Without smoothing, those uncovered frames are treated as not speaking, and a continuous speaker flickers off and on. Temporal smoothing holds the speaking state across those short gaps.

Smoothing supports three levels:

- **`off`** — No smoothing. The raw per-frame speaking decision is returned.
- **`low`** — Light smoothing that removes brief single-frame flips.
- **`high`** *(default)* — Stronger smoothing that keeps a diarized speaker's state stable across short pauses and detection noise.

Smoothing is a deploy-time setting that can be overridden per request.

### Deploy-Time Configuration

Set the `NV_AI4M_ASD_VA_SMOOTHING` environment variable when launching the container to select the default smoothing level for all requests:

```bash
docker run -it --rm --name=active-speaker-detection-nim \
  --runtime=nvidia \
  --gpus all \
  --shm-size=8GB \
  -e NGC_API_KEY=$NGC_API_KEY \
  -e NV_AI4M_ASD_VA_SMOOTHING=high \
  -e NIM_HTTP_API_PORT=8000 \
  -e NIM_GRPC_API_PORT=8001 \
  -p 8000:8000 \
  -p 8001:8001 \
  nvcr.io/nim/nvidia/active-speaker-detection:latest
```

Valid values are `off`, `low`, and `high` (case-insensitive). If the variable is unset or invalid, the server falls back to `high`.

### Per-Request Override

An individual client request can override the deployment default with the `voice_activity_smoothing` field in the `ActiveSpeakerDetectionConfig` proto. The field uses the `VoiceActivitySmoothing` enum:

| Enum value                             | Effect                                              |
| -------------------------------------- | --------------------------------------------------- |
| `VOICE_ACTIVITY_SMOOTHING_UNSPECIFIED` | Use the deployment default                          |
| `VOICE_ACTIVITY_SMOOTHING_OFF`         | Disable smoothing for this request.                 |
| `VOICE_ACTIVITY_SMOOTHING_LOW`         | Low smoothing for this request.                     |
| `VOICE_ACTIVITY_SMOOTHING_HIGH`        | High smoothing for this request.                    |

The following example overrides the deployment default:

```python
detection_config = ActiveSpeakerDetectionConfig(
    input_video_config=video_config,
    input_audio_config=audio_config,
    audio_source_config=AUDIO_SOURCE_CONFIG_SEPARATE_STREAM,
    voice_activity_smoothing=VOICE_ACTIVITY_SMOOTHING_HIGH,
)
```

When the field is omitted or set to `VOICE_ACTIVITY_SMOOTHING_UNSPECIFIED`, the server applies the deploy-time default.

## Multi-View Speaker Identification

By default, each video stream is processed as an isolated session, so the same physical person appearing in two concurrent camera streams is not correlated across those streams. Enabling the multi-view model correlates the same people across concurrent camera streams, so tracking IDs stay consistent and a speaker can be followed as they move between views.

<Note>
Multi-view is a deploy-time selection. When it is disabled (the default), each stream is treated as an independent session.
</Note>
To enable cross-stream identification, set `NV_AI4M_ASD_MULTIVIEW=true` when launching the container. This selects the multi-view inference model, which shares the same input and output tensors as the single-view model:

```bash
docker run -it --rm --name=active-speaker-detection-nim \
  --runtime=nvidia \
  --gpus all \
  --shm-size=8GB \
  -e NGC_API_KEY=$NGC_API_KEY \
  -e NV_AI4M_ASD_MULTIVIEW=true \
  -e NIM_HTTP_API_PORT=8000 \
  -e NIM_GRPC_API_PORT=8001 \
  -p 8000:8000 \
  -p 8001:8001 \
  nvcr.io/nim/nvidia/active-speaker-detection:latest
```
<Note>
Streams that need to share tracking IDs must be submitted concurrently within the same session. Only concurrent streams are batched and processed together as a single session, which is what enables ID sharing.
</Note>
### Output Semantics in Multi-View Mode

Multi-view mode changes only the meaning of `tracking_id`:

- **Default model:** IDs are local to each stream.
- **Multi-view mode:** IDs are assigned from a shared face gallery across streams in the Triton backend process.

The output shape is unchanged.

The speaking-state fields remain stream-local. The same global `tracking_id` can have different `is_speaking`, `audio_id`, and `confidence` values on different streams when the views differ in face angle, size, occlusion, or audio-visual sync evidence.

## Input Stream Handling

The Active Speaker Detection NIM ingests video, audio, and diarization streams on independent feeder threads. Each input stream is decoupled from the request thread, so a stalled or late stream does not block the request thread or starve the other input streams. This behavior is governed by per-unit coverage timeouts, buffer caps, and an explicit diarization miss policy.

### Per-Unit Coverage Timeouts

Each input stream has its own per-unit wait budget. The budget is applied to each unit of work (for example, each decoded video frame), not a single time for the whole request.

| Environment Variable                | Default | Description                                                              |
| ----------------------------------- | ------- | ------------------------------------------------------------------------ |
| `NV_AI4M_ASD_VIDEO_TIMEOUT_S`       | `30.0`  | Per-unit budget for the video stream to cover a frame in seconds. The same budget bounds the wait for the first decoded frame. |
| `NV_AI4M_ASD_AUDIO_TIMEOUT_S`       | `30.0`  | Per-unit budget waiting for audio samples before the audio stream times out in seconds. |
| `NV_AI4M_ASD_DIARIZATION_TIMEOUT_S` | `30.0`  | Per-unit wait budget for diarization coverage in seconds. Applies only when diarization is included in the request. |

Each budget must be greater than zero. A `0`, negative, or non-numeric value is rejected at startup: the NIM logs a warning and uses the `30.0` second default instead. A literal `zero` second budget would expire before any producer could deliver the first unit, failing every request.

### Buffer Caps

Each feeder bounds how much input data it will hold per stream while its queued for processing. These caps are an out-of-memory backstop, not flow control: a stream only accumulates when its reader has stopped consuming, and exceeding the cap fails the request with a `RESOURCE_EXHAUSTED` gRPC error rather than blocking the request thread (which would stall every other stream).

<Note>
Exercise caution when raising concurrency together with these buffer caps. Caps apply **per request**, so total host memory scales with `NV_AI4M_MAX_CONCURRENCY_PER_GPU` and the number of GPUs. At high concurrency, lower the caps so a stalled reader cannot exhaust host memory, and keep timeout budgets consistent with the reduced buffers. Raising both concurrency and cap values without reviewing memory headroom is not recommended.
</Note>

| Environment Variable                                | Default   | Description |
| --------------------------------------------------- | --------- | ----------- |
| `NV_AI4M_ASD_MAX_VIDEO_INPUT_BUFFER_MB`             | `2048`    | Maximum encoded video, in MB, that the NIM queues when it cannot decode video as fast as it arrives. Exceeding this cap fails the request with `RESOURCE_EXHAUSTED`. |
| `NV_AI4M_ASD_MAX_AUDIO_INPUT_BUFFER_MB`             | `512`     | Maximum encoded audio, in MB, that the NIM queues when it cannot consume audio as fast as it arrives. Exceeding this cap fails the request with `RESOURCE_EXHAUSTED`. |
| `NV_AI4M_ASD_MAX_DIARIZATION_INPUT_BUFFER_SEGMENTS` | `1000000` | Maximum number of diarization segments the NIM queues when they arrive faster than they are consumed. Exceeding this cap fails the request with `RESOURCE_EXHAUSTED`. |
| `NV_AI4M_ASD_MAX_VIDEO_QUEUE_SIZE`                  | `60`      | Maximum number of decoded video frames that can be queued. If the queue is full and diarization is still incomplete, the NIM continues with the available data (refer to [Diarization Miss Policy](#diarization-miss-policy)) instead of blocking decode. |

### Diarization Miss Policy

When diarization coverage is not reached within the budget, the `NV_AI4M_ASD_DIARIZATION_TIMEOUT_MODE` environment variable controls the behavior:

| Mode              | Behavior                                                                                                                                 |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `adaptive` *(default)*    | Never latch single-channel on a miss. Keep using whatever diarization is buffered and re-adopt late segments; only a genuine absence of diarization uses the single-channel floor. |
| `single_speaker`  | Fall back to a single active speaker and continue processing with the available speaker data.                                 |
| `terminate`       | Abort the request with an error.                                                                                                         |

For low-latency `adaptive` streams, you can set a one-time warmup barrier with `NV_AI4M_ASD_DIARIZATION_FIRST_PACKET_TIMEOUT_S` (default `0.0`, disabled). This waits for the first diarization segment before the inference loop starts, which absorbs upstream warmup and avoids a frame-0 race that would mislatch the single-channel fallback.

For more information about Maxine NIM clients, refer to the GitHub repository [NVIDIA-Maxine/nim-clients](https://github.com/NVIDIA-Maxine/nim-clients).