Advanced Usage#

This page describes deployment options, request parameters, and runtime behavior for the Eye Contact NIM.

Model Profiles and Selection#

A NIM model profile defines which model engines the NIM can use and the criteria it uses to choose them. Each profile is identified by a unique hash of its contents.

You can select a profile at deployment time by following the Profile Selection steps. If you do not set a profile, the NIM chooses one that matches the GPU compute capability.

Model profiles are embedded within the NIM container in a Model Manifest file, which is by default placed at /etc/nim/config/model_manifest.yaml within the container filesystem.

The following table lists model profile IDs that you can specify in NIM_MODEL_PROFILE.

GPU Architecture (compute capability)	Model Profile ID
Blackwell (cc 12.0)	eb02786152e604d9495f891e38b2da550db36fa2c9e91d25211b0c1c8c9d6175
Ada (cc 8.9)	85f1043c43b43b954053946f6649e6a24f57a125e74ee656668d7889d0f9a186
Ampere (cc 8.6)	95ab357f947b3020e1ead7b5e039740d6a24438a83c3a6dcb70e4d4a96aabe10
Turing (cc 7.5)	fabc73f93374ed9ad5069e6e05a0a95ad9e28e1ab65b3a235199b53320c77a3c

Profile Selection#

To select a model profile for deployment, set a specific profile ID with -e NIM_MODEL_PROFILE=<value>. You can find the valid profile IDs by using the list-model-profiles utility:

docker run --rm --runtime=nvidia -e NGC_API_KEY=$NGC_API_KEY --gpus=all \
  nvcr.io/nim/nvidia/maxine-eye-contact:latest list-model-profiles

The utility prints the detected system information followed by the profiles that are compatible and incompatible with that hardware. Each entry lists the profile ID followed by its tags. The following example shows the output on an NVIDIA RTX 6000 Ada Generation (compute capability 8.9) system, where only the matching profile is compatible:

SYSTEM INFO
- Free GPUs:
  -  [26b1:10de] (0) NVIDIA RTX 6000 Ada Generation [current utilization: 1%]
MODEL PROFILES
- Compatible with system:
    - 85f1043c43b43b954053946f6649e6a24f57a125e74ee656668d7889d0f9a186 - backend:trt|compute_capability:8.9|model_type:trt|pp:1|precision:fp32|tp:1
- Incompatible with system:
    - 95ab357f947b3020e1ead7b5e039740d6a24438a83c3a6dcb70e4d4a96aabe10 - backend:trt|compute_capability:8.6|model_type:trt|pp:1|precision:fp32|tp:1
    - eb02786152e604d9495f891e38b2da550db36fa2c9e91d25211b0c1c8c9d6175 - backend:trt|compute_capability:12.0|model_type:trt|pp:1|precision:fp32|tp:1
    - fabc73f93374ed9ad5069e6e05a0a95ad9e28e1ab65b3a235199b53320c77a3c - backend:trt|compute_capability:7.5|model_type:trt|pp:1|precision:fp32|tp:1

Then run the NIM launch command:

docker run -it --rm --name=maxine-eye-contact-nim \
  --runtime=nvidia \
  --gpus all \
  --shm-size=8GB \
  -e NGC_API_KEY=$NGC_API_KEY \
  -e NIM_MODEL_PROFILE=$MODEL_PROFILE_ID \
  -e NIM_HTTP_API_PORT=8000 \
  -e NV_AI4M_MAX_CONCURRENCY_PER_GPU=1 \
  -p 8000:8000 \
  -p 8001:8001 \
  nvcr.io/nim/nvidia/maxine-eye-contact:latest

Note

NIM_MODEL_PROFILE is an optional parameter. If the model profile ID is not supplied, the NIM automatically selects a matching profile ID based on the target hardware architecture.

However, if NIM_MODEL_PROFILE is used, ensure that the associated GPU architecture is compatible with the target hardware. If an incorrect model profile ID is used, a deserialization error occurs on inference.

Model Caching#

When the container starts for the first time, it downloads the required models from NGC. To avoid downloading the models on subsequent runs, you can cache them locally by using a cache directory:

# Create the cache directory on the host machine
export LOCAL_NIM_CACHE=~/.cache/nim
mkdir -p "$LOCAL_NIM_CACHE"
chmod 777 $LOCAL_NIM_CACHE

# Choose model profile id based on target architecture.
export MODEL_PROFILE_ID=<enter_valid_model_profile_id>

# Run the container with the cache directory mounted in the appropriate location
docker run -it --rm --name=maxine-eye-contact-nim \
  --runtime=nvidia \
  --gpus all \
  --shm-size=8GB \
  -e NGC_API_KEY=$NGC_API_KEY \
  -e NIM_MODEL_PROFILE=$MODEL_PROFILE_ID \
  -e NIM_HTTP_API_PORT=8000 \
  -p 8000:8000 \
  -p 8001:8001 \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  nvcr.io/nim/nvidia/maxine-eye-contact:latest

For more information about MODEL_PROFILE_ID, refer to Model Profiles and Selection.

SSL Enablement#

Eye-Contact NIM provides an SSL mode to ensure secure communication between clients and the server by encrypting data in transit. To enable SSL, you need to provide the path to the SSL certificate and key files in the container. The following example shows how to do this:

export NGC_API_KEY=<add-your-api-key>
SSL_CERT=path/to/ssl_key

docker run -it --rm --name=maxine-eye-contact-nim \
  --runtime=nvidia \
  --gpus all \
  --shm-size=8GB \
  -v $SSL_CERT:/opt/nim/crt/:ro \
  -e NGC_API_KEY=$NGC_API_KEY \
  -p 8000:8000 \
  -p 8001:8001\
  -e NIM_SSL_MODE="mtls" \
  -e NIM_SSL_CERT_PATH="/opt/nim/crt/ssl_cert_server.pem" \
  -e NIM_SSL_KEY_PATH="/opt/nim/crt/ssl_key_server.pem" \
  nvcr.io/nim/nvidia/maxine-eye-contact:latest

NIM_SSL_MODE can be set to “mtls”, “tls”, or “disabled”. If set to “mtls”, the container uses mutual TLS authentication. If set to “tls”, the container uses TLS authentication. For more information, refer to NIM SSL Configuration.

Be sure to verify the permissions of the SSL certificate and key files on the host machine. The container will not be able to access the files if they are not readable by the user running the container.

Multiple Concurrent Inputs#

To run the server in multi-input concurrent mode, set the environment variable NV_AI4M_MAX_CONCURRENCY_PER_GPU to an integer greater than 1 in the server container. The server will then accept as many concurrent inputs per GPU as specified by the NV_AI4M_MAX_CONCURRENCY_PER_GPU variable.

Since Triton distributes the workload equally across all GPUs, if there are NUM_GPUS GPUs, the total number of concurrent inputs supported by the server will be NUM_GPUS * NV_AI4M_MAX_CONCURRENCY_PER_GPU.

The Eye Contact NIM uses NVENC/NVDEC hardware acceleration for video encoding and decoding.

GPUs without NVENC/NVDEC hardware support are not supported, including A100, H100, and B100 products.
Some GPUs support only a limited number of concurrent NVENC sessions, which means NIM can only process that same number of concurrent inputs on those GPUs.
Some GPUs support only certain YUV formats in h264 reading and writing.

For details, refer to the Video Encode and Decode GPU Support Matrix.

Note

If incoming requests to the NIM exceed the GPU’s maximum concurrent encode/decode limit, the processing fails.

Feature Parameters#

The following parameters are available to configure the functionality of the NIM.

Encoding Options#

lossless - Enables lossless video encoding. This setting overrides any bitrate configuration to ensure maximum quality output, though it results in larger file sizes.

python eye-contact.py --lossless

bitrate - Sets the target bitrate for video encoding in bits per second (bps). Higher bitrates result in better video quality but larger file sizes. When not specified, the server automatically selects a bitrate based on the input video resolution. For example, setting --bitrate 5000000 targets 5 Mbps encoding. This parameter only applies when lossless mode is disabled.

python eye-contact.py --bitrate 5000000

idr-interval - Sets the interval between IDR (Instantaneous Decoding Refresh) frames in the encoded video. IDR frames are special I-frames that clear all reference buffers, allowing the video to be decoded from that point without needing previous frames. A lower interval (e.g. 8 frames) provides better seeking/editing capabilities but increases file size, while a higher interval (e.g. 30 frames) reduces file size but may make seeking less precise. The default is 8 frames.

python eye-contact.py --idr-interval 10

Custom encoding - Allows passing custom encoding parameters as a JSON string. This provides fine-grained control over the encoding process.

python eye-contact.py --custom-encoding-params '{"idrinterval": "20", "maxbitrate": "3000000"}'

Note

Custom encoding parameters are for expert users who need fine-grained control over video encoding. Incorrect values may cause encoding failures or poor quality output. To configure the NVENC encoder please refer to Properties of the DeepStream H264 encoder plugin.

Arguments to Control Feature Behavior#

The following arguments affect the overall behavior of the feature, such as enabling or disabling temporal filtering or gaze redirection.

temporal - (UINT32) Flag to control temporal filtering (default 0xffffffff). When set to true, the landmark computation for eye contact is temporally optimized.
detect_closure - (UINT32) Flag to toggle detection of eye closure and occlusion. If turned off, blink and occlusion detection turns off. This might be desirable during estimation-only mode if you still want to obtain gaze estimation in case of occlusion. Not recommended for gaze redirection. Value is either 0 or 1 (default 0).
eye_size_sensitivity - (UINT32) Eye size sensitivity parameter that modifies the blending parameters to use a larger region around the eyes for blending. Integer value from 2 to 6 (default 3).

Multi-Person Mode#

By default, the NIM redirects the gaze of a single subject in the video. Enable multi-person mode to redirect the gaze of every detected face in the frame instead of a single subject. This is useful for scenes with multiple participants, such as group calls or panel recordings, where all faces should appear to look toward the camera.

Multi-person mode supports up to eight faces per video stream.

enable_multi_person - Flag to redirect gaze for every detected face in the frame instead of a single subject. Pass --enable-multi-person to turn it on; when the flag is omitted, gaze is redirected for a single subject (default).

To try multi-person mode with the sample client, pass the flag along with a video that contains multiple faces:

python eye-contact.py --target 127.0.0.1:8001 --input <path_to_video_with_multiple_faces> --enable-multi-person --output output.mp4

Randomized Look Away Parameters#

A continuous redirection of gaze to look at the camera might give a perception of staring. Some users might find this effect unnatural or undesired. To occasionally break eye contact, you can enable randomized look away in gaze redirection.

Although the gaze is always expected to redirect toward the camera within the range of operation, enabling look away makes the user occasionally break gaze lock to the camera with a micro-movement of the eyes at randomly chosen time intervals. The enable_lookaway parameter must be set to true to enable this feature. Additionally, you can use the optional parameters lookaway_max_offset, lookaway_interval_min, and lookaway_interval_range to tune the extent and frequency of look away.

enable_lookaway - (UINT32) Flag to toggle look away. If set to on, the eyes are redirected to look away for a random period occasionally to avoid staring. Value is either 0 or 1 (default 0).
lookaway_max_offset - (UINT32) Maximum value of gaze offset angle (degrees) during a random look away when look away is enabled. Requires the enable_lookaway parameter to be set to true. Integer value from 1 to 10 (default 5).
lookaway_interval_min - (UINT32) Minimum limit for the number of frames at which random look away occurs when look away is enabled. Requires the enable_lookaway parameter to be set to true. Integer value from 1 to 600 (default 3).
lookaway_interval_range - (UINT32) Range for picking the number of frames at which random look away occurs when look away is enabled. Requires the enable_lookaway parameter to be set to true. Integer value from 1 to 600 (default 8).

Range Control#

The gaze redirection feature redirects the eyes to look at the camera within a certain range of head and eye motion in which eye contact is desired and looks natural. Beyond this range, the feature gradually transitions away from looking at the camera toward the estimated gaze and eventually turns off in a seamless manner.

To provide for various use cases and user preferences, we provide range parameters for the user to control the range of gaze angles and head poses in which gaze redirection occurs and the range in which transition occurs before the redirection is turned off. These are optional parameters.

gaze_pitch_threshold_low and gaze_yaw_threshold_low define the parameters for the pitch and yaw angles of the estimated gaze within which gaze is redirected toward the camera. Beyond these angles, redirected gaze transitions away from the camera and toward the estimated gaze, turning off redirection beyond gaze_pitch_threshold_high and gaze_yaw_threshold_high respectively.

Similarly, head_pitch_threshold_low and head_yaw_threshold_low define the parameters for pitch and yaw angles of the head pose within which gaze is redirected toward the camera. Beyond these angles, redirected gaze transitions away from the camera and toward the estimated gaze, turning off redirection beyond head_pitch_threshold_high and head_yaw_threshold_high.

gaze_pitch_threshold_low - (FP32) Gaze pitch threshold (degrees) at which the redirection starts transitioning away from camera toward estimated gaze. Float value from 10 to 35 (default 20).
gaze_pitch_threshold_high - (FP32) Gaze pitch threshold (degrees) at which the redirection is equal to estimated gaze and the gaze redirection is turned off beyond this angle. Float value from 10 to 35 (default 30).
gaze_yaw_threshold_low - (FP32) Gaze yaw threshold (degrees) at which the redirection starts transitioning away from camera toward estimated gaze. Float value from 10 to 35 (default 20).
gaze_yaw_threshold_high - (FP32) Gaze yaw threshold (degrees) at which the redirection is equal to estimated gaze and the gaze redirection is turned off beyond this angle. Float value from 10 to 35 (default 30).
head_pitch_threshold_low - (FP32) Head pose pitch threshold (degrees) of the estimated head pose at which redirection starts transitioning away from camera and toward the estimated gaze. Float value from 10 to 35 (default 15).
head_pitch_threshold_high - (FP32) Head pose pitch threshold (degrees) of the estimated head pose at which redirection equals the estimated gaze and redirection is turned off beyond this angle. Float value from 10 to 35 (default 25).
head_yaw_threshold_low - (FP32) Head pose yaw threshold (degrees) at which the redirection starts transitioning away from camera toward estimated gaze. Float value from 10 to 35 (default 25).
head_yaw_threshold_high - (FP32) Head pose yaw threshold (degrees) of the estimated head pose at which redirection equals the estimated gaze and redirection is turned off beyond this angle. Float value from 10 to 35 (default 30).

For more details on the technical aspects of this algorithm, refer to the technical blog post Improve Human Connection in Video Conferences with NVIDIA Maxine Eye Contact.