Skip to navigation

Basic Inference

View as MarkdownOpen in Claude
  1. Perform a health check on the gRPC endpoint.

    • Install grpcurl from github.com/fullstorydev/grpcurl/releases.

      Example commands to run on Ubuntu:

      wget https://github.com/fullstorydev/grpcurl/releases/download/v1.9.1/grpcurl_1.9.1_linux_amd64.deb
      sudo dpkg -i grpcurl_1.9.1_linux_amd64.deb
    • Download the health checking proto:

      wget https://raw.githubusercontent.com/grpc/grpc/master/src/proto/grpc/health/v1/health.proto
    • Run the health check:

      grpcurl --plaintext --proto health.proto localhost:8001 grpc.health.v1.Health/Check

      If the service is ready, you get a response similar to the following:

      { "status": "SERVING" }

    For using grpcurl with an SSL enabled server, avoid using the --plaintext argument, and use --cacert with a CA certificate, --key with a private key, or --cert with a certificate file. For more details, refer to grpcurl --help.

  2. Download the Eye Contact Python client code by cloning the Clients repository (NVIDIA-Maxine/nim-clients):

    git clone https://github.com/NVIDIA-Maxine/nim-clients.git
    # Go to the 'eye-contact' folder
    cd nim-clients/eye-contact/
  3. Install the required dependencies:

    sudo apt-get install python3-pip
    pip install -r requirements.txt

Compile the Protos (Optional)

If you want to use the client code provided in the Clients repository (NVIDIA-Maxine/nim-clients), you can skip this step.

The proto files are available in the eye-contact/protos folder. You can compile them to generate client interfaces in your preferred programming language. For more details, refer to Supported languages in the gRPC documentation.

The following is an example of how to compile the protos for Python on Linux and Windows. The grpcio version needed for compilation can be referred from requirements.txt

To compile protos on Linux:

# Go to eye-contact/protos/linux folder
cd eye-contact/protos/linux
chmod +x compile_protos.sh
./compile_protos.sh

To compile protos on Windows:

# Go to eye-contact/protos/windows folder
cd eye-contact/protos/windows
./compile_protos.bat

Input and Output

The input and output of the NVIDIA Eye Contact NIM are MP4 files. The input file must use the H.264 codec for video and can include audio. The output video file also uses the H.264 codec for video and contains the same audio as the input video file.

Videos with Variable Frame Rate (VFR) are not supported.

Input Modes

The NVIDIA Eye Contact NIM supports two distinct input processing modes based on the input file capabilities:

Transactional Mode (Default)

In transactional mode, the entire input video file must be received and processed as a complete unit by the NIM before returning results.

This mode is suitable for the following use cases:

  • Processing of small video files, as these files are copied to NIM in entirety before inference can begin.
  • Applications that can wait for complete processing before receiving output.
  • Videos that are not optimized for streaming.

To run the NVIDIA Eye Contact NIM in transactional mode, simply run the NVIDIA Eye Contact sample client without any additional flags:

Go to the scripts directory:

cd scripts

Send a gRPC request:

python eye-contact.py --target <server_ip:port> \
--input <input file path> \
--output <output file path along with file name>

Streaming Mode

In streaming mode, the NIM can process incoming video frames incrementally as soon as frame information is available. The output frames are streamed back to the client immediately after processing.

This is the preferred mode for the following use cases:

  • Streamable video inputs.
  • Applications that do not want to wait until the entire file is completely uploaded.
  • Large video files that can benefit from incremental processing and thereby not constrained by the disk space on the server.
AspectTransactional ModeStreaming Mode
Data StorageEntire video and audio files are temporarily copied on diskOnly frames being processed are temporarily copied in memory
Processing StartNIM waits to receive entire files before startingNIM starts processing as soon as data chunk for first frame arrives
Processing TimingProcessing begins after all data is receivedContinuous processing without waiting for complete input
Output DeliveryComplete output video is returned to client after inference is finished for whole videoOutput frames are generated and returned immediately

The --streaming option works with streamable videos where metadata is positioned at the beginning of the file. Videos that are not streamable can be easily converted to a streamable format.

To make any video streamable, use FFmpeg with the following command:

ffmpeg -i sample_video.mp4 -movflags +faststart sample_streamable.mp4

You can then specify the streamable video as input to the NIM by using the --input parameter and enabling the --streaming option.

python eye-contact.py --target <server_ip:port> \
--input <input file path> \
--output <output file path along with file name> \
--streaming

The first inference is not indicative of the model’s actual performance because it includes the time taken by the Triton Inference Server to load the models in addition to the time required to process the inference request.

Usage for Preview API Request

python eye-contact.py --preview-mode \
--target grpc.nvcf.nvidia.com:443 \
--function-id <function_id> \
--api-key $API_KEY_REQUIRED_IF_EXECUTING_OUTSIDE_NGC \
--input <input_file_path> \
--output <output_file_path>

To view details of command-line arguments, run this command:

python eye-contact.py -h

You get a response similar to the following. All parameters are optional.

options:
-h, --help show this help message and exit
--preview-mode Flag to send request to preview NVCF NIM server on https://build.nvidia.com/nvidia/eyecontact/api. (default: False)
--ssl-mode {DISABLED,MTLS,TLS} Flag to set SSL mode, default is DISABLED (default: DISABLED)
--ssl-key SSL_KEY The path to ssl private key. (default: ../ssl_key/ssl_key_client.pem)
--ssl-cert SSL_CERT The path to ssl certificate chain. (default: ../ssl_key/ssl_cert_client.pem)
--ssl-root-cert SSL_ROOT_CERT The path to ssl root certificate. (default: ../ssl_key/ssl_ca_cert.pem)
--target TARGET IP:port of gRPC service, when hosted locally. Use grpc.nvcf.nvidia.com:443 when hosted on NVCF. (default: 127.0.0.1:8001)
--api-key API_KEY NGC API key required for authentication, utilized when using TRY API ignored otherwise (default: None)
--function-id FUNCTION_ID NVCF function ID for the service, utilized when using TRY API ignored otherwise (default: None)
--input INPUT The path to the input video file. (default: ../assets/sample_transactional.mp4)
--output OUTPUT The path for the output video file. (default: output.mp4)
--streaming Flag to enable grpc streaming mode. Required for streamable video input. (default: False)
--bitrate BITRATE Output video bitrate in bps (default: 20000000). This is only applicable when lossless mode is disabled. (default: 20000000)
--idr-interval IDR_INTERVAL The interval for IDR frames in the output video. This is only applicable when lossless mode is disabled. (default: 8) (default: 8)
--lossless Flag to enable lossless mode for video encoding. (default: False)
--custom-encoding-params CUSTOM_ENCODING_PARAMS Custom encoding parameters in JSON format. (default: None)
--temporal TEMPORAL Flag to control temporal filtering (default: 4294967295) (default: 4294967295)
--detect-closure DETECT_CLOSURE Flag to toggle detection of eye closure and occlusion on/off (default: 0) (default: 0)
--eye-size-sensitivity EYE_SIZE_SENSITIVITY Eye size sensitivity parameter (default: 3, range: [2, 6]) (default: 3)
--enable-lookaway {0,1} Flag to toggle look away on/off (default: 0) (default: 0)
--lookaway-max-offset LOOKAWAY_MAX_OFFSET Maximum value of gaze offset angle (degrees) during a random look away (default: 5, range: [1, 10]) (default: 5)
--lookaway-interval-min LOOKAWAY_INTERVAL_MIN Minimum limit for the number of frames at which random look away occurs (default: 3, range: [1, 600]) (default: 3)
--lookaway-interval-range LOOKAWAY_INTERVAL_RANGE Range for picking the number of frames at which random look away occurs (default: 8, range: [1, 600]) (default: 8)
--gaze-pitch-threshold-low GAZE_PITCH_THRESHOLD_LOW Gaze pitch threshold (degrees) at which the redirection starts transitioning (default: 20.0, range: [10, 35]) (default: 20.0)
--gaze-pitch-threshold-high GAZE_PITCH_THRESHOLD_HIGH Gaze pitch threshold (degrees) at which the redirection is equal to estimated gaze (default: 30.0, range: [10, 35]) (default: 30.0)
--gaze-yaw-threshold-low GAZE_YAW_THRESHOLD_LOW Gaze yaw threshold (degrees) at which the redirection starts transitioning (default: 20.0, range: [10, 35]) (default: 20.0)
--gaze-yaw-threshold-high GAZE_YAW_THRESHOLD_HIGH Gaze yaw threshold (degrees) at which the redirection is equal to estimated gaze (default: 30.0, range: [10, 35]) (default: 30.0)
--head-pitch-threshold-low HEAD_PITCH_THRESHOLD_LOW Head pose pitch threshold (degrees) at which the redirection starts transitioning away from camera towards estimated gaze (default: 15.0, range: [10, 35]) (default: 15.0)
--head-pitch-threshold-high HEAD_PITCH_THRESHOLD_HIGH Head pose pitch threshold (degrees) at which the redirection is equal to estimated gaze (default: 25.0, range: [10, 35]) (default: 25.0)
--head-yaw-threshold-low HEAD_YAW_THRESHOLD_LOW Head pose yaw threshold (degrees) at which the redirection starts transitioning (default: 25.0, range: [10, 35]) (default: 25.0)
--head-yaw-threshold-high HEAD_YAW_THRESHOLD_HIGH Head pose yaw threshold (degrees) at which the redirection is equal to estimated gaze (default: 30.0, range: [10, 35]) (default: 30.0)

Example Commands

Basic inference with default settings:

python eye-contact.py --target 127.0.0.1:8001 --input ../assets/sample_transactional.mp4 --output output.mp4

Using streaming mode for streamable videos:

python eye-contact.py --target 127.0.0.1:8001 --input ../assets/sample_streamable.mp4 --output output.mp4 --streaming

Advanced configuration example:

python eye-contact.py --target 127.0.0.1:8001 --input ../assets/sample_transactional.mp4 --output output.mp4 --custom-encoding-params '{"control-rate": "1", "bitrate": 3000000, "tuning-info-id": "3", "temporalaq": true}'

Python Notebook

For an interactive experience and to explore all feature parameters, we provide a comprehensive Python notebook that demonstrates the Eye Contact service capabilities.

The Python notebook is located at examples/maxine-eye-contact.ipynb in the eye-contact client folder.