Basic Inference
-
Perform a health check on the gRPC endpoint.
-
Install
grpcurlfrom github.com/fullstorydev/grpcurl/releases.Example commands to run on Ubuntu:
-
Download the health checking proto:
-
Run the health check:
If the service is ready, you get a response similar to the following:
For using grpcurl with an SSL enabled server, avoid using the
--plaintextargument, and use--cacertwith a CA certificate,--keywith a private key, or--certwith a certificate file. For more details, refer togrpcurl --help. -
-
Download the Eye Contact Python client code by cloning the Clients repository (NVIDIA-Maxine/nim-clients):
-
Install the required dependencies:
Compile the Protos (Optional)
If you want to use the client code provided in the Clients repository (NVIDIA-Maxine/nim-clients), you can skip this step.
The proto files are available in the eye-contact/protos folder. You can compile them to generate client interfaces in your preferred programming language. For more details, refer to Supported languages in the gRPC documentation.
The following is an example of how to compile the protos for Python on Linux and Windows.
The grpcio version needed for compilation can be referred from requirements.txt
To compile protos on Linux:
To compile protos on Windows:
Input and Output
The input and output of the NVIDIA Eye Contact NIM are MP4 files. The input file must use the H.264 codec for video and can include audio. The output video file also uses the H.264 codec for video and contains the same audio as the input video file.
Videos with Variable Frame Rate (VFR) are not supported.
Input Modes
The NVIDIA Eye Contact NIM provides two modes for processing input files: streaming and transactional.
Streaming Mode
Streaming mode is the recommended way to use the NVIDIA Eye Contact NIM. It allows inference to begin without receiving the whole video from the client. It processes video frames incrementally, and inference begins as soon as the first frame of information is available. The output frames are streamed back to the client immediately after inference. This mode delivers the lowest latency and best resource efficiency, and it scales well to large files.
The Eye Contact NIM server automatically detects streamable videos and uses the streaming inference path internally. You do not need to enable any server-side setting or pass any flag to the NIM to get streaming behavior. Simply provide a streamable video as input.
Use streaming mode for these use cases:
- Best overall performance. The NIM is optimized for this path.
- Streamable video inputs.
- Applications that benefit from receiving output as it is generated, without waiting for the entire file to be uploaded.
- Large video files that benefit from incremental processing and reduced disk I/O.
Streaming mode works with streamable videos in which metadata is positioned at the beginning of the file. Videos that are not streamable can be easily converted to a streamable format.
To make any video streamable, use FFmpeg with the following command:
To run the NVIDIA Eye Contact NIM in streaming mode, go to the scripts directory and pass the streamable video as input:
The --streaming flag in the sample client is optional and does not change the inference behavior on the server. It only enables streaming-mode logging on the client side. The server automatically detects streamable inputs and uses the streaming inference path internally regardless of whether the flag is passed.
Transactional Mode
In transactional mode, the NIM receives the entire video file before processing can begin. The server automatically falls back to this mode when the input video is not streamable.
Transactional mode applies to the following use cases:
- Videos that are not optimized for streaming (such as non-streamable MP4 files in which metadata is located at the end of the file, requiring the entire file to be downloaded before playback can begin).
- Processing of small video files for which streaming overhead is unnecessary.
- Applications that can wait for complete processing before receiving output.
To run the NVIDIA Eye Contact NIM with a non-streamable input, pass the file as input. The server detects that the video is not streamable and processes it in transactional mode automatically:
For best performance, convert your videos to a streamable format and use streaming mode. See the FFmpeg command earlier on this page.
The first inference is not indicative of the model’s actual performance because it includes the time taken by the Triton Inference Server to load the models in addition to the time required to process the inference request.
Usage for Preview API Request
To view details of command-line arguments, run this command:
You get a response similar to the following. All parameters are optional.
Example Commands
Recommended: Provide a streamable video as input. The server automatically uses the streaming inference path.
Optional: Specify --streaming to enable streaming-mode logging on the client side. This does not change server behavior.
Non-streamable input. The server automatically falls back to transactional mode.
Advanced configuration example:
Python Notebook
For an interactive experience and to explore all feature parameters, we provide a comprehensive Python notebook that demonstrates the Eye Contact service capabilities.
The Python notebook is located at examples/maxine-eye-contact.ipynb in the eye-contact client folder.