Overview

NVIDIA Maxine Eye Contact NIM leverages state-of-the-art AI models to dynamically redirect a user’s eye position towards the camera in real-time to simulate natural eye contact and enhance remote digital engagement. NVIDIA Maxine Eye Contact NIM models are built on the NVIDIA software platform, incorporating CUDA, TensorRT, and Triton to offer out-of-the-box GPU acceleration.

NVIDIA Maxine Eye Contact operates on a region of interest around the eyes, also known as the eye patch. The eye patch is extracted from a video frame using the NVIDIA Maxine face tracking pipeline, which computes the 2D facial landmarks and the 6DOF head pose from the video frame. This head pose is then fed into the eye contact network.

The eye contact network has a disentangled encoder-decoder architecture. The encoder estimates the gaze angle from the input eye patch along with a set of features, also known as embeddings. Based on these embeddings, the decoder performs redirection of the gaze in the input patch to make the face look forward.

The final stage of the pipeline involves blending the eye patch back into the original video frame using an inverse transformation. More details on the model can be found here.

NVIDIA Maxine Eye Contact NIM can be tried out at this link.