> This page is for NVIDIA Eye Contact NIM, version 1.2.0.
> For other versions, use one of these documentation indexes:
> - 1.5.0 (Latest) (default): https://docs.nvidia.com/nim/maxine/eye-contact/1.5.0/llms.txt
> - 1.4.0: https://docs.nvidia.com/nim/maxine/eye-contact/1.4.0/llms.txt
> - 1.3.0: https://docs.nvidia.com/nim/maxine/eye-contact/1.3.0/llms.txt
> - 1.2.0: https://docs.nvidia.com/nim/maxine/eye-contact/1.2.0/llms.txt
> - 1.1.0: https://docs.nvidia.com/nim/maxine/eye-contact/1.1.0/llms.txt
> - 1.0.0: https://docs.nvidia.com/nim/maxine/eye-contact/1.0.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nim/maxine/llms.txt.

# Basic Inference

1. Perform a health check on the gRPC endpoint.

   - Install `grpcurl` from [github.com/fullstorydev/grpcurl/releases](https://github.com/fullstorydev/grpcurl/releases).

      Example commands to run on Ubuntu:
      ```bash
      wget https://github.com/fullstorydev/grpcurl/releases/download/v1.9.1/grpcurl_1.9.1_linux_amd64.deb
      sudo dpkg -i grpcurl_1.9.1_linux_amd64.deb
      ```

   - Download the health checking proto:
      ```bash
      wget https://raw.githubusercontent.com/grpc/grpc/master/src/proto/grpc/health/v1/health.proto
      ```

   - Run the health check:
      ```bash
      grpcurl --plaintext --proto health.proto localhost:8001 grpc.health.v1.Health/Check
      ```

     If the service is ready, you get a response similar to the following:
      ```bash
      { "status": "SERVING" }
      ```

   <Note>
   For using grpcurl with an SSL enabled server, avoid using the `--plaintext` argument, and use `--cacert` with a CA certificate, `--key` with a private key, or `--cert` with a certificate file. For more details, refer to `grpcurl --help`.
   </Note>

2. Download the Eye Contact Python client code by cloning the Clients repository ([NVIDIA-Maxine/nim-clients](https://github.com/NVIDIA-Maxine/nim-clients)):

   ```bash
   git clone https://github.com/NVIDIA-Maxine/nim-clients.git

   # Go to the 'eye-contact' folder

   cd nim-clients/eye-contact/ 
   ```

3. Install the required dependencies:

   ```bash
   sudo apt-get install python3-pip
   pip install -r requirements.txt
   ```   

##  Compile the Protos (Optional)

If you want to use the client code provided in the Clients repository ([NVIDIA-Maxine/nim-clients](https://github.com/NVIDIA-Maxine/nim-clients)), you can skip this step.

The proto files are available in the `eye-contact/protos` folder. You can compile them to generate client interfaces in your preferred programming language. For more details, refer to [Supported languages](https://grpc.io/docs/languages/) in the gRPC documentation.

The following is an example of how to compile the protos for Python on Linux and Windows.
The `grpcio` version needed for compilation can be referred from `requirements.txt`

To compile protos on Linux:

```bash
# Go to eye-contact/protos/linux folder
cd eye-contact/protos/linux

chmod +x compile_protos.sh
./compile_protos.sh
```

To compile protos on Windows:

```bash
# Go to eye-contact/protos/windows folder
cd eye-contact/protos/windows

./compile_protos.bat
```

## Input and Output

The input and output of the NVIDIA Eye Contact NIM are `mp4` files. The input file must use the H.264 codec for video and can include audio. The output video file also uses the H.264 codec for video and contains the same audio as the input video file.
<Note>
Videos with Variable Frame Rate (VFR) are not supported.
</Note>

### Input Modes
The NVIDIA Eye Contact NIM supports two distinct input processing modes based on the input file capabilities:

#### Transactional Mode (Default)

In transactional mode, the entire input video file must be received and processed as a complete unit by the NIM before returning results.

This mode is suitable for the following use cases:

- Processing of small video files, as these files are copied to NIM in entirety before inference can begin.
- Applications that can wait for complete processing before receiving output.
- Videos that are not optimized for streaming.

To run the NVIDIA Eye Contact NIM in transactional mode, simply run the NVIDIA Eye Contact sample client without any additional flags:

Go to the `scripts` directory:
```bash
cd scripts
```

Send a gRPC request:
```bash
python eye-contact.py --target <server_ip:port> \
   --input <input file path> \
   --output <output file path along with file name> 
```

#### Streaming Mode

In streaming mode, the NIM can process incoming video frames incrementally as soon as frame information is available. The output frames are streamed back to the client immediately after processing.

This is the preferred mode for the following use cases:

- Streamable video inputs.
- Applications that do not want to wait until the entire file is completely uploaded.
- Large video files that can benefit from incremental processing and thereby not constrained by the disk space on the server.

| Aspect | Transactional Mode | Streaming Mode |
|--------|-------------------|----------------|
| **Data Storage** | Entire video and audio files are temporarily copied on disk | Only frames being processed are temporarily copied in memory |
| **Processing Start** | NIM waits to receive entire files before starting | NIM starts processing as soon as data chunk for first frame arrives |
| **Processing Timing** | Processing begins after all data is received | Continuous processing without waiting for complete input |
| **Output Delivery** | Complete output video is returned to client after inference is finished for whole video | Output frames are generated and returned immediately |

The `--streaming` option works with streamable videos where metadata is positioned at the beginning of the file. Videos that are not streamable can be easily converted to a streamable format.

To make any video streamable, use FFmpeg with the following command:
```bash
   ffmpeg -i sample_video.mp4 -movflags +faststart sample_streamable.mp4
```
You can then specify the streamable video as input to the NIM by using the `--input` parameter and enabling the `--streaming` option.
```bash
   python eye-contact.py --target <server_ip:port> \
      --input <input file path> \
      --output <output file path along with file name> \
      --streaming 
```

<Note>
The first inference is not indicative of the model's actual performance because it includes the time taken by the Triton Inference Server to load the models in addition to the time required to process the inference request. 
</Note>

### Usage for Preview API Request

```bash
   python eye-contact.py --preview-mode \
      --target grpc.nvcf.nvidia.com:443 \
      --function-id <function_id> \
      --api-key $API_KEY_REQUIRED_IF_EXECUTING_OUTSIDE_NGC \
      --input <input_file_path> \
      --output <output_file_path>
```

To view details of command-line arguments, run this command:
```bash
python eye-contact.py -h
```
You get a response similar to the following. All parameters are optional.

```bash
options:
  -h, --help                                             show this help message and exit
  --preview-mode                                         Flag to send request to preview NVCF NIM server on https://build.nvidia.com/nvidia/eyecontact/api. (default: False)
  --ssl-mode {DISABLED,MTLS,TLS}                         Flag to set SSL mode, default is DISABLED (default: DISABLED)
  --ssl-key SSL_KEY                                      The path to ssl private key. (default: ../ssl_key/ssl_key_client.pem)
  --ssl-cert SSL_CERT                                    The path to ssl certificate chain. (default: ../ssl_key/ssl_cert_client.pem)
  --ssl-root-cert SSL_ROOT_CERT                          The path to ssl root certificate. (default: ../ssl_key/ssl_ca_cert.pem)
  --target TARGET                                        IP:port of gRPC service, when hosted locally. Use grpc.nvcf.nvidia.com:443 when hosted on NVCF. (default: 127.0.0.1:8001)
  --api-key API_KEY                                      NGC API key required for authentication, utilized when using TRY API ignored otherwise (default: None)
  --function-id FUNCTION_ID                              NVCF function ID for the service, utilized when using TRY API ignored otherwise (default: None)
  --input INPUT                                          The path to the input video file. (default: ../assets/sample_transactional.mp4)
  --output OUTPUT                                        The path for the output video file. (default: output.mp4)
  --streaming                                            Flag to enable grpc streaming mode. Required for streamable video input. (default: False)
  --bitrate BITRATE                                      Output video bitrate in bps (default: 3000000). This is only applicable when lossless mode is disabled. (default: 3000000)
  --idr-interval IDR_INTERVAL                            The interval for IDR frames in the output video. This is only applicable when lossless mode is disabled. (default: 8) (default: 8)
  --lossless                                             Flag to enable lossless mode for video encoding. (default: False)
  --custom-encoding-params CUSTOM_ENCODING_PARAMS        Custom encoding parameters in JSON format. (default: None)
  --temporal TEMPORAL                                    Flag to control temporal filtering (default: 4294967295) (default: 4294967295)
  --detect-closure DETECT_CLOSURE                        Flag to toggle detection of eye closure and occlusion on/off (default: 0) (default: 0)
  --eye-size-sensitivity EYE_SIZE_SENSITIVITY            Eye size sensitivity parameter (default: 3, range: [2, 6]) (default: 3)
  --enable-lookaway {0,1}                                Flag to toggle look away on/off (default: 0) (default: 0)
  --lookaway-max-offset LOOKAWAY_MAX_OFFSET              Maximum value of gaze offset angle (degrees) during a random look away (default: 5, range: [1, 10]) (default: 5)
  --lookaway-interval-min LOOKAWAY_INTERVAL_MIN          Minimum limit for the number of frames at which random look away occurs (default: 100, range: [1, 600]) (default: 100)
  --lookaway-interval-range LOOKAWAY_INTERVAL_RANGE      Range for picking the number of frames at which random look away occurs (default: 250, range: [1, 600]) (default: 250)
  --gaze-pitch-threshold-low GAZE_PITCH_THRESHOLD_LOW    Gaze pitch threshold (degrees) at which the redirection starts transitioning (default: 20.0, range: [10, 35]) (default: 20.0)
  --gaze-pitch-threshold-high GAZE_PITCH_THRESHOLD_HIGH  Gaze pitch threshold (degrees) at which the redirection is equal to estimated gaze (default: 30.0, range: [10, 35]) (default: 30.0)
  --gaze-yaw-threshold-low GAZE_YAW_THRESHOLD_LOW        Gaze yaw threshold (degrees) at which the redirection starts transitioning (default: 20.0, range: [10, 35]) (default: 20.0)
  --gaze-yaw-threshold-high GAZE_YAW_THRESHOLD_HIGH      Gaze yaw threshold (degrees) at which the redirection is equal to estimated gaze (default: 30.0, range: [10, 35]) (default: 30.0)
  --head-pitch-threshold-low HEAD_PITCH_THRESHOLD_LOW    Head pose pitch threshold (degrees) at which the redirection starts transitioning away from camera towards estimated gaze (default: 15.0, range: [10, 35]) (default: 15.0)
  --head-pitch-threshold-high HEAD_PITCH_THRESHOLD_HIGH  Head pose pitch threshold (degrees) at which the redirection is equal to estimated gaze (default: 15.0, range: [10, 35]) (default: 15.0)
  --head-yaw-threshold-low HEAD_YAW_THRESHOLD_LOW        Head pose yaw threshold (degrees) at which the redirection starts transitioning (default: 15.0, range: [10, 35]) (default: 15.0)
  --head-yaw-threshold-high HEAD_YAW_THRESHOLD_HIGH      Head pose yaw threshold (degrees) at which the redirection is equal to estimated gaze (default: 15.0, range: [10, 35]) (default: 15.0)
```

### Example Commands

Basic inference with default settings:
```bash
python eye-contact.py --target 127.0.0.1:8001 --input ../assets/sample_transactional.mp4 --output output.mp4
```

Using streaming mode for streamable videos:
```bash
python eye-contact.py --target 127.0.0.1:8001 --input ../assets/sample_streamable.mp4 --output output.mp4 --streaming
```

Advanced configuration example:
```bash
python eye-contact.py --target 127.0.0.1:8001 --input ../assets/sample_transactional.mp4 --output output.mp4     --custom-encoding-params '{"control-rate": "1", "bitrate": 3000000, "tuning-info-id": "3", "temporalaq": true}'
```

### Python Notebook

For an interactive experience and to explore all feature parameters, we provide a comprehensive Python notebook that demonstrates the Eye Contact service capabilities.

The Python notebook is located at `examples/maxine-eye-contact.ipynb` in the `eye-contact` client folder.