> This page is for NVIDIA Audio2Face-2D NIM, version 1.0.0 (Latest) (default).
> For other versions, use one of these documentation indexes:
> - 1.0.0 (Latest) (default): https://docs.nvidia.com/nim/maxine/audio2face-2d/1.0.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nim/maxine/llms.txt.

# Basic Inference

1. Perform a health check on the gRPC endpoint.

   - Install `grpcurl` from [github.com/fullstorydev/grpcurl/releases](https://github.com/fullstorydev/grpcurl/releases).

      Example commands to run on Ubuntu:
      ```bash
      wget https://github.com/fullstorydev/grpcurl/releases/download/v1.9.1/grpcurl_1.9.1_linux_amd64.deb
      sudo dpkg -i grpcurl_1.9.1_linux_amd64.deb
      ```

   - Download the health checking proto:
      ```bash
      wget https://raw.githubusercontent.com/grpc/grpc/master/src/proto/grpc/health/v1/health.proto
      ```

   - Run the health check:
      ```bash
      grpcurl --plaintext --proto health.proto localhost:8001 grpc.health.v1.Health/Check
      ```

   - If the service is ready, you get a response similar to the following:
      ```bash
      { "status": "SERVING" }
      ```

   <Note>
   For using grpcurl with an SSL-enabled server, avoid using the `--plaintext` argument, and use `--cacert` with a CA certificate, `--key` with a private key, or `--cert` with a certificate file. For more details, refer to `grpcurl --help`.
   </Note>



2. Download the Audio2Face-2D client code by cloning the [gRPC Client repository](https://github.com/NVIDIA-Maxine/nim-clients):

   ```bash
   git clone https://github.com/NVIDIA-Maxine/nim-clients.git

   # Go to the 'audio2face-2d' folder
   cd nim-clients/audio2face-2d/
   ```

3. Install the required dependencies:
   
   - For python client
   ```bash
   # Install python on Linux
   sudo apt-get install python3-pip
   pip install -r python/requirements.txt
   ```   

   - For Node.js client
   ```bash
   # Add the repo and install the latest stable Node.js on Linux
   curl -fsSL https://deb.nodesource.com/setup_current.x | sudo -E bash -
   sudo apt install nodejs

   # Install all the required packages using package.json file in nodejs directory  
   npm install --prefix nodejs/ 
   ```
##  Compile the Protos (Optional)

If you want to use the client code provided in the [github Client repository](https://github.com/NVIDIA-Maxine/nim-clients), you can skip this step.

The proto files are available in the `audio2face-2d/protos` folder. You can compile them to generate client interfaces in your preferred programming language. For more details, refer to [Supported languages](https://grpc.io/docs/languages/) in the gRPC documentation.

Here is an example of how to compile the protos for Python and Node.js on Linux and Windows.

### Python

The `grpcio` version needed for compilation can be referred from `requirements.txt`

To compile protos on Linux, run:

```bash
// Go to audio2face-2d/protos/linux/python folder
cd audio2face-2d/protos/linux/python

chmod +x compile_protos.sh
./compile_protos.sh
```

To compile protos on Windows, run:

```bash
// Go to audio2face-2d/protos/windows/python folder
cd audio2face-2d/protos/windows/python

./compile_protos.bat
```

### Node.js
Before running the Node.js client, you can choose to compile the protos. 

To compile protos on Linux, run:
```bash
// Go to audio2face-2d/protos/linux/nodejs folder
cd audio2face-2d/protos/linux/nodejs

chmod +x compile_protos.sh
./compile_protos.sh
```
To compile protos on Windows, run:
```bash
// Go to audio2face-2d/protos/windows/nodejs folder
cd audio2face-2d/protos/windows/nodejs

./compile_protos.bat
```

## Running Inference via Python Script

You can use the sample client script in the Audio2Face-2D GitHub repo to send a gRPC request to the hosted NIM server:

1. Go to the Python scripts directory.
   ```bash
   cd python/scripts
   ```

2. Run the command to send a gRPC request (all command-line parameters are optional).
   ```bash
   python audio2face-2d.py --target <server_ip:port> --audio-input <input audio file path> --portrait-input <input portrait image file path> --output <output file path and the file name> --head-rotation-animation-filepath <rotation animation filepath> --head-translation-animation-filepath <translation animation filepath> --ssl-mode <ssl mode value> --ssl-key <ssl key file path> --ssl-cert <ssl cert filepath> --ssl-root-cert <ssl root cert filepath>
   ```

<Note>
The first inference is not indicative of the model's actual performance, as it includes the time taken by the Triton Inference Server to load the models for the first time. The subsequent inference requests will reflect the actual processing performance.

For Blackwell GPUs, the initial inference might time out because of the time needed to load the models. If timeout occurs, send another request; subsequent inferences will reflect the actual processing performance.
</Note>

To view details of command-line arguments, run this command:
```bash
python audio2face-2d.py -h
```

You will get a response similar to the following.
```bash
options:
   -h, --help                                                                 show this help message and exit
   --ssl-mode {DISABLED,MTLS,TLS}                                             Flag to set SSL mode, default is None
   --ssl-key SSL_KEY                                                          The path to ssl private key.
   --ssl-cert SSL_CERT                                                        The path to ssl certificate chain.
   --ssl-root-cert SSL_ROOT_CERT                                              The path to ssl root certificate.
   --target TARGET                                                            IP:port of gRPC service, when hosted locally.
   --audio-input AUDIO_INPUT                                                  The path to the input audio file.
   --portrait-input PORTRAIT_INPUT                                            The path to the input portrait file.
   --output OUTPUT                                                            The path for the output video file.
   --head-rotation-animation-filepath HEAD_ROTATION_ANIMATION_FILEPATH        The path for the head_rotation_animation.csv file. Only required for HEAD_POSE_MODE_USER_DEFINED_ANIMATION
   --head-translation-animation-filepath HEAD_TRANSLATION_ANIMATION_FILEPATH  The path for the head_translation_animation.csv file. Only required for HEAD_POSE_MODE_USER_DEFINED_ANIMATION
```

If the command-line arguments are not passed, the script uses the following default values:

## Default Command-Line Arguments

| **Argument**                       | **Default Value**                        |
|------------------------------------|------------------------------------------|
| `target`                           | `127.0.0.1:8001`                         | 
| `portrait-input`                   | `../../assets/sample_portrait_image.png` | 
| `audio-input`                      | `../../assets/sample_audio.wav`          | 
| `output`                           |  `output.mp4` in the current directory   | 
| `head-rotation-animation-filepath` | `../../assets/head_rotation_animation.csv` |
| `head-translation-animation-filepath` | `../../assets/head_translation_animation.csv`| 
| `ssl-mode`                         | `DISABLED`                               |
| `ssl-key`                          | `../ssl_key/ssl_key_client.pem`          |
| `ssl-cert`                         | `../ssl_key/ssl_cert_client.pem`         |
| `ssl-root-cert`                    | `../ssl_key/ssl_ca_cert.pem`             |



## Running Inference via Node.js Script

A Node.js client, similar to a Python client, can also be used to exercise the A2F2D feature by sending gRPC requests to a hosted NIM server.
The `audio2face-2d.js` Node.js script takes a portrait image and a `.wav` or `.pcm` audio file (default is `.wav`) and generates the MP4 video output.

Go the Node.js script folder
```bash
cd nodejs/scripts
```

Run the following command to send a gRPC request (all command-line parameters are optional):
```bash
node audio2face-2d.js --target <server_ip:port> --audio-input <input audio file path> --portrait-input <input portrait image file path> --output <output file path and the file name> --format <wav/pcm> --head-rotation-animation-filepath <rotation animation file path> --head-translation-animation-filepath <translation animation filepath> --ssl-mode <ssl mode value> --ssl-key <ssl key file path> --ssl-cert <ssl cert filepath> --ssl-root-cert <ssl root cert filepath>
```

The default configuration expected for PCM audio format in the Node.js client is as follows:

- Sample rate: 48kHz
- Channels: Mono-channel
- Bit Depth: 16

If any other configuration is needed, change it in the Node.js client `audio2face-2d/nodejs/scripts/audio2face-2d.js` in the function `sendInputAudioChunks()`. 

To view details of the command-line arguments, run this command:
```bash
node audio2face-2d.js -h
```

You will get a response similar to the following.
```bash
Options:
  --ssl-mode <type>                             Flag to set SSL mode (default: "DISABLED").
  --ssl-key <type>                              Path to SSL private key. (default: "../ssl_key/ssl_key_client.pem").
  --ssl-cert <type>                             Path to SSL certificate chain. (default: "../ssl_key/ssl_cert_client.pem").
  --ssl-root-cert <type>                        Path to SSL root certificate. (default: "../ssl_key/ssl_ca_cert.pem").
  --target <type>                               IP:port of gRPC service, when hosted locally. (default: "127.0.0.1:8001").
  --audio-input <type>                          Path to the input audio file. (default: "../../assets/sample_audio.wav").
  --head-rotation-animation-filepath <type>     Path to head rotation animation csv file. (default: "../../assets/head_rotation_animation.csv").
  --head-translation-animation-filepath <type>  Path to head translation animation csv file. (default: "../../assets/head_translation_animation.csv").
  --portrait-input <type>                       Path to the input portrait file. (default: "../../assets/sample_portrait_image.png").
  --format <type>                               Audio format - wav or pcm (default: "wav").
  --output <type>                               Path for the output video file. (default: "output.mp4").
  -h, --help                                    Display help for command.
```

**Performance data**

The following table shows the performance data for the Audio2Face-2D model on various GPUs and modes with the provided sample input files and the Python client.

| GPU        | Model Selection | Mode                                      | Elapsed Time (s) |
|------------|-----------------|-------------------------------------------|------------------|
| L40        | Performance     | ANIMATION_CROPPING_MODE_FACEBOX           | 20.48            |
|            | Performance     | ANIMATION_CROPPING_MODE_REGISTRATION_BLENDING | 60.07            |
|            | Quality         | ANIMATION_CROPPING_MODE_FACEBOX           | 23.73            |
|            | Quality         | ANIMATION_CROPPING_MODE_REGISTRATION_BLENDING | 66.56            |
| A10        | Performance     | ANIMATION_CROPPING_MODE_FACEBOX           | 33.83            |
|            | Performance     | ANIMATION_CROPPING_MODE_REGISTRATION_BLENDING | 74.40            |
|            | Quality         | ANIMATION_CROPPING_MODE_FACEBOX           | 39.04            |
|            | Quality         | ANIMATION_CROPPING_MODE_REGISTRATION_BLENDING | 83.14            |
| T4         | Performance     | ANIMATION_CROPPING_MODE_FACEBOX           | 50.12            |
|            | Performance     | ANIMATION_CROPPING_MODE_REGISTRATION_BLENDING | 88.76           |
|            | Quality         | ANIMATION_CROPPING_MODE_FACEBOX           | 78.42            |
|            | Quality         | ANIMATION_CROPPING_MODE_REGISTRATION_BLENDING | 115.33           |
| V100       | Performance     | ANIMATION_CROPPING_MODE_FACEBOX           | 31.08            |
|            | Performance     | ANIMATION_CROPPING_MODE_REGISTRATION_BLENDING | 71.17            |
|            | Quality         | ANIMATION_CROPPING_MODE_FACEBOX           | 39.85            |
|            | Quality         | ANIMATION_CROPPING_MODE_REGISTRATION_BLENDING | 78.19            |
| RTX 5090   | Performance     | ANIMATION_CROPPING_MODE_FACEBOX           | 25.36            |
|            | Performance     | ANIMATION_CROPPING_MODE_REGISTRATION_BLENDING | 51.45            |
|            | Quality         | ANIMATION_CROPPING_MODE_FACEBOX           | 27.83            |
|            | Quality         | ANIMATION_CROPPING_MODE_REGISTRATION_BLENDING | 60.27            |
| RTX 4090   | Performance     | ANIMATION_CROPPING_MODE_FACEBOX           | 22.80            |
|            | Performance     | ANIMATION_CROPPING_MODE_REGISTRATION_BLENDING | 51.09            |
|            | Quality         | ANIMATION_CROPPING_MODE_FACEBOX           | 24.79            |
|            | Quality         | ANIMATION_CROPPING_MODE_REGISTRATION_BLENDING | 50.61            |