Low-FPS and latency Troubleshooting

Real-time interaction with Audio2Face-3D Authoring Microservice is very sensitive to latency.

You can use the provided sample application to diagnose latency issues with the Microservice.

The sections below will help you root-cause the latency issues that you might be facing.

Latency with the MACE Plugin

Current MACE plugin is performing sequential request to Audio2Face-3D Authoring Microservice. This means having latency higher than 33ms will make Avatar Face update to be lower than 30 FPS. This is a known limitation and will be addressed in future releases.

Implementing your own client

If you implement your own client, we recommend you to perform your gRPC request asynchronously to minimize the latency in between requests.

E.g.: There are 30 requests to make with 33ms of interval (30 FPS) and the end-2-end latency is 100ms:

In-sync

You make request 1 as 0ms; you wait to receive it until 100ms, then make the request 2, then it takes 100ms again; etc…

For 30 requests the total time it will take will be 3000ms. So 10 FPS; this is not ideal for real-time interaction

Async

You make request 1 as 0ms; then make the request 2 at 33ms; 3 at 66ms; 4 at 99ms; etc…

You receive request 1 as 100ms; request 2 as 133ms; request 3 as 166ms; request 4 as 199ms; etc…

For 30 requests the total time it will take will be 1100ms. So 30 FPS; this is good for real-time interaction with a little delay of 100ms; which should not be too noticeable.

Lowering latency of the A2F-3D Authoring Microservice

In order to serve more client simultaneously the Microservice processes incoming requests in batch. This batch size can be reduced to allow faster processing. However this will reduced the amount client that can be served with a low latency at the same time.

Hereunder you will find updated config files and docker-compose to reduce this batch size to 2.

Docker Compose

You will need 1 updated docker compose file:

docker-compose.yaml

services:
  a2f-authoring:
    image: nvcr.io/nvidia/ace/a2f-authoring:0.3.6
    volumes:
      - models_a2f:/tmp/a2f
      - models_a2e:/tmp/a2e
      - ./claire_v2.3-config.json:/app/configs/claire_v2.3-config.json:ro
      - ./mark_v2.3-config.json:/app/configs/mark_v2.3-config.json:ro
      - ./james_v2.3-config.json:/app/configs/james_v2.3-config.json:ro
    command: bash -c 'python3 generate_trt_models.py ${A2F_3D_MODEL_NAME}_v2.3 trt_params_prod.json && a2f-authoring /app/configs/${A2F_3D_MODEL_NAME}_v2.3-config.json'
    network_mode: "host"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [ gpu ]

volumes:
  models_a2f: {}
  models_a2e: {}

And 3 configuration files in the same folder:

Note

The clib_db_ttl_refresh and clib_db_ttl_check_interval_seconds options contain a typo in their names. This will be corrected in upcoming versions.

james_v2.3-config.json

{
  "a2e_batch_size": 2,
  "a2e_config_path": "/app/configs/a2e-config.json",
  "a2f_batch_size": 2,
  "a2f_config_path": "/app/configs/james_v2.3-proc-config.json",
  "endpoint": "0.0.0.0:50051",
  "clip_db_ttl_seconds": 3600,
  "clip_db_max_size_bytes": 10737418240,
  "clib_db_ttl_refresh_on_use": false,
  "clib_db_ttl_check_interval_seconds": 60
}

claire_v2.3-config.json

{
  "a2e_batch_size": 2,
  "a2e_config_path": "/app/configs/a2e-config.json",
  "a2f_batch_size": 2,
  "a2f_config_path": "/app/configs/claire_v2.3-proc-config.json",
  "endpoint": "0.0.0.0:50051",
  "clip_db_ttl_seconds": 3600,
  "clip_db_max_size_bytes": 10737418240,
  "clib_db_ttl_refresh_on_use": false,
  "clib_db_ttl_check_interval_seconds": 60
}

mark_v2.3-config.json

{
  "a2e_batch_size": 2,
  "a2e_config_path": "/app/configs/a2e-config.json",
  "a2f_batch_size": 2,
  "a2f_config_path": "/app/configs/mark_v2.3-proc-config.json",
  "endpoint": "0.0.0.0:50051",
  "clip_db_ttl_seconds": 3600,
  "clip_db_max_size_bytes": 10737418240,
  "clib_db_ttl_refresh_on_use": false,
  "clib_db_ttl_check_interval_seconds": 60
}

Copy these 4 files to your current folder.

You can then set the A2F_3D_MODEL_NAME environment variable to either james, mark or claire depending on your preference for deployment and run docker compose up.

export A2F_3D_MODEL_NAME=james
docker compose up

Wait for the service to display readiness and you can start interacting with it.

...
a2f-authoring-1  | 2024-09-18T15:33:57.763448Z  INFO a2f_authoring: Service is initialized!

Running the docker container

In that case you will need only the config files:

Note

The clib_db_ttl_refresh and clib_db_ttl_check_interval_seconds options contain a typo in their names. This will be corrected in upcoming versions.

james_v2.3-config.json

{
  "a2e_batch_size": 2,
  "a2e_config_path": "/app/configs/a2e-config.json",
  "a2f_batch_size": 2,
  "a2f_config_path": "/app/configs/james_v2.3-proc-config.json",
  "endpoint": "0.0.0.0:50051",
  "clip_db_ttl_seconds": 3600,
  "clip_db_max_size_bytes": 10737418240,
  "clib_db_ttl_refresh_on_use": false,
  "clib_db_ttl_check_interval_seconds": 60
}

claire_v2.3-config.json

{
  "a2e_batch_size": 2,
  "a2e_config_path": "/app/configs/a2e-config.json",
  "a2f_batch_size": 2,
  "a2f_config_path": "/app/configs/claire_v2.3-proc-config.json",
  "endpoint": "0.0.0.0:50051",
  "clip_db_ttl_seconds": 3600,
  "clip_db_max_size_bytes": 10737418240,
  "clib_db_ttl_refresh_on_use": false,
  "clib_db_ttl_check_interval_seconds": 60
}

mark_v2.3-config.json

{
  "a2e_batch_size": 2,
  "a2e_config_path": "/app/configs/a2e-config.json",
  "a2f_batch_size": 2,
  "a2f_config_path": "/app/configs/mark_v2.3-proc-config.json",
  "endpoint": "0.0.0.0:50051",
  "clip_db_ttl_seconds": 3600,
  "clip_db_max_size_bytes": 10737418240,
  "clib_db_ttl_refresh_on_use": false,
  "clib_db_ttl_check_interval_seconds": 60
}

and to replace them into the /app/configs folder.

Copy them to your current folder.

Below is an example for Claire model, assuming the config files are in the current folder.

First start the container with a terminal:

docker run -it --rm --network=host --gpus  all -v $PWD:/my_mounted_files nvcr.io/nvidia/ace/a2f-authoring:0.3.6 bash

Then inside the container:

cp /my_mounted_files/claire_v2.3-config.json /app/configs/claire_v2.3-config.json
./run_claire_model.sh

Latency due to geolocation

The clients of A2F-3D Authoring Microservice should be geolocated as close as possible to the deployment. The farther clients are located the less interactive the parameter tweaking experience will be.