Observability#

Overview#

The Audio2Face-3D (A2F-3D) microservice is designed to capture traces and metrics and export them in OpenTelemetry (OTel) format. This format is compatible with most modern observability tools, enabling efficient monitoring an analysis of service performance.

In addition to OTel, the A2F-3D microservice supports exporting metrics to Prometheus.

Traces#

To enable traces exporting, override telemetry.traces_enabled to true.

The default endpoint between Audio2Face-3D and OTLP traces is http://localhost:4318/v1/traces/.

For all available configurations for observability with OpenTelemetry and how to override them, see the Configuration page.

Audio2Face-3D captures the following traces:

Trace

Description

nvidia_ace.services.a2f_controller.v1.A2FControllerService/ProcessAudioStream

Trace coresponding to the full gRPC request for the bidirectional streaming mode

nvidia_ace.services.a2f.v1.A2FService/PushAudioStream

Trace coresponding to the full gRPC request for the unary gRPC mode.

Metrics#

To enable metrics exporting:

  • override telemetry.metrics_enabled to true

  • override telemetry.metrics_exporter to either prometheus or otlp, depending on what solution you use to collect the metrics

The default endpoint between Audio2Face-3D and OTLP metrics is http://localhost:4318/v1/metrics. The default port between Audio2Face-3D and Prometheus is 9464.

For all available configurations for observability with OpenTelemetry and how to override them, see the Configuration page.

Metric name

Metric type

Description

streams_in_use

Gauge

How many streams are in use by the microservice

streams_available

Gauge

How many streams are stil aviable.

Quick start the NIM container and visualize metrics and traces#

You can use the provided docker compose setup to start telemetry collecting services alongside Audio2Face-3D NIM in a local deployment:

  • Prometheus: open source solution to gather and observe metrics.

  • Jaeger: open source solution to gather and observe traces.

Clone the repository: NVIDIA/Audio2Face-3D-Samples

Go to quick-start/ subfolder.

Then run docker compose up and specify the model name to use:

$ A2F_3D_MODEL_NAME={model_name} docker compose -f docker-compose-with-observability.yml up

The A2F_3D_MODEL_NAME can be one of the following: claire, mark, james.

It should take a few minutes to boot up the first time and you should see the following output when successful

✔ Container a2f-init                   Created                                                                                                                                                                                  0.0s
✔ Container quick-start-prometheus-1   Created                                                                                                                                                                                  0.0s
✔ Container quick-start-jaeger-1       Created                                                                                                                                                                                  0.0s
✔ Container quick-start-a2f-service-1  Created                                                                                                                                                                                  0.0s
Attaching to a2f-init, a2f-service-1, jaeger-1, prometheus-1

You are now running a local deployment of the Audio2Face-3D NIM.

To try it out, you can use the sample app provided. After sending requests to the Audio2Face-3D NIM, you can visualize the metrics and traces by going to:

  • http://localhost:9090/graph for Prometheus UI to see metrics

    • Search in the query bar for one for of the metrics in the table above and move to the Graph tab.

    • To see more meaningful data, try zooming in on the graph by adjusting the time interval.

    ../../../../_images/prometheus_example.png
  • http://localhost:16686/search for Jaeger UI to see traces

    • In the Search section on the left, in the Service dropdown select audio2face then click Find Traces.

    ../../../../_images/jaeger_example.png