Observability

NeMo Retriever Text Reranking NIM (Text Reranking NIM) has support for exporting metrics and traces in an OpenTelemetry-compatible format.

Additionally, the underlying Triton service exposes its own metrics through a Prometheus endpoint.

To collect these metrics and traces, export them to a running OpenTelemetry Collector instance, which can then export them to any OTLP-compatible backend.

You can collect metrics from both the Text Reranking NIM container and the underlying Triton instance.

Service Metrics

To enable metrics exporting from the NIM, set the OTEL_SERVICE_NAME, OTEL_METRICS_EXPORTER and OTEL_EXPORTER_OTLP_ENDPOINT environment variables when launching the Text Reranking NIM container.

Triton Metrics

Triton exposes its metrics on port 8002 in Prometheus format. To collect these metrics, use a Prometheus receiver to scrape the Triton endpoint and export them in an OpenTelemetry compatible format. See the following example for details.

To enable exporting of traces from the NIM web service, set the OTEL_SERVICE_NAME, OTEL_TRACES_EXPORTER and OTEL_EXPORTER_OTLP_ENDPOINT environment variables when launching the Text Reranking NIM container.

The following example requires that an instance of the OpenTelemetry Collector is running at <opentelemetry-collector-endpoint> on port <opentelemetry-collector-port>.

Launching the NIM Container with OpenTelemetry Enabled

            

            
# Choose a container name for bookkeeping
export NIM_MODEL_NAME=nvidia/nv-rerankqa-mistral-4b-v3
export CONTAINER_NAME=$(basename $NIM_MODEL_NAME)

# Choose a NIM Image from NGC
export IMG_NAME="nvcr.io/nim/$NIM_MODEL_NAME:1.0.0"

# Set the OTEL environment variables to enable metrics exporting
export OTEL_SERVICE_NAME=$CONTAINER_NAME
export OTEL_METRICS_EXPORTER=otlp
export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_ENDPOINT="http://<opentelemetry-collector-endpoint>:<opentelemetry-collector-port>"

docker run -it --rm --name=$CONTAINER_NAME \
  ... \
  -e OTEL_SERVICE_NAME \
  -e OTEL_METRICS_EXPORTER \
  -e OTEL_TRACES_EXPORTER \
  -e OTEL_EXPORTER_OTLP_ENDPOINT \
  ... \
  $IMG_NAME
        

Receiving and Exporting Telemetry Data with the OpenTelemetry Collector

The following OpenTelemetry Collector configuration enables both metrics and tracing exports.

Two receivers are defined:

• The OTLP receiver is capable of receiving both metrics and trace data from the NIM.

• A Prometheus receiver is used for scraping Triton’s own metrics.

Three exporters are defined:

• A Zipkin exporter to export to a running Zipkin instance.

• An OTLP exporter to export to a downstream collector or backend. For example, Datadog.

• A debug exporter which prints received data to the console. This is useful for testing and development purposes.

Traces are configured to be received exclusively by the OTLP receiver, and exported by both the Zipkin and debug exporters. Metrics are configured to be received by both the OTLP and Prometheus receivers, and exported by the OTLP and debug exporters.

            

            
receivers:
  otlp:
    protocols:
      grpc:
      http:
        cors:
          allowed_origins:
            - "*"
  prometheus:
    config:
      scrape_configs:
        - job_name: nim-triton-metrics
          scrape_interval: 10s
          static_configs:
            - targets: ["<nim-endpoint>:8002"]
exporters:
  # NOTE: Prior to v0.86.0 use `logging` instead of `debug`.
  zipkin:
    endpoint: "<zipkin-endpoint>:<zipkin-port>/api/v2/spans"
  otlp:
    endpoint: "<otlp-metrics-endpoint>:<otlp-metrics-port>"
    tls:
      insecure: true
  debug:
    verbosity: detailed
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [debug, zipkin]
    metrics:
      receivers: [otlp, prometheus]
      exporters: [debug, otlp]