Observability for NVIDIA NIM for LLMs#

NIM provides Prometheus metrics indicating request statistics. You can use these metrics to create dashboards with Grafana. By default, these metrics are available at http://localhost:8000/v1/metrics.

The following table describes the available metrics.

Category

Metric

Metric Name

Description

Granularity

Frequency

KV Cache

GPU Cache Usage

gpu_cache_usage_perc

GPU KV-cache usage. 1 means 100 percent usage

Per model

Per iteration

Count

Running Count

num_requests_running

Number of requests currently running on GPU

Per model

Per iteration

Waiting Count

num_requests_waiting

Number of requests waiting to be processed

Per model

Per iteration

Max Request Count

num_request_max

Max number of requests that can be run concurrently by the model

Per model

Per iteration

Total Prompt Token Count

prompt_tokens_total

Number of prefill tokens processed

Per model

Per iteration

Total Generation Token Count

generation_tokens_total

Number of generation tokens processed

Per model

Per iteration

Latency

Time to First Token

time_to_first_token_seconds

Histogram of time to first token in seconds

Per model

Per request

Time per Output Token

time_per_output_token_seconds

Histogram of time per output token in seconds

Per model

Per request

End to End

e2e_request_latency_seconds

Histogram of end to end request latency in seconds

Per model

Per request

Count

Prompt Token Count

request_prompt_tokens

Histogram of number of prefill tokens processed

Per model

Per request

Generation Token Count

request_generation_tokens

Histogram of number of generation tokens processed

Per model

Per request

Finished Request Count

request_finish_total

Number of finished requests, with label indicating finish reason

Per model

Per request

Success Request Count

request_success_total

Number of successful requests, requests with finish reason “stop” or “length” are counted

Per model

Per request

Failure Request Count

request_failure_total

Number of failed requests, requests with other finish reason are counted

Per model

Per request

Prometheus#

To install Prometheus for scraping metrics from NIM, download the latest Prometheus version appropriate for your system.

wget https://github.com/prometheus/prometheus/releases/download/v2.52.0/prometheus-2.52.0.linux-amd64.tar.gz
tar -xvzf prometheus-2.52.0.linux-amd64.tar.gz
cd prometheus-2.52.0.linux-amd64/

Edit the Prometheus configuration file to scrape from the NIM endpoint. Make sure the targets field point to localhost:8000

vi prometheus.yml

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # In Prometheus 2.53, metrics_path defaults to '/metrics'
    # Previous versions use '/v1/metrics'.
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:8000"]

Next, run the Prometheus server: ./prometheus --config.file=./prometheus.yml

Use a browser to verify that the NIM target was detected by the Prometheus server: http://localhost:9090/targets?search=. You can also click on the NIM target URL link to explore generated metrics.

Grafana#

You can use Grafana for visualizing NIM metrics. Install the latest Grafana version for your system.

wget https://dl.grafana.com/oss/release/grafana-11.0.0.linux-amd64.tar.gz
tar -zxvf grafana-11.0.0.linux-amd64.tar.gz

Run the Grafana server:

cd grafana-v11.0.0/
./bin/grafana-server

To access the Grafana dashboard, point your browser to http://localhost:3000. Log in using the defaults:

username: admin 
password: admin

First, configure the data source in Grafana. Click Data Source, select Prometheus, and set the URL to localhost:9090. After saving the configuration, you should see a success message. You can now create a dashboard using NIM metrics or import this example dashboard.

NIM Dashboard Example