Observability#
NIM provides Prometheus metrics indicating request statistics. These metrics can be used to create dashboards with the Grafana dashboard. By default, these metrics are available at http://localhost:8000/metrics.
curl -X 'GET' 'http://0.0.0.0:8000/metrics'
The following table describes the available metrics.
Category |
Metric |
Metric Name |
Description |
Granularity |
Frequency |
---|---|---|---|---|---|
KV Cache Count |
GPU Cache Usage |
gpu_cache_usage_perc |
GPU KV-cache usage. 1 means 100 percent usage |
Per model |
Per iteration |
Running Count |
num_requests_running |
Number of requests currently running on GPU |
Per model |
Per iteration |
|
Waiting Count |
num_requests_waiting |
Number of requests waiting to be processed |
Per model |
Per iteration |
|
Max Request Count |
num_request_max |
Max number of concurrently running requests |
Per model |
Per iteration |
|
Total Prompt Token Count |
prompt_tokens_total |
Number of prefill tokens processed |
Per model |
Per iteration |
|
Total Generation Token Count |
generation_tokens_total |
Number of generation tokens processed |
Per model |
Per iteration |
|
Latency |
Time to First Token |
time_to_first_token_seconds |
Histogram of time to first token in seconds |
Per model |
Per request |
Time per Output Token |
time_per_output_token_seconds |
Histogram of time per output token in seconds |
Per model |
Per request |
|
End to End Request Latency |
e2e_request_latency_seconds |
Histogram of end to end request latency in seconds |
Per model |
Per request |
|
Vision Encoder Latency |
vision_encoder_latency_seconds |
Histogram of vision encoder latency in seconds |
Per model |
Per request |
|
Count |
Prompt Token Count |
request_prompt_tokens |
Histogram of number of prefill tokens processed |
Per model |
Per request |
Generation Token Count |
request_generation_tokens |
Histogram of number of generation tokens processed |
Per model |
Per request |
|
Finished Request Count |
request_finish_total |
Number of finished requests, with label indicating finish reason |
Per model |
Per request |
|
Successful Request Count |
request_success_total |
Number of successful requests |
Per model |
Per request |
|
Successful Request Count |
request_failure_total |
Number of failed requests |
Per model |
Per request |
|
Image Count |
request_image_count |
Histogram of the number of images per request |
Per model |
Per request |
|
Image |
Image size |
image_size_pixels |
Histogram of image sizes in pixels |
Per model |
Per request |
Prometheus#
To install Prometheus for scraping metrics from NIM, download the latest Prometheus version appropriate for your system.
wget https://github.com/prometheus/prometheus/releases/download/v2.52.0/prometheus-2.52.0.linux-amd64.tar.gz
tar -xvzf prometheus-2.52.0.linux-amd64.tar.gz
cd prometheus-2.52.0.linux-amd64/
Edit the Prometheus configuration file prometheus.yml
to scrape from the NIM endpoint. Make sure the targets
field point to localhost:8000
.
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: "prometheus"
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ["localhost:8000"]
Run the Prometheus server
./prometheus --config.file=./prometheus.yml
Open a browser and point it to http://localhost:9090/targets?search=
to check that the NIM target was detected by the Prometheus server. You can also click on the NIM target url link to explore the generated metrics.
Grafana#
We can use Grafana for dashboarding NIM metrics. Install the latest Grafana version appropriate for your system.
wget https://dl.grafana.com/oss/release/grafana-11.0.0.linux-amd64.tar.gz
tar -zxvf grafana-11.0.0.linux-amd64.tar.gz
cd grafana-v11.0.0/
Run the Grafana server
./bin/grafana-server
To access the Grafana dashboard, point your browser to http://localhost:3000
. You will need to log in using the defaults.
username: admin
password: admin
The first step is to configure the source from which Grafana can scrape metrics. Click on the “Data Source” button, select Prometheus, and specify the Prometheus URL localhost:9090
. After saving the configuration, you should see a success message.
Now you are ready to create a dashboard with metrics from NIM. This Grafana tutorial provides step-by-step instructions on building a dashboard from scratch. Alternatively, you can jump start using this example dashboard
.

Vision metrics#
Vision-related metrics (e.g., Vision Encoder Latency) are only available for the TRT-LLM backend (the vLLM backend only collects end-to-end metrics). Consider starting with this example dashboard
instead if NIM runs TRT-LLM optimized engines.