Skip to navigation

Observability for Nemotron Safety Guard NIM

View as Markdown

NIM provides Prometheus metrics for request statistics. You can use these metrics to create dashboards with Grafana. By default, metrics are available at http://localhost:8000/v1/metrics.

CategoryMetricMetric nameDescriptionGranularityFrequency
KV CacheGPU Cache Usagegpu_cache_usage_percGPU KV-cache usage. A value of 1 means 100% usage.Per modelPer iteration
CountRunning Countnum_requests_runningNumber of requests currently running on GPU.Per modelPer iteration
CountWaiting Countnum_requests_waitingNumber of requests waiting to be processed.Per modelPer iteration
CountMax Request Countnum_request_maxMaximum number of requests that can run concurrently by the model.Per modelPer iteration
CountPrompt Token Countprompt_tokens_totalNumber of prefill tokens processed.Per modelPer iteration
CountGeneration Token Countgeneration_tokens_totalNumber of generation tokens processed.Per modelPer iteration
LatencyTime to First Tokentime_to_first_token_secondsHistogram of time to first token in seconds.Per modelPer request
LatencyTime per Output Tokentime_per_output_token_secondsHistogram of time per output token in seconds.Per modelPer request
LatencyEnd to Ende2e_request_latency_secondsHistogram of end-to-end request latency in seconds.Per modelPer request
CountPrompt Token Countrequest_prompt_tokensHistogram of prefill tokens processed.Per modelPer request
CountGeneration Token Countrequest_generation_tokensHistogram of generation tokens processed.Per modelPer request
CountFinished Request Countrequest_finish_totalNumber of finished requests, with a label indicating the finish reason.Per modelPer request
CountSuccess Request Countrequest_success_totalNumber of successful requests. Requests with finish reason stop or length are counted.Per modelPer request
CountFailure Request Countrequest_failure_totalNumber of failed requests. Requests with other finish reasons are counted.Per modelPer request

Prometheus

Install Prometheus and configure its prometheus.yml file to scrape NIM at localhost:8000:

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:8000"]

Start Prometheus:

./prometheus --config.file=./prometheus.yml

Open http://localhost:9090/targets to verify that Prometheus detects the NIM target.

Grafana

Install Grafana, then start its server:

./bin/grafana-server

Open http://localhost:3000 and sign in with the default admin / admin credentials. Add Prometheus as a data source with the URL localhost:9090.