Skip to navigation
User Guide

Observability

View as Markdown

This guide explains how to monitor the health, queue depth, and throughput of the NIM for BGR. You can access operational metrics directly through the built-in status endpoint, or integrate with Prometheus and Grafana for continuous monitoring.

BGR-Specific Metrics

NIM for BGR exposes operational metrics using the GET /v1/status endpoint. These metrics provide visibility into the batched geometry relaxation workload.

Aggregate Metrics

Aggregate metrics track the combined geometry relaxation tasks, API requests, and overall queue depths across all active GPU workers:

MetricTypeDescription
tasks_receivedcounterTotal geometry relaxation tasks received
tasks_completedcounterTotal tasks completed
queue_size_atomsgaugeCurrent number of atoms queued for processing
queue_size_tasksgaugeCurrent number of tasks waiting in the queue
requests_receivedcounterTotal API requests received
requests_completedcounterTotal API requests completed
requests_in_progressgaugeAPI requests currently being processed
max_system_sizegaugeMaximum atoms per batch for the current GPU configuration

Per-Worker Metrics

Each GPU worker reports its own metrics from the following:

MetricTypeDescription
devicestringGPU device identifier (for example, cuda:0)
tasks_receivedcounterTasks received by this worker
tasks_completedcounterTasks completed by this worker
queue_size_atomsgaugeAtoms queued on this worker
queue_size_tasksgaugeTasks queued on this worker
batch_sizegaugeCurrent batch size
max_batch_sizegaugeMaximum batch size for this GPU
gpu_modelstringGPU model name

Query Status

Query the status endpoint to retrieve metrics:

curl -s http://localhost:8000/v1/status | python3 -m json.tool

For the full response schema and an example, refer to the GET /v1/status endpoint in the API Reference.

Prometheus

NIM for BGR exposes Prometheus metrics for request statistics at http://localhost:8000/v1/metrics.

To install Prometheus and scrape metrics:

  1. Download the latest Prometheus release for your system:

    wget https://github.com/prometheus/prometheus/releases/download/v2.52.0/prometheus-2.52.0.linux-amd64.tar.gz
    tar -xvzf prometheus-2.52.0.linux-amd64.tar.gz
    cd prometheus-2.52.0.linux-amd64/

  2. Edit the prometheus.yml file to scrape the NIM for BGR endpoint:

    scrape_configs:
      - job_name: "nim-bgr"
        static_configs:
          - targets: ["localhost:8000"]

  3. Start the Prometheus server:

    ./prometheus --config.file=./prometheus.yml

  4. Verify the target in Prometheus by navigating to http://localhost:9090/targets?search=.

Grafana

Visualize metrics with Grafana.

  1. Install the latest Grafana release for your system:

    wget https://dl.grafana.com/oss/release/grafana-11.0.0.linux-amd64.tar.gz
    tar -zxvf grafana-11.0.0.linux-amd64.tar.gz
    cd grafana-v11.0.0/

  2. Start the Grafana server:

    ./bin/grafana-server

  3. Access the dashboard by navigating to http://localhost:3000 and log in with the default credentials:

    • Username: admin
    • Password: admin

  4. Configure the Prometheus data source:

    • Click Data Source.
    • Select Prometheus.
    • Set the URL to localhost:9090.
    • Save the configuration.