Skip to navigation

Observability

View as MarkdownOpen in Claude

NIM provides Prometheus metrics for Relight requests, the GPU, the server process, and the Python runtime. For per-model inference counts and latencies, use the Triton metrics. You can use these metrics to create dashboards with Grafana. By default, these metrics are available at http://0.0.0.0:8000/v1/metrics.

These commands need a NIM launched as in Getting Started, with ports 8000 and 9002 published, and curl.

You can use the following command to retrieve the metrics:

curl -X 'GET' 'http://localhost:8000/v1/metrics'

The following table describes the available metrics.

CategoryMetric NameDescription
GPUgpu_power_usage_wattsGPU instantaneous power, in watts
gpu_power_limit_wattsMaximum GPU power limit, in watts
gpu_total_energy_consumption_joulesGPU total energy consumption, in joules
gpu_utilizationGPU utilization rate (0.0–1.0)
gpu_memory_total_bytesTotal GPU memory, in bytes
gpu_memory_used_bytesUsed GPU memory, in bytes
Processprocess_virtual_memory_bytesVirtual memory size in bytes
process_resident_memory_bytesResident memory size in bytes
process_start_time_secondsStart time of the process since Unix epoch in seconds
process_cpu_seconds_totalTotal user and system CPU time spent in seconds
process_open_fdsNumber of open file descriptors
process_max_fdsMaximum number of open file descriptors
Pythonpython_gc_objects_collected_totalObjects collected during GC
python_gc_objects_uncollectable_totalUncollectable objects found during GC
python_gc_collections_totalNumber of times this generation was collected
python_infoPython platform information
Requestrequest_count_totalRelight requests, by http_status: the HTTP equivalent of the gRPC status (200 OK, 400 invalid input, 429 at capacity, 499 cancelled, 503 unavailable, 500 internal error)
request_latency_secondsRelight request duration, from the start of the call to the end of the response stream, in seconds

Both request metrics carry endpoint="/nvidia.ai4m.relighting.v1.VideoRelightingService/Relight" and count every request, including requests the NIM rejects before inference, such as RESOURCE_EXHAUSTED or INVALID_ARGUMENT. The process_* and python_* metrics describe the NIM’s HTTP front-end process, not the gRPC inference service or Triton. The Triton failure counters below count only inference that fails inside Triton.

HTTP Endpoints

Besides /v1/metrics, the HTTP port serves the following endpoints:

curl -s localhost:8000/v1/health/live # liveness: the server process is up
curl -s localhost:8000/v1/health/ready # readiness: 200 after the models are warmed up, 503 before
curl -s localhost:8000/v1/metadata # asset, license, model, and version metadata
curl -s localhost:8000/v1/manifest # model manifest
curl -s localhost:8000/v1/license # license information

Triton Metrics

NIM also exposes Triton Inference Server metrics that provide detailed information about model inference performance, request handling, and system utilization. By default, these Triton metrics are available at http://localhost:9002/metrics.

Use the following command to retrieve the Triton metrics:

curl localhost:9002/metrics

The following are key Triton metrics:

  • Request metrics: Counts of successful and failed inference requests.
  • Inference metrics: Request queue times, compute times, and overall request durations.
  • Model metrics: Model loading times, execution counts, and batch statistics.
  • Memory metrics: GPU and CPU memory usage for inference operations.

For comprehensive documentation on all available Triton metrics, their descriptions, and usage examples, see Metrics in the Triton Inference Server guide.