Observability
NIM provides Prometheus metrics indicating request statistics. You can use these metrics to create Grafana dashboards. By default, these metrics are available at http://0.0.0.0:8000/v1/metrics.
Use the following command to retrieve the metrics:
The following table describes the available metrics.
Triton Metrics
NIM also exposes Triton Inference Server metrics that provide detailed information about model inference performance, request handling, and system utilization. By default, these Triton metrics are available at http://localhost:9002/metrics.
Use the following command to retrieve the Triton metrics:
Note: When running NIM in a container, ensure that port 9002 is properly forwarded by including the
-p 9002:9002flag in your Docker run command.
The following are key Triton metrics:
- Request metrics: Counts of successful and failed inference requests.
- Inference metrics: Request queue times, compute times, and overall request durations.
- Model metrics: Model loading times, execution counts, and batch statistics.
- Memory metrics: GPU and CPU memory usage for inference operations.
- Cache metrics: Response cache hit and miss rates (when caching is enabled).
For comprehensive documentation on all available Triton metrics, their descriptions, and usage examples, see Metrics in the Triton Inference Server guide.