> This page is for NVIDIA Relighting NIM, version 1.2.0 (Latest) (default).
> For other versions, use one of these documentation indexes:
> - 1.2.0 (Latest) (default): https://docs.nvidia.com/nim/maxine/relighting/1.2.0/llms.txt
> - 1.1.0: https://docs.nvidia.com/nim/maxine/relighting/1.1.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nim/maxine/llms.txt.

# Observability

NIM provides [Prometheus](https://prometheus.io/) metrics for Relight requests, the GPU, the server process, and the Python runtime. For per-model inference counts and latencies, use the [Triton metrics](#triton-metrics). You can use these metrics to create dashboards with [Grafana](https://grafana.com/docs/grafana/latest/dashboards/use-dashboards/). By default, these metrics are available at `http://0.0.0.0:8000/v1/metrics`.

These commands need a NIM launched as in [Getting Started](/nim/maxine/relighting/getting-started#launching-the-nim-container), with ports 8000 and 9002 published, and `curl`.

You can use the following command to retrieve the metrics:
```bash
curl -X 'GET' 'http://localhost:8000/v1/metrics'
```

The following table describes the available metrics.
| Category | Metric Name                              | Description                                       |
|----------|------------------------------------------|---------------------------------------------------|
| GPU      | gpu_power_usage_watts                    | GPU instantaneous power, in watts                 |
|       | gpu_power_limit_watts                    | Maximum GPU power limit, in watts                 |
|       | gpu_total_energy_consumption_joules      | GPU total energy consumption, in joules           |
|       | gpu_utilization                          | GPU utilization rate (0.0–1.0)                  |
|       | gpu_memory_total_bytes                   | Total GPU memory, in bytes                        |
|       | gpu_memory_used_bytes                    | Used GPU memory, in bytes                         |
| Process  | process_virtual_memory_bytes             | Virtual memory size in bytes                      |
|   | process_resident_memory_bytes            | Resident memory size in bytes                     |
|   | process_start_time_seconds               | Start time of the process since Unix epoch in seconds |
|   | process_cpu_seconds_total                | Total user and system CPU time spent in seconds   |
|   | process_open_fds                         | Number of open file descriptors                   |
|   | process_max_fds                          | Maximum number of open file descriptors           |
| Python   | python_gc_objects_collected_total        | Objects collected during GC                       |
|    | python_gc_objects_uncollectable_total    | Uncollectable objects found during GC             |
|    | python_gc_collections_total              | Number of times this generation was collected     |
|    | python_info                              | Python platform information                       |
| Request  | request_count_total                      | Relight requests, by `http_status`: the HTTP equivalent of the gRPC status (200 OK, 400 invalid input, 429 at capacity, 499 cancelled, 503 unavailable, 500 internal error) |
|   | request_latency_seconds                  | Relight request duration, from the start of the call to the end of the response stream, in seconds |

Both request metrics carry `endpoint="/nvidia.ai4m.relighting.v1.VideoRelightingService/Relight"` and count every request, including requests the NIM rejects before inference, such as `RESOURCE_EXHAUSTED` or `INVALID_ARGUMENT`. The `process_*` and `python_*` metrics describe the NIM's HTTP front-end process, not the gRPC inference service or Triton. The Triton failure counters below count only inference that fails inside Triton.

## HTTP Endpoints

Besides `/v1/metrics`, the HTTP port serves the following endpoints:

```bash
curl -s localhost:8000/v1/health/live    # liveness: the server process is up
curl -s localhost:8000/v1/health/ready   # readiness: 200 after the models are warmed up, 503 before
curl -s localhost:8000/v1/metadata       # asset, license, model, and version metadata
curl -s localhost:8000/v1/manifest       # model manifest
curl -s localhost:8000/v1/license        # license information
```

## Triton Metrics

NIM also exposes Triton Inference Server metrics that provide detailed information about model inference performance, request handling, and system utilization. By default, these Triton metrics are available at `http://localhost:9002/metrics`.

Use the following command to retrieve the Triton metrics:
```bash
curl localhost:9002/metrics
```


The following are key Triton metrics:

- **Request metrics:** Counts of successful and failed inference requests.
- **Inference metrics:** Request queue times, compute times, and overall request durations.
- **Model metrics:** Model loading times, execution counts, and batch statistics.
- **Memory metrics:** GPU and CPU memory usage for inference operations.

For comprehensive documentation on all available Triton metrics, their descriptions, and usage examples, see [Metrics](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/metrics.html) in the [Triton Inference Server](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/introduction/index.html) guide.