> This page is for NVIDIA Active Speaker Detection on Holoscan for Media, version 1.0.0 (Latest) (default).
> For other versions, use one of these documentation indexes:
> - 1.0.0 (Latest) (default): https://docs.nvidia.com/nim/maxine/active-speaker-detection-h4m/1.0.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nim/maxine/llms.txt.

# Observability

## Platform Monitoring

Platform-level monitoring—GPU (DCGM), SR-IOV network, Precision Time Protocol, and NMOS Registry dashboards—is provided by Holoscan for Media and documented in [Platform Monitoring](https://docs.nvidia.com/holoscan-for-media/latest/user-guide/platform-monitoring/monitoring-and-dashboard.html) in the Holoscan for Media user guide.

The following sections cover monitoring specific to the NIM.

---

## Health Probes

Charts and the operator configure HTTP probes on the application health port (default `8000`):


| Probe     | Endpoint           | Purpose                                         |
| --------- | ------------------ | ----------------------------------------------- |
| Startup   | `/v1/health/live`  | Checks whether the container has started.             |
| Liveness  | `/v1/health/ready` | Checks whether the process is alive.                  |
| Readiness | `/v1/health/ready` | Checks whether the service is ready to serve traffic. |


Check health from inside the cluster:

```bash
kubectl exec -it deploy/<nim-deployment> -- curl -sS localhost:8000/v1/health/ready
```

When the NIM is ready, the response body is as follows:

```json
{"object":"health.response","message":"ASD NIM is ready","status":"ready"}
```

Port-forward for local debugging:

```bash
kubectl port-forward deploy/<nim-deployment> 8000:8000
curl -sS http://127.0.0.1:8000/v1/health/ready
```

## HTTP Endpoints

The NIM exposes the following HTTP endpoints on port `8000`. Use `port-forward` to access them locally:

```bash
kubectl port-forward deploy/<nim-deployment> 8000:8000
```

| Endpoint | Description |
|----------|-------------|
| `GET /v1/health/live` | Liveness check—returns `200` when the container is running. |
| `GET /v1/health/ready` | Readiness check—returns `200` when the NIM is ready to process streams. |
| `GET /v1/metrics` | Prometheus metrics (NIM process and GPU). For details, refer to [Prometheus Metrics](#prometheus-metrics). |
| `GET /v1/version` | Returns the NIM release and API version: `{"release":"1.0.0","api":"3.1.0"}`. |
| `GET /v1/metadata` | Returns the loaded model URL and selected profile checksum. Useful for confirming which model is active without inspecting the container filesystem. |
| `GET /v1/manifest` | Returns full profile resolution details, including GPU type and workspace hash for each available profile. |
| `GET /v1/models` | Lists the models currently loaded in the NIM. |
| `GET /v1/license` | Returns license information for the running NIM. |
| `GET /docs` | Swagger UI—interactive API documentation. |
| `GET /openapi.json` | Full OpenAPI schema for all endpoints. |


### Useful Queries

Confirm which model and profile are loaded:

```bash
curl -s http://localhost:8000/v1/metadata | jq '.modelInfo[0].modelUrl, .selectedModelProfileId'
```

Check the NIM release and API version:

```bash
curl -s http://localhost:8000/v1/version
```

List all available endpoints:

```bash
curl -s http://localhost:8000/openapi.json | jq '.paths | keys'
```

---

## Prometheus Metrics

The NIM exposes [Prometheus](https://prometheus.io/) metrics at `http://<pod-ip>:8000/v1/metrics`. These metrics can be integrated into the Grafana monitoring stack provided by Holoscan for Media; refer to [Platform Monitoring](https://docs.nvidia.com/holoscan-for-media/latest/user-guide/platform-monitoring/monitoring-and-dashboard.html).

Retrieve metrics from inside the cluster:

```bash
kubectl exec -it deploy/<nim-deployment> -- curl -sS localhost:8000/v1/metrics
```

The following metrics are available:


| Category | Metric Name                             | Description                                            |
| -------- | --------------------------------------- | ------------------------------------------------------ |
| GPU      | `gpu_power_usage_watts`                 | GPU instantaneous power, in watts.                      |
| GPU      | `gpu_power_limit_watts`                 | Maximum GPU power limit, in watts.                      |
| GPU      | `gpu_total_energy_consumption_joules`   | GPU total energy consumption, in joules.                |
| GPU      | `gpu_utilization`                       | GPU utilization rate (0.0–1.0).                         |
| GPU      | `gpu_memory_total_bytes`                | Total GPU memory, in bytes.                             |
| GPU      | `gpu_memory_used_bytes`                 | Used GPU memory, in bytes.                              |
| Process  | `process_virtual_memory_bytes`          | Virtual memory size, in bytes.                          |
| Process  | `process_resident_memory_bytes`         | Resident memory size, in bytes.                         |
| Process  | `process_start_time_seconds`            | Start time of the process since Unix epoch, in seconds. |
| Process  | `process_cpu_seconds_total`             | Total user and system CPU time, in seconds.             |
| Process  | `process_open_fds`                      | Number of open file descriptors.                        |
| Process  | `process_max_fds`                       | Maximum number of open file descriptors.                |
| Python   | `python_gc_objects_collected_total`     | Objects collected during garbage collection.            |
| Python   | `python_gc_objects_uncollectable_total` | Uncollectable objects found during garbage collection.  |
| Python   | `python_gc_collections_total`           | Number of times this generation was collected.          |
| Python   | `python_info`                           | Python platform information.                            |


## Triton Inference Server Metrics

The NIM container also runs Triton Inference Server locally. Triton metrics are exposed on port `9002` (`/metrics`) inside the pod:

```bash
kubectl exec -it deploy/<nim-deployment> -- curl -sS localhost:9002/metrics | head
```

<Note>
Expose port 9002 only if your security model allows it. Use a service, PodMonitor, or scrape configuration as appropriate for your cluster.
</Note>

Key Triton metrics include the following:

- **Request metrics:** Counts of successful and failed inference requests.
- **Inference metrics:** Request queue times, compute times, and overall request durations.
- **Model metrics:** Model loading times, execution counts, and batch statistics.
- **Memory metrics:** GPU and CPU memory usage for inference operations.
- **Cache metrics:** Response cache hit and miss rates (when caching is enabled).

For comprehensive documentation on all available Triton metrics, refer to [Metrics](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/metrics.html) in the Triton Inference Server guide.

## Operator Controller Metrics

The operator `controller-manager` exposes controller-runtime metrics on the manager metrics service (TLS, port per operator chart defaults). Refer to the operator Helm chart `values.yaml` for the metrics port and service configuration.

## Logs

### Pod Logs

Stream logs from the NIM pod using the deployment name:

```bash
kubectl logs deploy/<nim-deployment> --follow
```

To retrieve logs from a specific container within the pod, add `--container <container-name>`. List all containers in a pod with the following command:

```bash
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].name}'
```

On Red Hat OpenShift, replace `kubectl` with `oc`.

### Persistent Log Files

When the `nimLogs` persistent volume claim is enabled, the NIM writes time-stamped log files to the configured mount path (default `/workspace/nim-logs`). This allows log retention beyond the lifetime of the pod and is useful for post-mortem analysis.

Enable via Helm:

```yaml
nimLogs:
  enabled: true
  size: "5Gi"
  mountPath: "/workspace/nim-logs"
```

Or via the operator custom resource under `spec.parameters.nimLogs`.

To inspect log files directly from a running pod:

```bash
kubectl exec deploy/<nim-deployment> -- ls /workspace/nim-logs
```

For the full set of `nimLogs` options, see [Configuration Reference](/nim/maxine/active-speaker-detection-h4m/configuration/common#model-cache-and-logs-pvcs).

## NMOS

For NMOS deployments, verify device registrations using the NMOS Registry independently of the NIM health endpoints. NMOS Registry monitoring and dashboards are covered in [Platform Monitoring](https://docs.nvidia.com/holoscan-for-media/latest/user-guide/platform-monitoring/monitoring-and-dashboard.html#nmos-registry-monitoring) in the Holoscan for Media user guide.