Observability
Platform Monitoring
Platform-level monitoring—GPU (DCGM), SR-IOV network, Precision Time Protocol, and NMOS Registry dashboards—is provided by Holoscan for Media and documented in Platform Monitoring in the Holoscan for Media user guide.
The following sections cover monitoring specific to the NIM.
Health Probes
Charts and the operator configure HTTP probes on the application health port (default 8000):
Check health from inside the cluster:
Port-forward for local debugging:
HTTP Endpoints
The NIM exposes the following HTTP endpoints on port 8000. Use port-forward to access them locally:
Useful Queries
Confirm which model and profile are loaded:
Check the NIM release and API version:
List all available endpoints:
Prometheus Metrics
The NIM exposes Prometheus metrics at http://<pod-ip>:8000/v1/metrics. These metrics can be integrated into the Grafana monitoring stack provided by Holoscan for Media; refer to Platform Monitoring.
Retrieve metrics from inside the cluster:
The following metrics are available:
Triton Inference Server Metrics
The NIM container also runs Triton Inference Server locally. Triton metrics are exposed on port 9002 (/metrics) inside the pod:
On Red Hat OpenShift, replace kubectl with oc.
Expose port 9002 only if your security model allows it. Use a service, PodMonitor, or scrape configuration as appropriate for your cluster.
Triton Latency Metrics
The service exposes Triton latency metrics in microseconds (us).
Metrics
Supported Quantiles
Example Metric
PromQL Examples
Average request latency (ms):
P95 request latency (ms):
P99 request latency (ms):
For all Triton metrics, refer to Metrics in the Triton Inference Server user guide.
Operator Controller Metrics
The operator controller-manager exposes controller-runtime metrics on the manager metrics service (HTTPS on port 8443 in the shipped nvidia-lipsync-h4m-operator chart). Refer to the operator Helm chart values.yaml and templates/service-metrics.yaml if your release overrides bind address or TLS.
Logs
Pod Logs
Stream logs from the NIM pod using the deployment name:
To retrieve logs from a specific container within the pod, add --container <container-name>. List all containers in a pod with the following command:
On Red Hat OpenShift, replace kubectl with oc.
Persistent Log Files
When the serverLogs persistent volume claim is enabled, the chart mounts a PVC at /var/log/lipsync and the NIM writes time-stamped log files there. This allows log retention beyond the lifetime of the pod and is useful for post-mortem analysis.
Enable via Helm:
Or enable via the operator custom resource under spec.parameters.serverLogs.
To inspect log files directly from a running pod:
For the full set of serverLogs options, refer to Common Helm Configuration.
NMOS
For NMOS deployments, verify device registrations using the NMOS Registry independently of the NIM health endpoints. NMOS Registry monitoring and dashboards are covered in Platform Monitoring in the Holoscan for Media user guide.