Skip to navigation

Platform Monitoring

Platform-level monitoring—GPU (DCGM), SR-IOV network, Precision Time Protocol, and NMOS Registry dashboards—is provided by Holoscan for Media and documented in Platform Monitoring in the Holoscan for Media user guide.

The following sections cover monitoring specific to the NIM.


Health Probes

Charts and the operator configure HTTP probes on the application health port (default 8000):

ProbeEndpointPurpose
Startup/v1/health/liveChecks whether the container has started.
Liveness/v1/health/readyChecks whether the process is alive.
Readiness/v1/health/readyChecks whether the service is ready to serve traffic.

Check health from inside the cluster:

kubectl exec -it deploy/<nim-deployment> -- curl -sS localhost:8000/v1/health/ready

When the NIM is ready, the response body is as follows:

{"object":"health.response","message":"ASD NIM is ready","status":"ready"}

Port-forward for local debugging:

kubectl port-forward deploy/<nim-deployment> 8000:8000
curl -sS http://127.0.0.1:8000/v1/health/ready

HTTP Endpoints

The NIM exposes the following HTTP endpoints on port 8000. Use port-forward to access them locally:

kubectl port-forward deploy/<nim-deployment> 8000:8000
EndpointDescription
GET /v1/health/liveLiveness check—returns 200 when the container is running.
GET /v1/health/readyReadiness check—returns 200 when the NIM is ready to process streams.
GET /v1/metricsPrometheus metrics (NIM process and GPU). For details, refer to Prometheus Metrics.
GET /v1/versionReturns the NIM release and API version: {"release":"1.0.0","api":"3.1.0"}.
GET /v1/metadataReturns the loaded model URL and selected profile checksum. Useful for confirming which model is active without inspecting the container filesystem.
GET /v1/manifestReturns full profile resolution details, including GPU type and workspace hash for each available profile.
GET /v1/modelsLists the models currently loaded in the NIM.
GET /v1/licenseReturns license information for the running NIM.
GET /docsSwagger UI—interactive API documentation.
GET /openapi.jsonFull OpenAPI schema for all endpoints.

Useful Queries

Confirm which model and profile are loaded:

curl -s http://localhost:8000/v1/metadata | jq '.modelInfo[0].modelUrl, .selectedModelProfileId'

Check the NIM release and API version:

curl -s http://localhost:8000/v1/version

List all available endpoints:

curl -s http://localhost:8000/openapi.json | jq '.paths | keys'

Prometheus Metrics

The NIM exposes Prometheus metrics at http://<pod-ip>:8000/v1/metrics. These metrics can be integrated into the Grafana monitoring stack provided by Holoscan for Media; refer to Platform Monitoring.

Retrieve metrics from inside the cluster:

kubectl exec -it deploy/<nim-deployment> -- curl -sS localhost:8000/v1/metrics

The following metrics are available:

CategoryMetric NameDescription
GPUgpu_power_usage_wattsGPU instantaneous power, in watts.
GPUgpu_power_limit_wattsMaximum GPU power limit, in watts.
GPUgpu_total_energy_consumption_joulesGPU total energy consumption, in joules.
GPUgpu_utilizationGPU utilization rate (0.0–1.0).
GPUgpu_memory_total_bytesTotal GPU memory, in bytes.
GPUgpu_memory_used_bytesUsed GPU memory, in bytes.
Processprocess_virtual_memory_bytesVirtual memory size, in bytes.
Processprocess_resident_memory_bytesResident memory size, in bytes.
Processprocess_start_time_secondsStart time of the process since Unix epoch, in seconds.
Processprocess_cpu_seconds_totalTotal user and system CPU time, in seconds.
Processprocess_open_fdsNumber of open file descriptors.
Processprocess_max_fdsMaximum number of open file descriptors.
Pythonpython_gc_objects_collected_totalObjects collected during garbage collection.
Pythonpython_gc_objects_uncollectable_totalUncollectable objects found during garbage collection.
Pythonpython_gc_collections_totalNumber of times this generation was collected.
Pythonpython_infoPython platform information.

Triton Inference Server Metrics

The NIM container also runs Triton Inference Server locally. Triton metrics are exposed on port 9002 (/metrics) inside the pod:

kubectl exec -it deploy/<nim-deployment> -- curl -sS localhost:9002/metrics | head

Expose port 9002 only if your security model allows it. Use a service, PodMonitor, or scrape configuration as appropriate for your cluster.

Key Triton metrics include the following:

  • Request metrics: Counts of successful and failed inference requests.
  • Inference metrics: Request queue times, compute times, and overall request durations.
  • Model metrics: Model loading times, execution counts, and batch statistics.
  • Memory metrics: GPU and CPU memory usage for inference operations.
  • Cache metrics: Response cache hit and miss rates (when caching is enabled).

For comprehensive documentation on all available Triton metrics, refer to Metrics in the Triton Inference Server guide.

Operator Controller Metrics

The operator controller-manager exposes controller-runtime metrics on the manager metrics service (TLS, port per operator chart defaults). Refer to the operator Helm chart values.yaml for the metrics port and service configuration.

Logs

Pod Logs

Stream logs from the NIM pod using the deployment name:

kubectl logs deploy/<nim-deployment> --follow

To retrieve logs from a specific container within the pod, add --container <container-name>. List all containers in a pod with the following command:

kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].name}'

On Red Hat OpenShift, replace kubectl with oc.

Persistent Log Files

When the nimLogs persistent volume claim is enabled, the NIM writes time-stamped log files to the configured mount path (default /workspace/nim-logs). This allows log retention beyond the lifetime of the pod and is useful for post-mortem analysis.

Enable via Helm:

nimLogs:
enabled: true
size: "5Gi"
mountPath: "/workspace/nim-logs"

Or via the operator custom resource under spec.parameters.nimLogs.

To inspect log files directly from a running pod:

kubectl exec deploy/<nim-deployment> -- ls /workspace/nim-logs

For the full set of nimLogs options, see Configuration Reference.

NMOS

For NMOS deployments, verify device registrations using the NMOS Registry independently of the NIM health endpoints. NMOS Registry monitoring and dashboards are covered in Platform Monitoring in the Holoscan for Media user guide.