Skip to navigation

Observability for OpenFold3 NIM

View as Markdown

Use this documentation to learn about observability for OpenFold3 NIM.

About Observability

OpenFold3 NIM exposes health check endpoints and metrics, enabling monitoring of service health, model performance, resource utilization, and request patterns.

Health Check Endpoints

OpenFold3 NIM provides health check endpoints that can be used for Kubernetes liveness and readiness probes.

Readiness Check

Endpoint path: /v1/health/ready

Purpose: Indicates when the container is ready to accept traffic.

Response: Returns HTTP 200 when the service is ready, and HTTP 503 while it is not.

Example:

curl http://localhost:8000/v1/health/ready

Liveness Check

Endpoint path: /v1/health/live

Purpose: Indicates when to restart the container.

Response: Returns HTTP 200 when the service is live.

Example:

curl http://localhost:8000/v1/health/live

These endpoints return HTTP 200 when the service is ready/live and can be used for Kubernetes liveness and readiness probes.

Metrics

OpenFold3 NIM exposes metrics at the /v1/metrics endpoint. You can view them directly:

curl http://localhost:8000/v1/metrics

Available Metrics Summary

GPU Metrics:

  • GPU utilization rate (0.0 - 1.0)
  • GPU memory usage and total memory (bytes)
  • GPU power consumption (watts) and total energy (joules)

Request Metrics:

  • Total request count
  • Request latency distribution

Process Metrics:

  • CPU time, memory usage (resident and virtual)
  • Open file descriptors

Python Runtime:

  • Garbage collection statistics
  • Python version information

Collect Metrics With OpenTelemetry Collector

OpenTelemetry is a vendor-neutral observability framework that provides a standardized way to collect, process, and export telemetry data (metrics, logs, and traces).

The OpenTelemetry Collector acts as an intermediary service that:

  • Receives metrics from OpenFold3 NIM’s /v1/metrics endpoint using a Prometheus receiver
  • Processes the data (batching, filtering, enrichment)
  • Exports to various backends (for example, Prometheus, Datadog, New Relic, and Zipkin)

Prerequisites

An OpenTelemetry Collector must be running and configured to scrape metrics.

OpenTelemetry Collector Configuration

Create a configuration file otel-collector-config.yaml:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: nim-metrics
          scrape_interval: 10s
          metrics_path: "/v1/metrics"
          static_configs:
            - targets: ["localhost:8000"]
              labels:
                service: "openfold3-nim"
processors:
  batch:
    timeout: 10s
exporters:
  # Debug exporter - prints to console
  debug:
    verbosity: detailed
  
  # Prometheus exporter - exposes metrics on port 8889
  prometheus:
    endpoint: "0.0.0.0:8889"
    namespace: nim
service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [debug, prometheus]

Check for Port Conflicts

The Collector serves its own internal telemetry on port 8888 by default. Before running the OpenTelemetry Collector, check if port 8888 is already in use:

# On Linux/macOS
lsof -i :8888
# Or using netstat
netstat -tuln | grep 8888

If port 8888 is occupied, use the alternative configuration below, which uses different ports.

Alternative Configuration for Port Conflicts

If port 8888 is already in use, use this configuration (otel-collector-config.yaml), which binds to ports 8887 and 8889 instead:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: "nim-app"
          metrics_path: "/v1/metrics"
          static_configs:
            - targets: ["localhost:8000"]
processors:
  batch: {}
exporters:
  debug:
    verbosity: detailed
  prometheus:
    endpoint: "0.0.0.0:8889"
    namespace: nim
service:
  telemetry:
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: "0.0.0.0"
                port: 8887
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [debug, prometheus]

This configuration:

  • Uses port 8887 for the collector’s internal telemetry metrics
  • Uses port 8889 for the Prometheus exporter endpoint
  • Avoids conflicts with services running on port 8888

Run the Collector

Start the OpenTelemetry Collector with the configuration:

docker run -d --name otel-collector \
  --network host \
  -v $(pwd)/otel-collector-config.yaml:/etc/otelcol/config.yaml \
  otel/opentelemetry-collector:latest

The collector:

  • Scrapes metrics from localhost:8000/v1/metrics every 10 seconds
  • Exposes aggregated metrics at http://localhost:8889/metrics
  • Exposes the collector’s own telemetry at http://localhost:8887/metrics (if using the alternative configuration)

With the collected data, you can export to monitoring backends like Datadog, New Relic, or Zipkin by configuring additional exporters.

For more information about OpenTelemetry and configuring the collector:

Receivers:

Exporters:

Integration Examples: