> This page is for OpenFold3, version 1.6.0 (Latest) (default).
> For other versions, use one of these documentation indexes:
> - 1.6.0 (Latest) (default): https://docs.nvidia.com/nim/bionemo/openfold3/1.6.0/llms.txt
> - 1.5.0: https://docs.nvidia.com/nim/bionemo/openfold3/1.5.0/llms.txt
> - 1.4.0: https://docs.nvidia.com/nim/bionemo/openfold3/1.4.0/llms.txt
> - 1.3.0: https://docs.nvidia.com/nim/bionemo/openfold3/1.3.0/llms.txt
> - 1.2.0: https://docs.nvidia.com/nim/bionemo/openfold3/1.2.0/llms.txt
> - 1.1.0: https://docs.nvidia.com/nim/bionemo/openfold3/1.1.0/llms.txt
> - 1.0.0: https://docs.nvidia.com/nim/bionemo/openfold3/1.0.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nim/bionemo/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nim/bionemo/_mcp/server.

# Observability for OpenFold3 NIM

Use this documentation to learn about observability for OpenFold3 NIM.

## About Observability

OpenFold3 NIM exposes health check endpoints and metrics, enabling monitoring of service health, model performance, resource utilization, and request patterns.

## Health Check Endpoints

OpenFold3 NIM provides health check endpoints that can be used for Kubernetes liveness and readiness probes.

### Readiness Check

Endpoint path: `/v1/health/ready`

**Purpose:** Indicates when the container is ready to accept traffic.

**Response:** Returns HTTP 200 when the service is ready, and HTTP 503 while it is not.

**Example:**

```bash
curl http://localhost:8000/v1/health/ready
```

### Liveness Check

Endpoint path: `/v1/health/live`

**Purpose:** Indicates when to restart the container.

**Response:** Returns HTTP 200 when the service is live.

**Example:**

```bash
curl http://localhost:8000/v1/health/live
```

These endpoints return HTTP 200 when the service is ready/live and can be used for Kubernetes liveness and readiness probes.

## Metrics

OpenFold3 NIM exposes metrics at the `/v1/metrics` endpoint. You can view them directly:

```bash
curl http://localhost:8000/v1/metrics
```

### Available Metrics Summary

**GPU Metrics:**

- GPU utilization rate (0.0 - 1.0)
- GPU memory usage and total memory (bytes)
- GPU power consumption (watts) and total energy (joules)

**Request Metrics:**
- Total request count
- Request latency distribution

**Process Metrics:**
- CPU time, memory usage (resident and virtual)
- Open file descriptors

**Python Runtime:**
- Garbage collection statistics
- Python version information

## Collect Metrics With OpenTelemetry Collector

[OpenTelemetry](https://opentelemetry.io/) is a vendor-neutral observability framework that provides a standardized way to collect, process, and export telemetry data (metrics, logs, and traces).

The [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) acts as an intermediary service that:
- Receives metrics from OpenFold3 NIM's `/v1/metrics` endpoint using a Prometheus receiver
- Processes the data (batching, filtering, enrichment)
- Exports to various backends (for example, Prometheus, Datadog, New Relic, and Zipkin)

### Prerequisites

An OpenTelemetry Collector must be running and configured to scrape metrics.

### OpenTelemetry Collector Configuration

Create a configuration file `otel-collector-config.yaml`:

```yaml
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: nim-metrics
          scrape_interval: 10s
          metrics_path: "/v1/metrics"
          static_configs:
            - targets: ["localhost:8000"]
              labels:
                service: "openfold3-nim"

processors:
  batch:
    timeout: 10s

exporters:
  # Debug exporter - prints to console
  debug:
    verbosity: detailed
  
  # Prometheus exporter - exposes metrics on port 8889
  prometheus:
    endpoint: "0.0.0.0:8889"
    namespace: nim

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [debug, prometheus]
```

### Check for Port Conflicts

The Collector serves its own internal telemetry on port 8888 by default. Before running the OpenTelemetry Collector, check if port 8888 is already in use:

```bash
# On Linux/macOS
lsof -i :8888

# Or using netstat
netstat -tuln | grep 8888
```

If port 8888 is occupied, use the alternative configuration below, which uses different ports.

### Alternative Configuration for Port Conflicts

If port 8888 is already in use, use this configuration (`otel-collector-config.yaml`), which binds to ports 8887 and 8889 instead:

```yaml
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: "nim-app"
          metrics_path: "/v1/metrics"
          static_configs:
            - targets: ["localhost:8000"]

processors:
  batch: {}

exporters:
  debug:
    verbosity: detailed

  prometheus:
    endpoint: "0.0.0.0:8889"
    namespace: nim

service:
  telemetry:
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: "0.0.0.0"
                port: 8887

  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [debug, prometheus]
```

This configuration:
- Uses port 8887 for the collector's internal telemetry metrics
- Uses port 8889 for the Prometheus exporter endpoint
- Avoids conflicts with services running on port 8888

### Run the Collector

Start the OpenTelemetry Collector with the configuration:

```bash
docker run -d --name otel-collector \
  --network host \
  -v $(pwd)/otel-collector-config.yaml:/etc/otelcol/config.yaml \
  otel/opentelemetry-collector:latest
```

The collector:

- Scrapes metrics from `localhost:8000/v1/metrics` every 10 seconds
- Exposes aggregated metrics at `http://localhost:8889/metrics`
- Exposes the collector's own telemetry at `http://localhost:8887/metrics` (if using the alternative configuration)

With the collected data, you can export to monitoring backends like Datadog, New Relic, or Zipkin by configuring additional exporters.

## Related Information

For more information about OpenTelemetry and configuring the collector:

- [OpenTelemetry Collector Installation](https://opentelemetry.io/docs/collector/installation/) - Installation guide for different platforms
- [OpenTelemetry Collector Configuration](https://opentelemetry.io/docs/collector/configuration/) - Detailed configuration reference

**Receivers:**
- [Prometheus Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/prometheusreceiver/README.md) - Used to scrape metrics from `/v1/metrics` endpoint

**Exporters:**
- [Prometheus Exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/prometheusexporter/README.md) - Export metrics in Prometheus format
- [OTLP gRPC Exporter](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/otlpexporter/README.md) - Export to OTLP-compatible backends
- [Debug Exporter](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/debugexporter/README.md) - Print metrics to console for testing

**Integration Examples:**
- [OTLP Ingestion by Datadog Agent](https://docs.datadoghq.com/opentelemetry/interoperability/otlp_ingest_in_the_agent/?tab=host) - Integration with Datadog monitoring