> This page is for NVIDIA Cluster Readiness Engine, version v0.6.0.
> For other versions, use one of these documentation indexes:
> - Latest · v0.7.0 (default): https://docs.nvidia.com/cluster-readiness-engine/v0.7.0/llms.txt
> - v0.6.0: https://docs.nvidia.com/cluster-readiness-engine/v0.6.0/llms.txt
> - v0.5.0: https://docs.nvidia.com/cluster-readiness-engine/v0.5.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/cluster-readiness-engine/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/cluster-readiness-engine/_mcp/server.

# Prometheus Metrics

> Complete reference for all Prometheus metrics exposed by the NVIDIA Cluster Readiness Engine controller.

This page is the full reference for Prometheus metrics exposed by the NVIDIA Cluster Readiness Engine (NVCRE) controller. All metrics are registered with the controller-runtime metrics registry and served on the `/metrics` endpoint.

## Scrape configuration

The Helm chart ships a `ServiceMonitor` for the Prometheus Operator (enabled by default via `metrics.serviceMonitor.enabled`):

```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nvcre-metrics-monitor
  namespace: nvcre
  labels:
    release: prometheus
spec:
  endpoints:
    - path: /metrics
      port: https
      scheme: https
      honorLabels: true
      bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      tlsConfig:
        insecureSkipVerify: true  # Use cert-manager in production
  selector:
    matchLabels:
      control-plane: manager
```

`honorLabels: true` (chart default `metrics.serviceMonitor.honorLabels`) keeps the controller's own `namespace` and `job` labels on each series. Without it, Prometheus prefers scrape-target labels and renames the metric labels to `exported_namespace` / `exported_job`, so the PromQL examples below would match the controller Service rather than the Certification Job. Clusters that already built dashboards on `exported_*` can set `metrics.serviceMonitor.honorLabels=false`.

Tune scrape cadence with `metrics.serviceMonitor.interval` and `metrics.serviceMonitor.scrapeTimeout` (empty uses the Prometheus defaults). kube-prometheus-stack users should set `metrics.serviceMonitor.labels.release` to their Prometheus Helm release name (default `prometheus`). See `helm/cluster-readiness-engine/values.yaml` for the full set of knobs.

## Job status metrics

| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `nvcre_job_status` | Gauge | `namespace`, `job`, `workflow`, `status` | Current status of burn-in Jobs. Value is `1` for the current status, `0` for others. Status values: `in_progress`, `succeeded`, `failed`. |

The gauge is set for all three status values on each update, ensuring that a transition from `in_progress` to `succeeded` also zeroes out the `in_progress` series. Metrics are cleaned up when a Job is deleted.

## Certification and Workflow status metrics

Certification and Workflow conditions are the operator-facing source of truth (`kubectl get certifications,workflows`). These gauges mirror `nvcre_job_status` so a status panel can show the verdict of a run even when Job series are missing, stale, or wrong.

| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `nvcre_certification_status` | Gauge | `namespace`, `certification`, `status` | Current status of Certifications. Value is `1` for the current status, `0` for others. Status values: `in_progress`, `succeeded`, `failed` (mapped from the InProgress / Succeeded / Failed condition types). |
| `nvcre_workflow_status` | Gauge | `namespace`, `workflow`, `certification`, `status` | Current status of Workflows. Value is `1` for the current status, `0` for others. Same status values as Certification. `certification` is the value of the Workflow's `nvcre.nvidia.com/certification` label, or empty when that label is absent. |

These series are built at scrape time from the elected leader's informer cache. Each scrape lists Certifications and Workflows and emits `1` for the true InProgress / Succeeded / Failed condition and `0` for the peers. An object with no phase condition `True` is omitted. Series disappear on the next scrape after the object leaves the cache; a changed `nvcre.nvidia.com/certification` label reports the new value and does not leave old series. Standby replicas emit nothing. Reason strings are not exported as labels.

These gauges do not change how or when `nvcre_job_status` is updated.

## Hardware failure metrics

| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `nvcre_hardware_failed_jobs_total` | Counter | `namespace`, `job`, `workflow` | Total number of Jobs that detected hardware failures. Incremented once per Job on first detection. |
| `nvcre_job_failed_nodes` | Gauge | `namespace`, `job`, `workflow` | Number of nodes with detected hardware failures per Job. |
| `nvcre_hardware_failures_detected_total` | Counter | `namespace`, `job`, `workflow`, `node` | Total number of hardware failures detected across all evaluations. Per-node granularity. |

## Node health check metrics

| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `nvcre_node_health_check_duration_seconds` | Histogram | `namespace`, `job`, `workflow` | Duration of node health check operations in seconds. Buckets: 1ms to ~16s (exponential). |
| `nvcre_nodes_evaluated_total` | Counter | `namespace`, `job`, `workflow` | Total number of node health evaluations performed. |

## Reconciliation metrics

| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `nvcre_reconcile_total` | Counter | `namespace`, `job`, `workflow`, `result` | Total number of reconciliation attempts. `result` values: `success`, `error`, `requeue`. |
| `nvcre_reconcile_duration_seconds` | Histogram | `namespace`, `job`, `workflow` | Duration of reconciliation operations in seconds. Buckets: 10ms to ~20s (exponential). |

## Workload metrics

| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `nvcre_workload_created_total` | Counter | `namespace`, `job`, `workflow` | Total number of workloads created by the Job controller. |

## Goodput metrics

All goodput metrics share the same label set: `namespace`, `measurement`, `job`, `workflow`.

| Metric | Type | Description |
|--------|------|-------------|
| `nvcre_goodput_ratio` | Gauge | Runtime goodput ratio (0.0 to 1.0) |
| `nvcre_goodput_avg_tflops_per_gpu` | Gauge | Average TFLOPS per GPU from goodput measurement |
| `nvcre_goodput_training_time_seconds` | Gauge | Total training wall-clock time in seconds |
| `nvcre_goodput_reschedule_time_seconds` | Gauge | Cumulative reschedule time in seconds |
| `nvcre_goodput_resume_time_seconds` | Gauge | Cumulative resume time in seconds |
| `nvcre_goodput_checkpoint_save_time_seconds` | Gauge | Cumulative checkpoint save time in seconds |
| `nvcre_goodput_warmup_time_seconds` | Gauge | Total warmup step time in seconds |
| `nvcre_goodput_non_warmup_time_seconds` | Gauge | Total non-warmup step time in seconds |
| `nvcre_goodput_avg_step_time_seconds` | Gauge | Average time per training step in seconds |
| `nvcre_goodput_lost_work_time_seconds` | Gauge | Cumulative lost work time in seconds (work done after last checkpoint, lost on restart) |

Goodput metrics are cleaned up when a GoodputMeasurement is deleted.

### Metric lifecycle

Goodput metrics are cleaned up at specific lifecycle events to prevent stale data in Prometheus:

| Event | What is cleaned up | What is preserved |
|-------|--------------------|-------------------|
| Job completes (succeeded or failed) | Operational metrics (every `nvcre_goodput_*` gauge except the ratio: TFLOPS, step time, training/warmup/non-warmup time, checkpoint, reschedule, resume, lost work) | `nvcre_goodput_ratio` (the outcome) |
| Job restarts from checkpoint | Instantaneous metrics (TFLOPS, avg step time) | All cumulative metrics |
| GoodputMeasurement deleted | All goodput metrics for that measurement | — |
| BandwidthMeasurement completes or is deleted | All NCCL bandwidth metrics for that measurement | — |

## NCCL bandwidth metrics

All NCCL bandwidth metrics share the same label set: `namespace`, `measurement`, `job`, `workflow`, `nccl_test`, `message_size_bytes`. Values are **gigabytes per second** (the nccl-tests `algbw` / `busbw` columns), not gigabits.

| Metric | Type | Description |
|--------|------|-------------|
| `nvcre_nccl_algbw_gbs` | Gauge | NCCL algorithmic bandwidth in GB/s per message size |
| `nvcre_nccl_busbw_gbs` | Gauge | NCCL bus bandwidth in GB/s per message size |
| `nvcre_nccl_algbw_gbps` | Gauge | **Deprecated.** Alias of `nvcre_nccl_algbw_gbs`. Dual-registered for one minor release; the `_gbps` suffix incorrectly implied gigabits. |
| `nvcre_nccl_busbw_gbps` | Gauge | **Deprecated.** Alias of `nvcre_nccl_busbw_gbs`. Dual-registered for one minor release; the `_gbps` suffix incorrectly implied gigabits. |

The `nccl_test` label identifies the collective operation (e.g., `all_reduce`, `all_gather`, `alltoall`). The `message_size_bytes` label tracks results per message size tested.

NCCL bandwidth metrics are cleaned up when a BandwidthMeasurement is deleted.

**Cardinality at scale:** NCCL metrics include a `message_size_bytes` label (typically 20-30 values per test). With 3 test types and 10 concurrent measurements, expect ~600-900 NCCL time series from the canonical `_gbs` names. During the deprecation window the `_gbps` aliases are dual-registered, which doubles that count. Goodput metrics produce 10 series per measurement. At 100+ concurrent Jobs, monitor your Prometheus memory and consider increasing `sampleInterval` or limiting concurrent Certifications.

## Topology metrics

| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `nvcre_topology_validated_nodes` | Gauge | `namespace`, `workflow`, `topology_key`, `domain`, `node` | Set to `1` for each node that passed burn-in validation. Per-node granularity for identifying exactly which nodes are certified in each topology domain. |
| `nvcre_topology_failed_nodes` | Gauge | `namespace`, `workflow`, `topology_key`, `domain`, `node` | Set to `1` for each node that failed burn-in validation. Useful for identifying bad switches, racks, or NVLink cliques from Prometheus. |

## Example PromQL queries

With the chart default `honorLabels: true`, filters on `namespace` and `job` refer to the Certification Job as documented in the tables above. If you scrape without honoring metric labels, those label names refer to the scrape target instead.

### Certification and Workflow status

```promql
# Certifications that have failed
nvcre_certification_status{status="failed"} == 1

# Currently running Certifications
nvcre_certification_status{status="in_progress"} == 1

# Certification status breakdown per namespace
sum by (namespace, status) (nvcre_certification_status == 1)

# Failed Workflows, grouped by nvcre.nvidia.com/certification
nvcre_workflow_status{status="failed"} == 1

# Workflows still in progress for a Certification
nvcre_workflow_status{status="in_progress", certification="gpu-cluster-cert"} == 1
```

### Job status

```promql
# Count of failed jobs
count(nvcre_job_status{status="failed"} == 1) by (namespace)

# All currently running jobs
nvcre_job_status{status="in_progress"} == 1

# Job status breakdown per namespace
sum by (namespace, status) (nvcre_job_status == 1)
```

### Hardware failures

```promql
# Nodes with hardware failures across all jobs
count(nvcre_hardware_failures_detected_total) by (node)

# Hardware failure rate (per hour)
sum(rate(nvcre_hardware_failures_detected_total[1h])) by (namespace)

# Jobs with the most failed nodes
topk(10, nvcre_job_failed_nodes)
```

### Goodput

```promql
# Average goodput across all measurements in a namespace
avg(nvcre_goodput_ratio) by (namespace)

# Jobs with low goodput (below 80%)
nvcre_goodput_ratio < 0.8

# Training time breakdown for a specific job
{__name__=~"nvcre_goodput_(training|reschedule|resume|checkpoint_save|warmup|non_warmup)_time_seconds", job="my-training-job"}

# Average TFLOPS per GPU across measurements
avg(nvcre_goodput_avg_tflops_per_gpu) by (namespace)
```

### NCCL bandwidth

```promql
# Bus bandwidth for all_reduce across all message sizes
nvcre_nccl_busbw_gbs{nccl_test="all_reduce"}

# Average algorithmic bandwidth per test type
avg(nvcre_nccl_algbw_gbs) by (nccl_test)

# Compare bandwidth across message sizes for a specific measurement
nvcre_nccl_algbw_gbs{measurement="nccl-allreduce-bw"}
```

The `_gbps` names remain as deprecated aliases for one minor release. New dashboards should query `_gbs`.

### Reconciliation performance

```promql
# Reconciliation error rate
sum(rate(nvcre_reconcile_total{result="error"}[5m])) by (namespace)

# 99th percentile reconciliation duration
histogram_quantile(0.99, sum(rate(nvcre_reconcile_duration_seconds_bucket[5m])) by (le))

# Node health check duration (p95)
histogram_quantile(0.95, sum(rate(nvcre_node_health_check_duration_seconds_bucket[5m])) by (le))
```

### Topology validation

```promql
# Nodes validated per topology domain
nvcre_topology_validated_nodes{topology_key="nvidia.com/gpu.clique"}

# Total validated nodes per workflow
sum(nvcre_topology_validated_nodes) by (namespace, workflow)
```

## Example alerting rules

The following `PrometheusRule` examples provide starting points for monitoring a burn-in cluster.

```yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: nvcre-alerts
  labels:
    app.kubernetes.io/name: nvcre
spec:
  groups:
    - name: burnin.rules
      rules:
        # Alert when hardware failure rate exceeds threshold
        - alert: NVCREHighHardwareFailureRate
          expr: |
            sum(rate(nvcre_hardware_failures_detected_total[1h])) by (namespace) > 0.5
          for: 10m
          labels:
            severity: critical
          annotations:
            summary: "High hardware failure rate in namespace {{ $labels.namespace }}"
            description: >
              More than 0.5 hardware failures per second detected in
              namespace {{ $labels.namespace }} over the last hour.

        # Alert when goodput drops below acceptable threshold
        - alert: NVCRELowGoodput
          expr: |
            nvcre_goodput_ratio < 0.75
            and nvcre_goodput_ratio > 0
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "Low goodput for measurement {{ $labels.measurement }}"
            description: >
              Goodput ratio for {{ $labels.measurement }} in
              {{ $labels.namespace }} is {{ $value | humanize }},
              below the 75% threshold. Check for frequent checkpointing
              or rescheduling overhead.

        # Alert when a Certification has failed
        - alert: NVCRECertificationFailed
          expr: |
            nvcre_certification_status{status="failed"} == 1
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "NVCRE Certification {{ $labels.certification }} failed"
            description: >
              Certification {{ $labels.certification }} in namespace
              {{ $labels.namespace }} is Failed. Inspect category status
              and Workflow conditions for the failing domain/variant.

        # Alert when a job appears stuck (in_progress for too long)
        - alert: NVCREJobStuck
          expr: |
            nvcre_job_status{status="in_progress"} == 1
            unless on(namespace, job) (
              increase(nvcre_reconcile_total[30m]) > 0
            )
          for: 30m
          labels:
            severity: warning
          annotations:
            summary: "NVCRE job {{ $labels.job }} appears stuck"
            description: >
              Job {{ $labels.job }} in namespace {{ $labels.namespace }}
              has been in_progress for at least 30 minutes with no
              reconciliation activity.
```

## See also

- [Certification API](/cluster-readiness-engine/v0.6.0/api-reference/certification) — the resource that emits certification status metrics
- [Job API](/cluster-readiness-engine/v0.6.0/api-reference/job) — the resource that emits job status and hardware failure metrics
- [GoodputMeasurement API](/cluster-readiness-engine/v0.6.0/api-reference/goodput-measurement) and [BandwidthMeasurement API](/cluster-readiness-engine/v0.6.0/api-reference/bandwidth-measurement) — the resources that populate goodput and bandwidth metrics
- [Workflow API](/cluster-readiness-engine/v0.6.0/api-reference/workflow) — the resource that emits workflow status and topology validation metrics
- [Goodput & Bandwidth Measurement](/cluster-readiness-engine/v0.6.0/concepts/goodput-bandwidth-measurement) — conceptual overview of the goodput formula and its components