Prometheus Metrics

This page is the full reference for Prometheus metrics exposed by the NVIDIA Cluster Readiness Engine (NVCRE) controller. All metrics are registered with the controller-runtime metrics registry and served on the /metrics endpoint.

Scrape configuration

The Helm chart ships a ServiceMonitor for the Prometheus Operator (enabled by default via metrics.serviceMonitor.enabled):

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nvcre-metrics-monitor
  namespace: nvcre
  labels:
    release: prometheus
spec:
  endpoints:
    - path: /metrics
      port: https
      scheme: https
      honorLabels: true
      bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      tlsConfig:
        insecureSkipVerify: true  # Use cert-manager in production
  selector:
    matchLabels:
      control-plane: manager

honorLabels: true (chart default metrics.serviceMonitor.honorLabels) keeps the controller’s own namespace and job labels on each series. Without it, Prometheus prefers scrape-target labels and renames the metric labels to exported_namespace / exported_job, so the PromQL examples below would match the controller Service rather than the Certification Job. Clusters that already built dashboards on exported_* can set metrics.serviceMonitor.honorLabels=false.

Tune scrape cadence with metrics.serviceMonitor.interval and metrics.serviceMonitor.scrapeTimeout (empty uses the Prometheus defaults). kube-prometheus-stack users should set metrics.serviceMonitor.labels.release to their Prometheus Helm release name (default prometheus). See helm/cluster-readiness-engine/values.yaml for the full set of knobs.

Lifecycle status metrics

Certification, Workflow and Job conditions are the operator-facing source of truth (kubectl get certifications,workflows,jobs.nvcre.nvidia.com). These gauges export them so a status panel or alert can show the verdict of a run.

These series are built at scrape time from the elected leader’s informer cache. Each scrape lists Certifications, Workflows and Jobs and emits 1 for the true InProgress / Succeeded / Failed condition and 0 for the peers, so a status written by any reconciler, including a Workflow’s timeoutPerJob write on its Job, shows on the next scrape, and the series are correct again after a controller restart. If more than one phase condition is True, failed wins, then succeeded, then in_progress. An object with no phase condition True is omitted. An object being deleted keeps its series until it leaves the cache, and they disappear on the next scrape after that; a changed parent label reports the new value and does not leave old series. Standby replicas emit nothing. Reason strings are not exported as labels.

Hardware failure metrics

Node health check metrics

Reconciliation metrics

Workload metrics

Goodput metrics

All goodput metrics share the same label set: namespace, measurement, job, workflow.

Goodput metrics are cleaned up when a GoodputMeasurement is deleted.

Metric lifecycle

Goodput metrics are cleaned up at specific lifecycle events to prevent stale data in Prometheus:

NCCL bandwidth metrics

All NCCL bandwidth metrics share the same label set: namespace, measurement, job, workflow, nccl_test, message_size_bytes. Values are gigabytes per second (the nccl-tests algbw / busbw columns), not gigabits.

The nccl_test label identifies the collective operation (e.g., all_reduce, all_gather, alltoall). The message_size_bytes label tracks results per message size tested.

NCCL bandwidth metrics are cleaned up when a BandwidthMeasurement is deleted.

Cardinality at scale: NCCL metrics include a message_size_bytes label (typically 20-30 values per test). With 3 test types and 10 concurrent measurements, expect ~600-900 NCCL time series from the canonical _gbs names. During the deprecation window the _gbps aliases are dual-registered, which doubles that count. Goodput metrics produce 10 series per measurement. At 100+ concurrent Jobs, monitor your Prometheus memory and consider increasing sampleInterval or limiting concurrent Certifications.

Topology metrics

Example PromQL queries

With the chart default honorLabels: true, filters on namespace and job refer to the Certification Job as documented in the tables above. If you scrape without honoring metric labels, those label names refer to the scrape target instead.

Certification and Workflow status

# Certifications that have failed
nvcre_certification_status{status="failed"} == 1
# Currently running Certifications
nvcre_certification_status{status="in_progress"} == 1
# Certification status breakdown per namespace
sum by (namespace, status) (nvcre_certification_status == 1)
# Failed Workflows, grouped by nvcre.nvidia.com/certification
nvcre_workflow_status{status="failed"} == 1
# Workflows still in progress for a Certification
nvcre_workflow_status{status="in_progress", certification="gpu-cluster-cert"} == 1

Job status

# Count of failed jobs
count(nvcre_job_status{status="failed"} == 1) by (namespace)
# All currently running jobs
nvcre_job_status{status="in_progress"} == 1
# Job status breakdown per namespace
sum by (namespace, status) (nvcre_job_status == 1)

Hardware failures

# Nodes with hardware failures across all jobs
count(nvcre_hardware_failures_detected_total) by (node)
# Hardware failure rate (per hour)
sum(rate(nvcre_hardware_failures_detected_total[1h])) by (namespace)
# Jobs with the most failed nodes
topk(10, nvcre_job_failed_nodes)

Goodput

# Average goodput across all measurements in a namespace
avg(nvcre_goodput_ratio) by (namespace)
# Jobs with low goodput (below 80%)
nvcre_goodput_ratio < 0.8
# Training time breakdown for a specific job
{__name__=~"nvcre_goodput_(training|reschedule|resume|checkpoint_save|warmup|non_warmup)_time_seconds", job="my-training-job"}
# Average TFLOPS per GPU across measurements
avg(nvcre_goodput_avg_tflops_per_gpu) by (namespace)

NCCL bandwidth

# Bus bandwidth for all_reduce across all message sizes
nvcre_nccl_busbw_gbs{nccl_test="all_reduce"}
# Average algorithmic bandwidth per test type
avg(nvcre_nccl_algbw_gbs) by (nccl_test)
# Compare bandwidth across message sizes for a specific measurement
nvcre_nccl_algbw_gbs{measurement="nccl-allreduce-bw"}

The _gbps names remain as deprecated aliases for one minor release. New dashboards should query _gbs.

Reconciliation performance

# Reconciliation error rate
sum(rate(nvcre_reconcile_total{result="error"}[5m])) by (namespace)
# 99th percentile reconciliation duration
histogram_quantile(0.99, sum(rate(nvcre_reconcile_duration_seconds_bucket[5m])) by (le))
# Node health check duration (p95)
histogram_quantile(0.95, sum(rate(nvcre_node_health_check_duration_seconds_bucket[5m])) by (le))

Topology validation

# Nodes validated per topology domain
nvcre_topology_validated_nodes{topology_key="nvidia.com/gpu.clique"}
# Total validated nodes per workflow
sum(nvcre_topology_validated_nodes) by (namespace, workflow)

Example alerting rules

The following PrometheusRule examples provide starting points for monitoring a burn-in cluster.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: nvcre-alerts
  labels:
    app.kubernetes.io/name: nvcre
spec:
  groups:
    - name: burnin.rules
      rules:
        # Alert when hardware failure rate exceeds threshold
        - alert: NVCREHighHardwareFailureRate
          expr: |
            sum(rate(nvcre_hardware_failures_detected_total[1h])) by (namespace) > 0.5
          for: 10m
          labels:
            severity: critical
          annotations:
            summary: "High hardware failure rate in namespace {{ $labels.namespace }}"
            description: >
              More than 0.5 hardware failures per second detected in
              namespace {{ $labels.namespace }} over the last hour.
        # Alert when goodput drops below acceptable threshold
        - alert: NVCRELowGoodput
          expr: |
            nvcre_goodput_ratio < 0.75
            and nvcre_goodput_ratio > 0
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "Low goodput for measurement {{ $labels.measurement }}"
            description: >
              Goodput ratio for {{ $labels.measurement }} in
              {{ $labels.namespace }} is {{ $value | humanize }},
              below the 75% threshold. Check for frequent checkpointing
              or rescheduling overhead.
        # Alert when a Certification has failed
        - alert: NVCRECertificationFailed
          expr: |
            nvcre_certification_status{status="failed"} == 1
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "NVCRE Certification {{ $labels.certification }} failed"
            description: >
              Certification {{ $labels.certification }} in namespace
              {{ $labels.namespace }} is Failed. Inspect category status
              and Workflow conditions for the failing domain/variant.
        # Alert when a job appears stuck (in_progress for too long)
        - alert: NVCREJobStuck
          expr: |
            nvcre_job_status{status="in_progress"} == 1
            unless on(namespace, job) (
              increase(nvcre_reconcile_total[30m]) > 0
            )
          for: 30m
          labels:
            severity: warning
          annotations:
            summary: "NVCRE job {{ $labels.job }} appears stuck"
            description: >
              Job {{ $labels.job }} in namespace {{ $labels.namespace }}
              has been in_progress for at least 30 minutes with no
              reconciliation activity.

See also

• Certification API — the resource that emits certification status metrics
• Job API — the resource that emits job status and hardware failure metrics
• GoodputMeasurement API and BandwidthMeasurement API — the resources that populate goodput and bandwidth metrics
• Workflow API — the resource that emits workflow status and topology validation metrics
• Goodput & Bandwidth Measurement — conceptual overview of the goodput formula and its components