Operations

Prometheus Metrics

View as Markdown

This page is the full reference for Prometheus metrics exposed by the NVIDIA Cluster Readiness Engine (NVCRE) controller. All metrics are registered with the controller-runtime metrics registry and served on the /metrics endpoint.

Scrape configuration

The Helm chart ships a ServiceMonitor for the Prometheus Operator (enabled by default via metrics.serviceMonitor.enabled):

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: nvcre-metrics-monitor
  namespace: nvcre
spec:
  endpoints:
    - path: /metrics
      port: https
      scheme: https
      bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      tlsConfig:
        insecureSkipVerify: true  # Use cert-manager in production
  selector:
    matchLabels:
      control-plane: manager

Job status metrics

MetricTypeLabelsDescription
nvcre_job_statusGaugenamespace, job, workflow, statusCurrent status of burn-in Jobs. Value is 1 for the current status, 0 for others. Status values: in_progress, succeeded, failed.

The gauge is set for all three status values on each update, ensuring that a transition from in_progress to succeeded also zeroes out the in_progress series. Metrics are cleaned up when a Job is deleted.

Hardware failure metrics

MetricTypeLabelsDescription
nvcre_hardware_failed_jobs_totalCounternamespace, job, workflowTotal number of Jobs that detected hardware failures. Incremented once per Job on first detection.
nvcre_job_failed_nodesGaugenamespace, job, workflowNumber of nodes with detected hardware failures per Job.
nvcre_hardware_failures_detected_totalCounternamespace, job, workflow, nodeTotal number of hardware failures detected across all evaluations. Per-node granularity.

Node health check metrics

MetricTypeLabelsDescription
nvcre_node_health_check_duration_secondsHistogramnamespace, job, workflowDuration of node health check operations in seconds. Buckets: 1ms to ~16s (exponential).
nvcre_nodes_evaluated_totalCounternamespace, job, workflowTotal number of node health evaluations performed.

Reconciliation metrics

MetricTypeLabelsDescription
nvcre_reconcile_totalCounternamespace, job, workflow, resultTotal number of reconciliation attempts. result values: success, error, requeue.
nvcre_reconcile_duration_secondsHistogramnamespace, job, workflowDuration of reconciliation operations in seconds. Buckets: 10ms to ~20s (exponential).

Workload metrics

MetricTypeLabelsDescription
nvcre_workload_created_totalCounternamespace, job, workflowTotal number of workloads created by the Job controller.

Goodput metrics

All goodput metrics share the same label set: namespace, measurement, job, workflow.

MetricTypeDescription
nvcre_goodput_ratioGaugeRuntime goodput ratio (0.0 to 1.0)
nvcre_goodput_avg_tflops_per_gpuGaugeAverage TFLOPS per GPU from goodput measurement
nvcre_goodput_training_time_secondsGaugeTotal training wall-clock time in seconds
nvcre_goodput_reschedule_time_secondsGaugeCumulative reschedule time in seconds
nvcre_goodput_resume_time_secondsGaugeCumulative resume time in seconds
nvcre_goodput_checkpoint_save_time_secondsGaugeCumulative checkpoint save time in seconds
nvcre_goodput_warmup_time_secondsGaugeTotal warmup step time in seconds
nvcre_goodput_non_warmup_time_secondsGaugeTotal non-warmup step time in seconds
nvcre_goodput_avg_step_time_secondsGaugeAverage time per training step in seconds
nvcre_goodput_lost_work_time_secondsGaugeCumulative lost work time in seconds (work done after last checkpoint, lost on restart)

Goodput metrics are cleaned up when a GoodputMeasurement is deleted.

Metric lifecycle

Goodput metrics are cleaned up at specific lifecycle events to prevent stale data in Prometheus:

EventWhat is cleaned upWhat is preserved
Job completes (succeeded or failed)Operational metrics (every nvcre_goodput_* gauge except the ratio: TFLOPS, step time, training/warmup/non-warmup time, checkpoint, reschedule, resume, lost work)nvcre_goodput_ratio (the outcome)
Job restarts from checkpointInstantaneous metrics (TFLOPS, avg step time)All cumulative metrics
GoodputMeasurement deletedAll goodput metrics for that measurement—
BandwidthMeasurement completes or is deletedAll NCCL bandwidth metrics for that measurement—

NCCL bandwidth metrics

All NCCL bandwidth metrics share the same label set: namespace, measurement, job, workflow, nccl_test, message_size_bytes.

MetricTypeDescription
nvcre_nccl_algbw_gbpsGaugeNCCL algorithmic bandwidth in GB/s per message size
nvcre_nccl_busbw_gbpsGaugeNCCL bus bandwidth in GB/s per message size

The nccl_test label identifies the collective operation (e.g., all_reduce, all_gather, alltoall). The message_size_bytes label tracks results per message size tested.

NCCL bandwidth metrics are cleaned up when a BandwidthMeasurement is deleted.

Cardinality at scale: NCCL metrics include a message_size_bytes label (typically 20-30 values per test). With 3 test types and 10 concurrent measurements, expect ~600-900 NCCL time series. Goodput metrics produce 10 series per measurement. At 100+ concurrent Jobs, monitor your Prometheus memory and consider increasing sampleInterval or limiting concurrent Certifications.

Topology metrics

MetricTypeLabelsDescription
nvcre_topology_validated_nodesGaugenamespace, workflow, topology_key, domain, nodeSet to 1 for each node that passed burn-in validation. Per-node granularity for identifying exactly which nodes are certified in each topology domain.
nvcre_topology_failed_nodesGaugenamespace, workflow, topology_key, domain, nodeSet to 1 for each node that failed burn-in validation. Useful for identifying bad switches, racks, or NVLink cliques from Prometheus.

Example PromQL queries

Job status

# Count of failed jobs
count(nvcre_job_status{status="failed"} == 1) by (namespace)
# All currently running jobs
nvcre_job_status{status="in_progress"} == 1
# Job status breakdown per namespace
sum by (namespace, status) (nvcre_job_status == 1)

Hardware failures

# Nodes with hardware failures across all jobs
count(nvcre_hardware_failures_detected_total) by (node)
# Hardware failure rate (per hour)
sum(rate(nvcre_hardware_failures_detected_total[1h])) by (namespace)
# Jobs with the most failed nodes
topk(10, nvcre_job_failed_nodes)

Goodput

# Average goodput across all measurements in a namespace
avg(nvcre_goodput_ratio) by (namespace)
# Jobs with low goodput (below 80%)
nvcre_goodput_ratio < 0.8
# Training time breakdown for a specific job
{__name__=~"nvcre_goodput_(training|reschedule|resume|checkpoint_save|warmup|non_warmup)_time_seconds", job="my-training-job"}
# Average TFLOPS per GPU across measurements
avg(nvcre_goodput_avg_tflops_per_gpu) by (namespace)

NCCL bandwidth

# Bus bandwidth for all_reduce across all message sizes
nvcre_nccl_busbw_gbps{nccl_test="all_reduce"}
# Average algorithmic bandwidth per test type
avg(nvcre_nccl_algbw_gbps) by (nccl_test)
# Compare bandwidth across message sizes for a specific measurement
nvcre_nccl_algbw_gbps{measurement="nccl-allreduce-bw"}

Reconciliation performance

# Reconciliation error rate
sum(rate(nvcre_reconcile_total{result="error"}[5m])) by (namespace)
# 99th percentile reconciliation duration
histogram_quantile(0.99, sum(rate(nvcre_reconcile_duration_seconds_bucket[5m])) by (le))
# Node health check duration (p95)
histogram_quantile(0.95, sum(rate(nvcre_node_health_check_duration_seconds_bucket[5m])) by (le))

Topology validation

# Nodes validated per topology domain
nvcre_topology_validated_nodes{topology_key="nvidia.com/gpu.clique"}
# Total validated nodes per workflow
sum(nvcre_topology_validated_nodes) by (namespace, workflow)

Example alerting rules

The following PrometheusRule examples provide starting points for monitoring a burn-in cluster.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: nvcre-alerts
  labels:
    app.kubernetes.io/name: nvcre
spec:
  groups:
    - name: burnin.rules
      rules:
        # Alert when hardware failure rate exceeds threshold
        - alert: NVCREHighHardwareFailureRate
          expr: |
            sum(rate(nvcre_hardware_failures_detected_total[1h])) by (namespace) > 0.5
          for: 10m
          labels:
            severity: critical
          annotations:
            summary: "High hardware failure rate in namespace {{ $labels.namespace }}"
            description: >
              More than 0.5 hardware failures per second detected in
              namespace {{ $labels.namespace }} over the last hour.
        # Alert when goodput drops below acceptable threshold
        - alert: NVCRELowGoodput
          expr: |
            nvcre_goodput_ratio < 0.75
            and nvcre_goodput_ratio > 0
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "Low goodput for measurement {{ $labels.measurement }}"
            description: >
              Goodput ratio for {{ $labels.measurement }} in
              {{ $labels.namespace }} is {{ $value | humanize }},
              below the 75% threshold. Check for frequent checkpointing
              or rescheduling overhead.
        # Alert when a job appears stuck (in_progress for too long)
        - alert: NVCREJobStuck
          expr: |
            nvcre_job_status{status="in_progress"} == 1
            unless on(namespace, job) (
              increase(nvcre_reconcile_total[30m]) > 0
            )
          for: 30m
          labels:
            severity: warning
          annotations:
            summary: "NVCRE job {{ $labels.job }} appears stuck"
            description: >
              Job {{ $labels.job }} in namespace {{ $labels.namespace }}
              has been in_progress for at least 30 minutes with no
              reconciliation activity.

See also