Operations

Monitoring

View as Markdown

Set up Prometheus metrics, structured log queries, and alert rules for the Cluster Readiness Engine (CRE) controller.

Prometheus integration

ServiceMonitor

The Helm chart installs a ServiceMonitor for the Prometheus Operator by default (metrics.serviceMonitor.enabled: true; set it to false on clusters without the Prometheus Operator CRDs). The shipped monitor scrapes the controller’s HTTPS metrics endpoint:

1apiVersion: monitoring.coreos.com/v1
2kind: ServiceMonitor
3metadata:
4  name: cluster-readiness-engine-metrics-monitor
5  namespace: cluster-readiness-engine
6spec:
7  endpoints:
8    - path: /metrics
9      port: https
10      scheme: https
11      bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
12      tlsConfig:
13        insecureSkipVerify: true  # Use cert-manager in production
14  selector:
15    matchLabels:
16      control-plane: manager

Verify it is installed:

$kubectl get servicemonitor -n cluster-readiness-engine

Key metrics

The table below highlights the most important metrics. See the Metrics Reference for the full list, labels, and example PromQL queries.

MetricTypeWhat it tells you
cre_job_statusGaugeCurrent state of each job (in_progress, succeeded, failed)
cre_job_failed_nodesGaugeNumber of nodes with hardware failures per job
cre_hardware_failures_detected_totalCounterCumulative hardware failure detections per node
cre_reconcile_duration_secondsHistogramHow long each reconcile loop takes
cre_reconcile_totalCounterReconcile attempts by result (success, error, requeue)
cre_goodput_ratioGaugeTraining efficiency from 0.0 to 1.0
cre_nccl_algbw_gbpsGaugeNCCL algorithmic bandwidth in GB/s per message size
cre_nccl_busbw_gbpsGaugeNCCL bus bandwidth in GB/s per message size
cre_topology_validated_nodesGaugeNodes that passed validation per topology domain

Structured logging

The controller emits structured zap logs. In the shipped manager configuration, zap development mode is enabled, so logs use console encoding unless you set --zap-encoder=json.

Key fields

FieldDescription
controllerWhich controller emitted the log (job, workflow, certification, goodputmeasurement)
namespaceKubernetes namespace of the resource
nameResource name
reconcileIDUnique ID for the reconcile pass — use this to trace a single reconciliation

Log levels

LevelFlagUse case
info (0)--zap-log-level=0Normal operations, status changes
debug (1)--zap-log-level=1Detailed troubleshooting

Enable debug logging by adding the flag to the manager container args in the Deployment spec:

1args:
2  - --zap-log-level=1

Example log queries

Using Loki or a similar log aggregation system (adjust the stream selector to how your agent labels the controller pods — the pods carry the control-plane=manager label in the cluster-readiness-engine namespace):

# Hardware failures
{namespace="cluster-readiness-engine"} |= "Hardware failure detected"
# Errors for a specific job
{namespace="cluster-readiness-engine"} |= "my-job" |= "error"
# All hardware failure detections
{namespace="cluster-readiness-engine"} |= "HardwareFailed"

If you switch the manager to JSON logs with --zap-encoder=json, you can filter on fields like controller, name, and reconcileID directly in your log backend.

Production GPU fleets require alerting across three dimensions: hardware failures (GPU faults, NVLink errors, ECC violations detected during workloads), performance regressions (bandwidth or goodput falling below expected thresholds), and operational health (controller reconciliation latency, error rates). The starter rules below cover all three.

1groups:
2  - name: cluster-readiness-engine
3    rules:
4      - alert: CREHardwareFailure
5        expr: cre_job_failed_nodes > 0
6        for: 1m
7        labels:
8          severity: warning
9        annotations:
10          summary: "Hardware failure in job {{ $labels.job }}"
11          description: "{{ $value }} node(s) failed in {{ $labels.namespace }}/{{ $labels.job }}."
12
13      - alert: CREJobStuck
14        expr: cre_job_status{status="in_progress"} == 1
15        for: 6h
16        labels:
17          severity: warning
18        annotations:
19          summary: "Job {{ $labels.job }} stuck in progress"
20          description: "Job has been in_progress for over 6 hours."
21
22      - alert: CRELowGoodput
23        expr: cre_goodput_ratio > 0 and cre_goodput_ratio < 0.5
24        for: 5m
25        labels:
26          severity: warning
27        annotations:
28          summary: "Low goodput for {{ $labels.measurement }}"
29          description: "Goodput ratio is {{ $value | humanizePercentage }}."
30
31      - alert: CREHighReconcileLatency
32        expr: |
33          histogram_quantile(0.95,
34            sum by (le) (rate(cre_reconcile_duration_seconds_bucket[5m]))
35          ) > 5
36        for: 10m
37        labels:
38          severity: warning
39        annotations:
40          summary: "Reconciliation P95 latency above 5s"
41
42      - alert: CREReconcileErrors
43        expr: |
44          sum(rate(cre_reconcile_total{result="error"}[5m]))
45          / sum(rate(cre_reconcile_total[5m])) > 0.1
46        for: 5m
47        labels:
48          severity: warning
49        annotations:
50          summary: "Reconciliation error rate above 10%"

Tune alert thresholds: the for durations and thresholds above are starting points. Adjust them based on your workload profiles — long-running multi-day training jobs will need a longer CREJobStuck threshold.

Next steps