NVCRE Certification Monitor Configuration

Overview

The NVCRE Certification Monitor reads NVIDIA Cluster Readiness Engine (NVCRE) Certification custom resources and publishes one health event per failed (node, variant, reason). This document covers every Helm configuration option, the state the monitor writes to the cluster, and how to observe and troubleshoot it.

Prerequisites

NVCRE

The NVIDIA Cluster Readiness Engine must be installed in the cluster: its controller, the nvcre.nvidia.com/v1alpha1 Certification CRD and the workload dependencies it brings in. Follow the NVCRE installation guide.

The monitor tolerates NVCRE being installed later. Without the CRD, every sweep logs Failed to list Certification CRs with a no matches for kind "Certification" error and returns; the pod stays Running and passes its readiness probe. Once NVCRE is installed the next sweep proceeds normally with no restart.

Cross-node identity

The monitor reports on nodes other than the one it runs on, so Platform Connectors only accept its events from an allowlisted, token-authenticated identity. Enabling the module adds its ServiceAccount to the derived allowlist automatically; no entry in global.platformConnectorAuth.crossNodeServiceAccounts is needed. Run the pod on a system or control-plane node pool rather than on GPU nodes serving tenants, for the reasons given in Platform Connectors.

Configuration Reference

Module Enable/Disable

Controls whether the nvcre-certification-monitor module is deployed. It ships disabled.

global:
  nvcreCertificationMonitor:
    enabled: true

Resources

nvcre-certification-monitor:
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 500m
      memory: 256Mi

Memory is dominated by the Node informer. The monitor strips every cached Node down to its name, UID, labels and the single annotation it reads, so the cache stays small even on large clusters.

Logging

nvcre-certification-monitor:
  logLevel: info  # Options: debug, info, warn, error

Resync Interval

nvcre-certification-monitor:
  resyncInterval: 15m

How often a full sweep runs. The first sweep runs immediately at startup, then every interval. There is no event-driven trigger, so a Certification reaching a terminal state is noticed within one interval. Shorter values give faster detection at the cost of more API reads; each sweep lists all Certification CRs from the informer cache and fetches the result ConfigMaps of terminal categories directly from the API server.

Processing Strategy

nvcre-certification-monitor:
  processingStrategy: EXECUTE_REMEDIATION  # or STORE_ONLY

EXECUTE_REMEDIATION is the normal mode: downstream modules set Node Conditions and apply taints. STORE_ONLY records and exports the events but keeps them out of the remediation pipeline (no Node Condition, no cordon), which is useful when first rolling the monitor out. The monitor’s own bookkeeping does not change with the strategy: in both modes it still writes its nvcre-cert-failures-details annotation and nvcre-cert-failure label on Nodes and the cert-processed and error-recovered annotations on Certifications, because that state is what keeps it from republishing the same failure every sweep.

Image and Scheduling

nvcre-certification-monitor:
  image:
    repository: ghcr.io/nvidia/nvsentinel/nvcre-certification-monitor
    pullPolicy: IfNotPresent
    tag: ""            # defaults to the chart appVersion
  priorityClassName: "" # overridden by the matching global when set
  podAnnotations: {}

The Deployment always runs one replica and the chart exposes no replicaCount. The sweep has no leader election, so a second pod would race the first on the node and cert annotations and publish duplicate health events.

Policies

Policies decide which failed-node rows become health events. A row that matches no policy produces no event, no annotation entry and no Node Condition.

Structure

nvcre-certification-monitor:
  policies:
    - name: certification-failures
      match: "(failedNode.reason == 'ThresholdViolation') || (failedNode.reason == 'WorkloadFailed')"

Each policy has a name used in logs and a match CEL expression. The expression is evaluated against every row of every Failed category of every terminal Certification. If any policy returns true, the row enters the desired set. The list is rendered into the monitor’s config file; a multi-line match is supported.

CEL Variables

All fields are strings.

Failure Reasons

HardwareFailureDetected is excluded on purpose. The node is already quarantined by whichever monitor found the underlying fault, and a second health event for the same node creates a lifecycle the monitor cannot recover on its own. Include it only if you accept a second hold that outlives the original quarantine.

Examples

Publish every failure reason:

policies:
  - name: all-failures
    match: "true"

Publish only NCCL categories, any reason except hardware pass-through:

policies:
  - name: nccl-only
    match: |
      category.domain == 'communication' &&
      failedNode.reason != 'HardwareFailureDetected'

Publish threshold violations for one specific test:

policies:
  - name: all-gather-threshold
    match: "category.variant == 'nccl-all-gather' && failedNode.reason == 'ThresholdViolation'"

Health Event Fields

The errorCode carries the variant because Platform Connectors collapse all of a node’s certification failures into the single NVCRECertFailed condition. The errorCode is what keeps two failing categories on one node as two separately clearable messages and two separate taint holds.

Fault Quarantine Ruleset

The Fault Quarantine chart ships a ruleset for this monitor. It matches fatal events from agent == 'nvcre-certification-monitor' with checkName == 'NVCRECertFailed', skips nodes opted out of NVSentinel management, and applies a taint without cordoning:

taint:
  key: "nvsentinel.dgxc.nvidia.com/nvcre-cert-failed"
  value: "true"
  effect: "NoSchedule"
cordon:
  shouldCordon: false

No cordon means a certification rerun can still be scheduled onto the failed node with a matching toleration, while ordinary workloads are kept off. To change the taint or add a cordon, edit the NVCRE certification monitor failure ruleset entry under fault-quarantine.ruleSets in your values file.

Cluster State Written by the Monitor

The node annotation is the monitor’s only record of what it is enforcing. The two Certification annotations let it tell a brand-new failure apart from one an operator already released, without consulting the health-events store.

Observing

Nodes currently held by a certification failure:

kubectl get nodes -l nvsentinel.dgxc.nvidia.com/nvcre-cert-failure=true

Which tuples hold a given node:

kubectl get node <node> -o jsonpath='{.metadata.annotations.nvsentinel\.dgxc\.nvidia\.com/nvcre-cert-failures-details}'

The Node Condition and taint set downstream:

kubectl get node <node> -o jsonpath='{.status.conditions[?(@.type=="NVCRECertFailed")]}'
kubectl get node <node> -o jsonpath='{.spec.taints}'

What the monitor has recorded on a Certification:

kubectl get certification -n <namespace> <name> \
  -o jsonpath='{.metadata.annotations}'

Metrics

The monitor exports Prometheus metrics on the metrics container port (global.metricsPort, default 2112) at /metrics.

A non-zero malformed_node_annotations, a rising sweep_errors_total or a rising publish_errors_total means the monitor cannot record or deliver failures and needs attention.

Recovery and Operator Actions

Deleting a Node object is never required for recovery. If a node is deleted while held, the monitor publishes healthy for its tuples and records them in error-recovered. If a node named in a not-yet-processed Certification does not exist, the monitor skips it with a warning and re-evaluates on every sweep.

RBAC Permissions

The chart creates a ClusterRole with exactly these rules:

- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list", "watch", "patch"]             # cert-failures annotation and label
- apiGroups: ["nvcre.nvidia.com"]
  resources: ["certifications"]
  verbs: ["get", "list", "watch", "patch"]             # cert-processed / error-recovered annotations
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get"]                                       # per-category result ConfigMaps, read uncached

ConfigMaps are fetched directly from the API server rather than through an informer, which is why only get is granted.

Troubleshooting