Preflight configuration

Preflight is a mutating admission webhook that injects GPU diagnostic init containers (DCGM diagnostics, optional NCCL loopback / all-reduce) into pods that request GPUs in namespaces you opt in via labels. It answers “is this GPU healthy enough to start?” before your workload runs—separate from continuous GPU health monitoring.

Prerequisites

• Helm subchart is off by default; enable with global.preflight.enabled (see below).
• cert-manager (or OpenShift service CA) for webhook TLS—same expectation as the rest of the NVSentinel chart.
• DCGM reachable from injected init containers (typically the NVIDIA GPU Operator’s DCGM / hostengine service). Configure the endpoint via DCGM_HOSTENGINE_ADDR on the preflight-dcgm-diag init container.
• Multi-node / gang checks (e.g. preflight-nccl-allreduce): enable gang coordination and configure gang discovery for your scheduler (see below). The default gang discovery uses the native Kubernetes gang APIs, which need Kubernetes 1.35 or later. On an earlier cluster, configure a PodGroup-based scheduler or turn gang coordination off — see Single-node and single-GPU clusters.

Enable preflight

1. Set the global flag:

global:
  preflight:
    enabled: true

2. Configure the preflight subchart under the top-level preflight: key (values merge into distros/kubernetes/nvsentinel/charts/preflight/values.yaml). At minimum, review initContainers (including DCGM and NCCL env vars) and webhook.failurePolicy.

3. Label namespaces where injection should apply:

kubectl label namespace {namespace} nvsentinel.nvidia.com/preflight=enabled

The chart default namespaceSelector matches that label.

Single-node and single-GPU clusters

The chart enables gang coordination by default, and the default gangDiscovery ({}) selects native Kubernetes gang discovery. Preflight validates that discoverer against the cluster at startup, so on a cluster older than Kubernetes 1.35 the preflight pod does not start. Its log names the requirement:

kubectl -n nvsentinel logs deploy/preflight

"level":"ERROR","msg":"Fatal error","error":"failed to create gang discoverer resolver: failed to create default gang discoverer: kubernetes native gang API not available (requires K8s 1.35 Workload or K8s 1.36+ PodGroup)"

A single node has no gang to coordinate. Turn coordination off and inject only the single-node GPU check:

global:
  preflight:
    enabled: true
preflight:
  gangCoordination:
    enabled: false
  initContainers:
    - name: preflight-dcgm-diag
      image:
        repository: ghcr.io/nvidia/nvsentinel/preflight-dcgm-diag
        tag: ""
      env:
        - name: DCGM_HOSTENGINE_ADDR
          value: "nvidia-dcgm.gpu-operator.svc:5555"
        - name: DCGM_DIAG_LEVEL
          value: "2"
      volumeMounts:
        - name: nvsentinel-socket
          mountPath: /var/run

Helm replaces the initContainers list rather than merging it, so this example also drops the two NCCL checks:

• preflight-nccl-allreduce needs gang coordination. With coordination off, the webhook injects no gang environment, the check exits 2 (NCCL_GANG_CONFIG_ERROR), and the pod stays in Init:Error.
• preflight-nccl-loopback measures the GPU-to-GPU path on the node, which a single-GPU node does not have. Keep it on a multi-GPU node and set BW_THRESHOLD_GBPS for the interconnect — approximately 15 for PCIe, 150 for NVLink.

The same values apply to a multi-node cluster whose scheduler is not gang-aware. If your scheduler does use PodGroups (Volcano, Run:ai / OSMO), keep gang coordination on and configure gang discovery instead — that path works on any Kubernetes version that runs the scheduler.

Image cache

Runs a DaemonSet that holds every configured check image on matching nodes, so the kubelet treats the images as in use and its garbage collector does not evict them.

preflight:
  imageCache:
    enabled: false
    resources:
      requests:
        cpu: 10m
        memory: 16Mi
      limits:
        cpu: 50m
        memory: 64Mi
    podAnnotations: {}

Off by default. Enable it where image pulls are slow or metered: preflight runs on the pod admission path, so a cold pull of a multi-gigabyte CUDA check image delays every GPU workload start on that node. The cost is one small pod per node, plus the disk the cached images occupy.

Init container placement

By default the webhook appends preflight init containers after any existing init containers in the pod spec. This ensures provider-injected setup containers (e.g., GCP TCPXO daemon) complete before preflight checks run.

Set initContainerPlacement to change this behavior:

# "append" (default): add after existing init containers
# "prepend": add before existing init containers
initContainerPlacement: "prepend"

Use prepend when preflight checks must run before other init containers — for example, to gate workload setup on GPU health validation.

Per-pod check selection

By default, all init containers with defaultEnabled: true (or omitted, which defaults to true) are injected into every GPU pod. To select a subset of checks for a specific pod, annotate it:

metadata:
  annotations:
    nvsentinel.nvidia.com/preflight-checks: "preflight-dcgm-diag,preflight-nccl-loopback"

Only the named containers are injected, in the order they appear in the annotation. Duplicate or unknown container names reject admission with an error.

An empty value disables all checks:

nvsentinel.nvidia.com/preflight-checks: ""

When the annotation is absent, defaultEnabled on each init container controls whether it runs. For gang-aware checks (nccl-allreduce), all pods in the gang must have the same annotation value — mismatches are detected and fail fast before torchrun launches.

See ADR-034 for design details.

Init containers (check configuration)

The initContainers list in the preflight chart defines which checks the webhook injects. Each entry is a standard corev1.Container plus preflight-specific controls such as defaultEnabled, inheritUserEnv, and inheritUserVolumeMounts — you control images, env vars, resource limits, security contexts, and volume mounts.

The webhook automatically injects these env vars into every init container (you do not need to set them):

For gang-aware containers the webhook also injects GANG_ID, GANG_CONFIG_DIR, GANG_TIMEOUT_SECONDS, and POD_NAME.

inheritUserEnv and inheritUserVolumeMounts both default to true when omitted, and the chart sets both to true on each built-in check. Every check therefore mirrors the workload’s NCCL and fabric configuration by default — see Fabric-specific NCCL configuration for the patterns that decide what is copied. Set either flag to false on an initContainers entry to give that check a curated environment instead.

preflight-dcgm-diag

Runs DCGM diagnostics against every GPU allocated to the pod via the remote hostengine.

Example values override:

initContainers:
  - name: preflight-dcgm-diag
    image:
      repository: ghcr.io/nvidia/nvsentinel/preflight-dcgm-diag
      tag: ""
    env:
      - name: DCGM_HOSTENGINE_ADDR
        value: "nvidia-dcgm-dra.gpu-operator.svc:5555,nvidia-dcgm.gpu-operator.svc:5555"
      - name: DCGM_DIAG_LEVEL
        value: "2"
      - name: DCGM_DIAG_STATUS_RETRY_MAX_ATTEMPTS
        value: "10"
      - name: DCGM_DIAG_STATUS_RETRY_INTERVAL_SECONDS
        value: "10"
    volumeMounts:
      - name: nvsentinel-socket
        mountPath: /var/run

If a DCGM_ST_* status still prevents the diagnostic from completing after retries, preflight-dcgm-diag emits a non-fatal unhealthy HealthEvent with RecommendedAction=NONE and exits successfully so the workload is not blocked by a preflight infrastructure failure.

preflight-nccl-loopback

Single-node NCCL all-reduce across all GPUs on the node. Validates intra-node interconnect (NVLink or PCIe).

Example values override:

initContainers:
  - name: preflight-nccl-loopback
    image: ghcr.io/nvidia/nvsentinel/preflight-nccl-loopback:latest
    env:
      - name: BW_THRESHOLD_GBPS
        value: "15"           # PCIe interconnect
      - name: TEST_SIZE_MB
        value: "512"

preflight-nccl-allreduce

Multi-node NCCL all-reduce across the entire gang. Requires gangCoordination.enabled: true and a gang-aware scheduler.

The container also requires IPC_LOCK capability for RDMA memory registration:

initContainers:
  - name: preflight-nccl-allreduce
    image: ghcr.io/nvidia/nvsentinel/preflight-nccl-allreduce:latest
    securityContext:
      capabilities:
        add: ["IPC_LOCK"]
    env:
      - name: BW_THRESHOLD_GBPS
        value: "100"
      - name: MESSAGE_SIZES
        value: "4G"

Fabric-specific NCCL configuration

When a check has inheritUserEnv or inheritUserVolumeMounts enabled (the default for both), the webhook copies matching NCCL env vars and volume mounts from the pod’s main containers using glob patterns:

ncclEnvPatterns:    ["NCCL_*", "FI_*", "LD_LIBRARY_PATH", "UCX_*", "TORCH_NCCL_*", "CUDA_DEVICE_ORDER"]
volumeMountPatterns: ["host-opt-amazon*", "nvtcpxo-*", "nccl-*", "dev-shm"]

This means if your training container already has the correct NCCL_TOPO_FILE, FI_PROVIDER, or LD_LIBRARY_PATH, a preflight init container inherits them with no manual configuration.

Inheritance is per init container and is on by default. Chart-defined env wins on a name conflict, so an inherited value only fills a name the check does not set itself. Set the flags to false on a check whose environment must not depend on workload NCCL tuning:

initContainers:
  - name: preflight-nccl-allreduce
    inheritUserEnv: false
    inheritUserVolumeMounts: false

For standalone testing (e.g. busybox main container), use ncclAllreduceExtraEnv and gangCoordination.extraHostPathMounts to provide fabric config explicitly.

Gang discovery

Gang discovery identifies pods that belong to the same scheduling group so multi-node preflight checks (NCCL all-reduce) know their peers. A pod carries a “gang anchor”—a reference to a parent object—that holds gang metadata such as the minimum member count.

Two discovery mechanisms are supported:

Native Kubernetes: schedulingGroup / workloadRef

The default when gangDiscovery is left empty ({}). Preflight first uses the Kubernetes 1.36 native PodGroup API when available, then falls back to the Kubernetes 1.35 native Workload API. Neither exists before Kubernetes 1.35, and preflight validates the discoverer at startup, so on an earlier cluster the pod fails to start — see Single-node and single-GPU clusters.

The PodGroup resource (scheduling.k8s.io/v1alpha2) and spec.schedulingGroup are alpha in Kubernetes 1.36 and disabled by default. Enable the GenericWorkload feature gate on the API server and scheduler to use this path.

In Kubernetes 1.36, each pod links to a PodGroup resource via spec.schedulingGroup:

spec:
  schedulingGroup:
    podGroupName: training-workers

The PodGroup object contains a gang policy with minCount:

apiVersion: scheduling.k8s.io/v1alpha2
kind: PodGroup
metadata:
  name: training-workers
spec:
  schedulingPolicy:
    gang:
      minCount: 2

No gangDiscovery configuration is needed for this path.

In Kubernetes 1.35, each pod links to a native Workload resource via spec.workloadRef:

spec:
  workloadRef:
    name: training-job-workload
    podGroup: workers

The Workload object contains pod groups and gang policy:

apiVersion: scheduling.k8s.io/v1alpha1
kind: Workload
metadata:
  name: training-job-workload
spec:
  podGroups:
    - name: workers
      policy:
        gang:
          minCount: 2

No gangDiscovery configuration is needed for this fallback path either.

The default chart RBAC grants read access to both native resources: scheduling.k8s.io/podgroups for Kubernetes 1.36 and scheduling.k8s.io/workloads for Kubernetes 1.35.

PodGroup-based schedulers (Volcano, Run:ai / OSMO, and similar)

For schedulers that use PodGroup CRDs, configure gangDiscovery with:

Volcano example:

gangDiscovery:
  name: "volcano"
  annotationKeys:
    - "scheduling.k8s.io/group-name"
  podGroupGVR:
    group: "scheduling.volcano.sh"
    version: "v1beta1"
    resource: "podgroups"
  minCountExpr: "podGroup.spec.minMember"

Volcano sets the scheduling.k8s.io/group-name annotation on each pod. The discoverer reads that annotation, fetches the corresponding PodGroup CRD, and evaluates minCountExpr to determine expected gang size.

OSMO + Kai scheduler example:

gangDiscovery:
  name: "osmo-with-kai"
  labelKeys:
    - "osmo.group_uuid"
  podGroupGVR:
    group: "scheduling.run.ai"
    version: "v2alpha2"
    resource: "podgroups"
  minCountExpr: "podGroup.spec.minMember"

Here membership is determined by a pod label instead of an annotation. The rest of the flow is the same: look up the PodGroup CRD and extract minCount via CEL.

OSMO + KAI Scheduler

KAI Scheduler uses the scheduling.run.ai/v2alpha2 PodGroup CRD. KAI Scheduler can run on its own, but in that setup the gang anchor is applied after admission, so preflight has no label or annotation to hook on when the webhook runs. This section documents the OSMO + KAI integration instead: OSMO creates the PodGroup and labels each pod with osmo.group_uuid at admission time. Preflight reads that label, fetches the PodGroup, and uses spec.minMember as the expected gang size for preflight-nccl-allreduce.

Step 1 — Enable preflight cluster-wide

global:
  preflight:
    enabled: true
preflight:
  gangDiscovery:
    name: "osmo-with-kai"
    labelKeys:
      - "osmo.group_uuid"
    podGroupGVR:
      group: "scheduling.run.ai"
      version: "v2alpha2"
      resource: "podgroups"
    minCountExpr: "podGroup.spec.minMember"

The chart’s built-in RBAC contributor role grants read access to scheduling.run.ai/podgroups when gangDiscovery.podGroupGVR is set in Helm values (see RBAC (aggregated ClusterRole) below).

Step 2 — Label namespaces for injection

kubectl label namespace {training-namespace} nvsentinel.nvidia.com/preflight=enabled

Step 3 — Verify image pull secrets

If preflight check images are in a private registry, set injectedImagePullSecrets in the preflight Helm values ([charts/preflight/values.yaml](../../distros/kubernetes/nvsentinel/charts/preflight/values.yaml)). The webhook injects those secrets into admitted pods as spec.imagePullSecrets.

Check whether any secrets are configured:

kubectl -n nvsentinel get configmap preflight -o jsonpath='{.data.config\.yaml}' | yq .imagePullSecrets

If imagePullSecrets is present, confirm each secret exists in your workload namespace:

kubectl -n {training-namespace} get secret {secret-name}

If injectedImagePullSecrets is empty ([]), no pull secrets are injected and this step can be skipped.

Step 4 — Submit a workload

Submit your OSMO gang-scheduled workload in the labeled namespace.

Step 5 — Verify injection

After a GPU pod is admitted, confirm the webhook injected preflight init containers and created the gang ConfigMap:

# Init containers injected (preflight-dcgm-diag, preflight-nccl-loopback, preflight-nccl-allreduce)
kubectl -n {training-namespace} get pod {pod-name} -o jsonpath='{range .spec.initContainers[*]}{.name}{": "}{.image}{"\n"}{end}'
# Gang ConfigMap created at admission
kubectl -n {training-namespace} get configmap -l nvsentinel.nvidia.com/managed-by=preflight -o yaml

The gang ID is {gangDiscovery.name}-{namespace}-{podGroupName}. For example, with name: osmo-with-kai in namespace team-a and PodGroup myjob, the gang ID is osmo-with-kai-team-a-myjob.

At admission, the ConfigMap contains gang_id, expected_count, master_port, peers, and master_addr. If the PodGroup is not present yet, expected_count is "0" and peers / master_addr are empty.

The ConfigMap name starts with preflight-, but long gang IDs are sanitized and truncated with a hash suffix, so use the label selector above instead of constructing the name by hand.

Grove

Grove is not supported for preflight gang coordination today. Its gang model is hierarchical — multiple nested scheduling layers — while preflight’s preflight-nccl-allreduce coordination assumes a single flat PodGroup per gang.

Grove’s hierarchy looks like this:

PodCliqueSet
  └── PodClique / PodCliqueScalingGroup
        └── PodGang
              └── podgroups[]
                    └── podReferences[]

Tracking issue: NVIDIA/NVSentinel#1354 — Parallelism-aware preflight checks.

Per-namespace gang discovery

The Helm gangDiscovery value sets only the cluster-wide default. To make a specific namespace use a different gang-scheduling system (for example, Volcano for one team while everyone else uses native Kubernetes), create a PreflightConfig custom resource in that namespace. It is reconciled at runtime — no Helm upgrade and no controller restart.

A pod is resolved as follows:

1. If the pod’s namespace has a PreflightConfig with a gangDiscovery block, that discoverer is used.
2. Otherwise, the cluster-wide gangDiscovery Helm value is used as the default.

PreflightConfig is namespaced (preflight.nvsentinel.nvidia.com/v1alpha1); the object’s own namespace is its scope. Its spec.gangDiscovery block uses the same schema as the Helm gangDiscovery value — an empty block selects native Kubernetes discovery. (spec is intentionally a container for per-namespace preflight settings, so future options can be added alongside gangDiscovery.)

# team-a uses Volcano; every other namespace uses the cluster-wide default.
apiVersion: preflight.nvsentinel.nvidia.com/v1alpha1
kind: PreflightConfig
metadata:
  name: default
  namespace: team-a
spec:
  gangDiscovery:
    name: "volcano"
    annotationKeys: ["scheduling.k8s.io/group-name"]
    podGroupGVR:
      group: "scheduling.volcano.sh"
      version: "v1beta1"
      resource: "podgroups"
    minCountExpr: "podGroup.spec.minMember"

At most one PreflightConfig should exist per namespace. If more than one is present, the oldest object (tie-broken by name) stays active so an existing working configuration is not disrupted; the additional objects are marked not ready as superseded. Check the resolved state via the object’s status:

$ kubectl -n team-a get preflightconfig
NAME      DISCOVERER   READY   AGE
default   volcano      True    10s

Readiness is reported via the Ready status condition (the READY column above is its status); its message explains why a config is not ready (invalid or superseded). Inspect it with kubectl -n team-a get preflightconfig default -o yaml under .status.conditions.

RBAC (aggregated ClusterRole)

The controller reads scheduler PodGroup resources cluster-wide through an aggregated ClusterRole ({release}-gang-discovery). The chart ships a built-in contributor role covering the native scheduling.k8s.io resources plus the default gangDiscovery.podGroupGVR. To let the controller read a scheduler CRD that isn’t covered (e.g. a namespace registers Volcano but the default is native), apply a ClusterRole labeled for aggregation — it is merged in automatically, with no preflight change or restart:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: preflight-gang-discovery-volcano
  labels:
    preflight.nvsentinel.nvidia.com/aggregate-to-gang-discovery: "true"
rules:
  - apiGroups: ["scheduling.volcano.sh"]
    resources: ["podgroups"]
    verbs: ["get", "list", "watch"]

Because ClusterRoles are cluster-scoped, creating one is a platform/cluster-admin action — a namespace tenant declares its scheduler via the PreflightConfig, while the platform grants the corresponding read access. Aggregation is eventually consistent, so a brief Forbidden window after adding a new contributor role is expected; the controller retries.

Each PreflightConfig is validated when reconciled: the gangDiscovery.podGroupGVR is resolved against the cluster’s API RESTMapper (and native specs verify the scheduling.k8s.io resources). An invalid or unresolvable config does not disrupt admission — the namespace falls back to the default and the error is surfaced in the object’s status.

Changing gang discovery configuration

PreflightConfig changes take effect on newly-admitted gangs and are applied per pod at admission. Avoid editing or deleting a namespace’s active PreflightConfig (or deleting it so a different one becomes active) while multi-node preflight gangs are being launched in that namespace.

The gang ID embeds the discoverer name ({discoverer}-{namespace}-{podGroup}), and discovery is resolved per pod. If the effective discoverer for a namespace changes mid-flight, pods of the same gang admitted before and after the change can derive different gang IDs and fail to coordinate (peers never converge). Such a gang’s preflight-nccl-allreduce check then waits until gangCoordination.timeout and fails — the pod stays in Init:Error and follows the normal NVSentinel quarantine path. This fails safe (no false “healthy” result) but causes a spurious preflight failure, so treat gang discovery config as a namespace setting to change during a quiet window. Adding a second PreflightConfig is safe — the active (oldest) one is unaffected (see above); the risk is specifically changing or removing the currently-active config.

Gang coordination Pod cache

When gang coordination is enabled, Preflight watches Pods cluster-wide because namespace-specific PreflightConfig resources can be added or removed while the process is running. The controller-runtime cache namespace set is fixed when the manager starts, so limiting it to the namespaces known at startup would miss Pods after a new namespace configuration is created.

To keep the cluster-wide cache small, Preflight stores only Pod identity and deletion metadata, annotations and labels used by configurable discoverers, the gang ConfigMap volume, node and scheduling-group references, and Pod IP and phase. Containers, init containers, unrelated volumes, conditions, managed fields, and other unused Pod data are discarded before caching.

Gang coordination

When gangCoordination.enabled is true (default in the preflight chart), the controller coordinates multi-node checks through ConfigMaps:

1. At admission time the webhook creates a skeleton ConfigMap for the gang and injects it as a volume mount on the pod’s preflight init containers.
2. As pods become ready the gang controller populates the ConfigMap with peer information (IP, rank).
3. Init containers read the ConfigMap at gangCoordination.configMapMountPath (default /etc/preflight) to discover the master address and peer list.

Each gang ConfigMap contains:

ConfigMaps are labeled nvsentinel.nvidia.com/managed-by: preflight and named with a preflight- prefix.

Key gangCoordination values

Set enabled: false when there is no gang to coordinate — a single node, or a scheduler that is not gang-aware. Drop preflight-nccl-allreduce from initContainers at the same time; see Single-node and single-GPU clusters.

gangCoordination:
  enabled: true
  timeout: "10m"            # Max wait for all members to register
  masterPort: 29500         # PyTorch distributed bootstrap port
  configMapMountPath: "/etc/preflight"
  # Azure InfiniBand topology (required for NDv4/v5)
  ncclTopoConfigMap: ""     # Pre-existing ConfigMap name, or use ncclTopoShape
  ncclTopoShape: ""         # "ndv4" or "ndv5" to auto-create from bundled XML
  extraHostPathMounts: []   # Host paths for NCCL/OFI/CUDA libraries
  extraVolumeMounts: []     # Mount existing pod volumes (e.g. GCP TCPXO plugin)
  # mirrorResourceClaims: true  # Mirror DRA claims to init containers (default true)

For DRA / device claims mirrored into init containers, see ADR-026 §DRA Integration and mirrorResourceClaims above. A DRA GPU pod always gets its claims mirrored, whatever mirrorResourceClaims says. See DRA GPU pods.

DRA GPU pods

A pod that gets its GPUs through Dynamic Resource Allocation (DRA) has no GPU resource from gpuResourceNames (for example nvidia.com/gpu). The webhook finds these pods from their resource claims and injects the checks into them too. This applies when GPUs are allocated through DRA, for example with the GPU Operator in GPUCluster mode or with the standalone NVIDIA DRA driver for GPUs.

The detection is always on and has no configuration value. Preflight does not read global.gpuDraEnabled. On a cluster without DRA, no pod has spec.resourceClaims, so the webhook makes no extra API calls.

How the webhook detects a DRA GPU pod

For a pod with no GPU resource request, the webhook reads each entry in spec.resourceClaims. It reads the ResourceClaimTemplate or the ResourceClaim that the entry names. The pod is a DRA GPU pod when one device request matches these rules:

One GPU request is enough. The webhook then injects the checks and copies all of the pod’s resource claims into each check, including NIC and IMEX claims. The NCCL checks need the NIC and IMEX devices to test the network. The webhook adds each claim one time only.

A pod that requests a GPU resource from gpuResourceNames does not go through this detection. The webhook makes no API calls for it, and its behaviour does not change.

Failures

The webhook does not reject a pod because of DRA detection. If it cannot read a claim or a template, it counts that entry as non-GPU and continues. If no entry is a GPU request, the webhook admits the pod without the checks. It never injects the checks without the pod’s claims, because a check without GPU access reports the node as unhealthy.

An Error log means the webhook counts that entry as non-GPU. The pod still gets the checks if another entry is a GPU request. An RBAC or API error usually affects every claim, so DRA GPU pods then do not get the checks. Look for Failed to read DRA claim, treating it as non-GPU in the preflight logs.

RBAC and API load

The chart always creates the ClusterRole preflight-dra and binds it to the preflight ServiceAccount. It gives get access to resourceclaims and resourceclaimtemplates in the resource.k8s.io API group, in all namespaces. It gives no list or watch access. It does not depend on gang coordination. On a cluster without DRA, the role exists but the webhook does not use it.

The webhook reads claims directly from the API server, with no cache. This keeps preflight memory usage the same on large clusters. Each DRA pod admission costs one get for each entry in spec.resourceClaims, until the webhook finds a GPU request.

Publishing to the deployment platform connector

By default the checks report through the node-local platform connector socket that the webhook mounts into the pod. A cluster can also run the deployment platform connector. Then set preflight.publishTo: deployment, and the checks publish directly to the deployment platform connector, like the health monitors do. Preflight then no longer needs the node-local DaemonSet. This setting needs platformConnector.deployment.enabled: true and platform-connector auth on. The chart refuses the other combinations.

What changes for an injected pod:

• Every check gets HEALTH_PUBLISH_TARGET, HEALTH_PUBLISH_TOKEN_PATH and HEALTH_PUBLISH_TLS_CA_FILE. The target is the Service of the deployment platform connector. The token is the same projected token as on the socket path, minted against the pod’s own ServiceAccount. The socket env and mount stay as they are. The checks ignore them while the target is set.
• The CA bundle comes from a ConfigMap named nvsentinel-platform-connector-ca. The preflight controller keeps this ConfigMap in every namespace that it injects into, because a pod cannot mount a Secret from the NVSentinel namespace. The controller creates it when a pod is admitted and for every labelled namespace. It rewrites every copy when cert-manager rotates the CA. The pod mounts it at /etc/nvsentinel/platform-connector-deployment-ca.
• The pod gets the label nvsentinel.nvidia.com/health-publisher: "true". The network policy of the deployment platform connector admits pods with this label in the namespaces that preflight.namespaceSelector selects. These are the namespaces that the webhook injects into. The policy cannot match workload pods on the host network. For such pods, set preflight.hostNetworkWorkloads: true. The chart then opens the gRPC rule to every source, and the caller token is the only gate.

A check waits at most five minutes in total for an unreachable deployment platform connector. This budget is HEALTH_PUBLISH_RETRY_WINDOW, a client default. One attempt lasts at most 30 seconds. After the budget is spent, each later report of the same run gets one attempt of 5 seconds. Thus a check that sends many reports, like preflight-dcgm-diag, waits through one outage at most. The deployment platform connector refuses a report for good when the report is invalid or names another node. The check then gives up that report at once.

An incomplete direct-mode environment is a configuration error. Examples are a target without a token path, no CA bundle without HEALTH_PUBLISH_INSECURE, or a bad HEALTH_PUBLISH_RETRY_WINDOW. A configuration error gives exit 1 for preflight-dcgm-diag and exit 2 for the two NCCL checks.

The exit codes below do not change:

• A report that the check cannot deliver still gives exit 3 for preflight-nccl-loopback and exit 4 for preflight-nccl-allreduce.
• preflight-dcgm-diag still keeps its diagnostic verdict.
• STORE_ONLY still turns a test failure into exit 0. Configuration errors keep their exit code.

The events, and what Fault Quarantine does with them, are the same on both paths.

Key Helm values (subchart)

Webhook

TLS and admission behaviour for the mutating webhook.

preflight:
  webhook:
    port: 8443
    failurePolicy: Fail
    timeoutSeconds: 10
    createIssuer: true
    certIssuer: ""
    caCertificateName: ""
    # certProvider: openshift-service-ca

failurePolicy

What the API server does when the webhook does not answer. Fail rejects the pod, so a webhook outage blocks GPU pod creation in opted-in namespaces. Ignore admits the pod without preflight checks, trading the gate for availability.

timeoutSeconds

How long the API server waits for the webhook, up to the Kubernetes maximum of 30. The default is 10. Preflight resolves gang membership during admission, so a cluster with slow PodGroup lookups may need more; every GPU pod creation waits on this call, so raise it only as far as the lookups need.

createIssuer

Creates the two-tier cert-manager CA for the webhook certificate. Set it to false to use a CA you already run, and then set certIssuer and caCertificateName.

certIssuer

Name of the existing cert-manager issuer to sign the webhook certificate. Used only when createIssuer is false.

caCertificateName

Name of the existing CA certificate whose bundle is injected into the webhook configuration. Used only when createIssuer is false. Without it the API server cannot verify the webhook, and admission fails closed under failurePolicy: Fail.

certProvider

cert-manager by default. Set it to openshift-service-ca on OpenShift to issue the webhook certificate with the built-in service-ca-operator instead of cert-manager.

Object selector (pod-level filtering)

By default the webhook intercepts all GPU pods in labeled namespaces. To further restrict which pods are intercepted, set objectSelector with standard Kubernetes label selectors. When empty ({}), no objectSelector is emitted and all pods in matching namespaces are intercepted.

Example — only intercept pods explicitly labeled for preflight:

objectSelector:
  matchLabels:
    nvsentinel.nvidia.com/preflight: "enabled"

matchExpressions are also supported:

objectSelector:
  matchExpressions:
    - key: nvsentinel.nvidia.com/preflight
      operator: In
      values: ["enabled", "true"]

This is useful when you want namespace-wide opt-in via namespaceSelector but only run preflight on specific workloads within those namespaces.

Full defaults and comments: distros/kubernetes/nvsentinel/charts/preflight/values.yaml.

Tilt development often trims init containers to DCGM-only; see distros/kubernetes/nvsentinel/values-tilt.yaml.

Observability

• Webhook pod: liveness/readiness probes use /healthz on the webhook port.
• Prometheus metric names for check containers and the injector are specified in ADR-026 § Metrics; wire scrapers to your init container images and deployment as your environment allows.

Debugging preflight failures

When a preflight check fails, the pod stays in Init:Error and the init container exits non-zero.

1. Check the exit code

kubectl -n {namespace} get pod {pod-name} -o jsonpath=\
'{range .status.initContainerStatuses[*]}{.name}{"\t"}{.state.terminated.exitCode}{"\n"}{end}'

Each check has its own codes:

The exit code decides whether the pod starts. The health event decides what happens to the node: Fault Quarantine cordons on preflight events with isFatal: true only. Hardware findings are fatal, and coordination or configuration problems are not — so a gang timeout holds the pod in Init:Error without cordoning the node.

A check that fails before it can report sends no event at all. A preflight-dcgm-diag configuration error (exit 1) and a preflight-nccl-loopback configuration error (exit 2) both block the pod and leave no event to find.

When processingStrategy is STORE_ONLY, a check reports its result and then exits 0, logging Check failed (STORE_ONLY — not blocking pod), so the event is recorded and the workload still starts. The gang stage of preflight-nccl-allreduce runs before the benchmark and returns exit 2 or 3 directly, so STORE_ONLY does not unblock a gang failure.

A check can exit 0 and still report an unhealthy GPU

preflight-dcgm-diag sends one health event for each DCGM test result. Three outcomes report a problem without failing the gate:

• A test result of warn.
• A test result of fail whose error code has no actionable remediation — DCGM_FR_XID_ERROR, for example.
• A DCGM_ST_* status that still prevents the diagnostic from completing after DCGM_DIAG_STATUS_RETRY_MAX_ATTEMPTS attempts.

Each one emits an unhealthy event with isFatal: false, and the check exits 0, so a warning or an infrastructure problem does not block the workload. The node then carries a preflight health event, and a node condition, while staying schedulable. A non-fatal event can still carry a recommended action, so read the event rather than inferring the finding from the exit code.

2. Check health events

All three checks (preflight-dcgm-diag, preflight-nccl-loopback, preflight-nccl-allreduce) report results as health events. From the NVSentinel repository root, query them with the MongoDB shell helper:

./scripts/mongodb-shell.sh

Once connected, filter for the failing node (replace {NODE_NAME}):

db.HealthEvents.find({"healthevent.nodename": "{NODE_NAME}"}).pretty()

See Connecting to the Datastore for more query examples.

Init container logs are not always available (for example if the pod was already cleaned up). In that case, rely on the health events and node conditions instead:

kubectl describe node {node-name} | grep -A3 Conditions

3. Act on the failure

• preflight-dcgm-diag failed — the GPU/node is in a bad state. The node needs to be fixed (reboot or terminate).
• preflight-nccl-loopback or preflight-nccl-allreduce failed — first confirm the BW_THRESHOLD_GBPS is realistic for the hardware. The default (150 for loopback, 100 for all-reduce) assumes NVLink; it is too high for PCIe-only interconnect or lower-bandwidth GPUs (for example, L40S sustains ~20 GB/s), where the check would never pass. If the threshold is set too high for your GPUs, lower it to the expected value for that interconnect. If the threshold is already correct for the hardware, do not lower it to force a pass — re-run the check once to rule out a false positive, and if it fails again, check the health event and take the needful action such as rebooting or terminating the node.

Related documentation

• ADR-026: Preflight checks
• ADR-035: Inline DCGM config — design rationale for inline env var configuration
• gRPC / TLS authentication (mentions preflight among webhooks)
• Helm chart README
• E2E test entry point: tests/preflight_test.go (build tag amd64_group)