> This page is for NVSentinel, version v1.26.0.
> For other versions, use one of these documentation indexes:
> - Latest · v1.28.0 (default): https://docs.nvidia.com/nvsentinel/v1.28.0/llms.txt
> - v1.27.0: https://docs.nvidia.com/nvsentinel/v1.27.0/llms.txt
> - v1.26.0: https://docs.nvidia.com/nvsentinel/v1.26.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nvsentinel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nvsentinel/_mcp/server.

# Preflight configuration

Preflight is a mutating admission webhook that injects GPU diagnostic init containers (DCGM diagnostics, optional NCCL loopback / all-reduce) into pods that request GPUs in namespaces you opt in via labels. It answers "is this GPU healthy enough to start?" before your workload runs—separate from continuous [GPU health monitoring](/nvsentinel/v1.26.0/components/gpu-health-monitor).


## Prerequisites

- Helm subchart is off by default; enable with `global.preflight.enabled` (see below).
- cert-manager (or OpenShift service CA) for webhook TLS—same expectation as the rest of the NVSentinel chart.
- DCGM reachable from injected init containers (typically the NVIDIA GPU Operator's DCGM / hostengine service). Configure the endpoint via `DCGM_HOSTENGINE_ADDR` on the `preflight-dcgm-diag` init container.
- Multi-node / gang checks (e.g. `preflight-nccl-allreduce`): enable gang coordination and configure gang discovery for your scheduler (see below). The default gang discovery uses the native Kubernetes gang APIs, which need Kubernetes 1.35 or later. On an earlier cluster, configure a PodGroup-based scheduler or turn gang coordination off — see [Single-node and single-GPU clusters](#single-node-and-single-gpu-clusters).

## Enable preflight

1. Set the global flag:

```yaml
global:
  preflight:
    enabled: true
```

2. Configure the preflight subchart under the top-level `preflight:` key (values merge into `distros/kubernetes/nvsentinel/charts/preflight/values.yaml`). At minimum, review `initContainers` (including DCGM and NCCL env vars) and `webhook.failurePolicy`.

3. Label namespaces where injection should apply:

```bash
kubectl label namespace {namespace} nvsentinel.nvidia.com/preflight=enabled
```

The chart default `namespaceSelector` matches that label.

### Single-node and single-GPU clusters

The chart enables gang coordination by default, and the default `gangDiscovery` (`{}`) selects native Kubernetes gang discovery. Preflight validates that discoverer against the cluster at startup, so on a cluster older than Kubernetes 1.35 the preflight pod does not start. Its log names the requirement:

```bash
kubectl -n nvsentinel logs deploy/preflight
```

```text
"level":"ERROR","msg":"Fatal error","error":"failed to create gang discoverer resolver: failed to create default gang discoverer: kubernetes native gang API not available (requires K8s 1.35 Workload or K8s 1.36+ PodGroup)"
```

A single node has no gang to coordinate. Turn coordination off and inject only the single-node GPU check:

```yaml
global:
  preflight:
    enabled: true

preflight:
  gangCoordination:
    enabled: false
  initContainers:
    - name: preflight-dcgm-diag
      image:
        repository: ghcr.io/nvidia/nvsentinel/preflight-dcgm-diag
        tag: ""
      env:
        - name: DCGM_HOSTENGINE_ADDR
          value: "nvidia-dcgm.gpu-operator.svc:5555"
        - name: DCGM_DIAG_LEVEL
          value: "2"
      volumeMounts:
        - name: nvsentinel-socket
          mountPath: /var/run
```

Helm replaces the `initContainers` list rather than merging it, so this example also drops the two NCCL checks:

- `preflight-nccl-allreduce` needs gang coordination. With coordination off, the webhook injects no gang environment, the check exits `2` (`NCCL_GANG_CONFIG_ERROR`), and the pod stays in `Init:Error`.
- `preflight-nccl-loopback` measures the GPU-to-GPU path on the node, which a single-GPU node does not have. Keep it on a multi-GPU node and set `BW_THRESHOLD_GBPS` for the interconnect — approximately `15` for PCIe, `150` for NVLink.

The same values apply to a multi-node cluster whose scheduler is not gang-aware. If your scheduler does use PodGroups (Volcano, Run:ai / OSMO), keep gang coordination on and configure [gang discovery](#gang-discovery) instead — that path works on any Kubernetes version that runs the scheduler.

### Image cache

Runs a DaemonSet that holds every configured check image on matching nodes, so the kubelet treats the images as in use and its garbage collector does not evict them.

```yaml
preflight:
  imageCache:
    enabled: false
    resources:
      requests:
        cpu: 10m
        memory: 16Mi
      limits:
        cpu: 50m
        memory: 64Mi
    podAnnotations: {}
```

Off by default. Enable it where image pulls are slow or metered: preflight runs on the pod admission path, so a cold pull of a multi-gigabyte CUDA check image delays every GPU workload start on that node. The cost is one small pod per node, plus the disk the cached images occupy.

## Init container placement

By default the webhook **appends** preflight init containers after any existing init containers in the pod spec. This ensures provider-injected setup containers (e.g., GCP TCPXO daemon) complete before preflight checks run.

Set `initContainerPlacement` to change this behavior:

```yaml
# "append" (default): add after existing init containers
# "prepend": add before existing init containers
initContainerPlacement: "prepend"
```

Use `prepend` when preflight checks must run before other init containers — for example, to gate workload setup on GPU health validation.

## Per-pod check selection

By default, all init containers with `defaultEnabled: true` (or omitted, which defaults to true) are injected into every GPU pod. To select a subset of checks for a specific pod, annotate it:

```yaml
metadata:
  annotations:
    nvsentinel.nvidia.com/preflight-checks: "preflight-dcgm-diag,preflight-nccl-loopback"
```

Only the named containers are injected, in the order they appear in the annotation. Duplicate or unknown container names reject admission with an error.

An empty value disables all checks:

```yaml
nvsentinel.nvidia.com/preflight-checks: ""
```

When the annotation is absent, `defaultEnabled` on each init container controls whether it runs. For gang-aware checks (`nccl-allreduce`), all pods in the gang must have the same annotation value — mismatches are detected and fail fast before torchrun launches.

See [ADR-034](../designs/034-preflight-check-selection.md) for design details.

## Init containers (check configuration)

The `initContainers` list in the preflight chart defines which checks the webhook injects. Each entry is a standard `corev1.Container` plus preflight-specific controls such as `defaultEnabled`, `inheritUserEnv`, and `inheritUserVolumeMounts` — you control images, env vars, resource limits, security contexts, and volume mounts.

The webhook automatically injects these env vars into every init container (you do not need to set them):

| Env var | Source | Purpose |
|---------|--------|---------|
| `NODE_NAME` | Downward API (`spec.nodeName`) | Kubernetes node name for health events |
| `PLATFORM_CONNECTOR_SOCKET` | Chart `connectorSocket` | Unix socket for the platform-connector gRPC endpoint |
| `PROCESSING_STRATEGY` | Chart `processingStrategy` | `EXECUTE_REMEDIATION` or `STORE_ONLY` — controls downstream action |
| `PLATFORM_CONNECTOR_TOKEN_PATH` | Chart `global.platformConnectorAuth` | Projected token the check presents, when platform-connector auth is on |
| `HEALTH_PUBLISH_TARGET`, `HEALTH_PUBLISH_TOKEN_PATH`, `HEALTH_PUBLISH_TLS_CA_FILE` (or `HEALTH_PUBLISH_INSECURE`) | Chart `publishTo: deployment` | Direct connection to the deployment platform connector, see [below](#publishing-to-the-deployment-platform-connector) |

For gang-aware containers the webhook also injects `GANG_ID`, `GANG_CONFIG_DIR`, `GANG_TIMEOUT_SECONDS`, and `POD_NAME`.

`inheritUserEnv` and `inheritUserVolumeMounts` both default to `true` when omitted, and the chart sets both to `true` on each built-in check. Every check therefore mirrors the workload's NCCL and fabric configuration by default — see [Fabric-specific NCCL configuration](#fabric-specific-nccl-configuration) for the patterns that decide what is copied. Set either flag to `false` on an `initContainers` entry to give that check a curated environment instead.

### preflight-dcgm-diag

Runs DCGM diagnostics against every GPU allocated to the pod via the remote hostengine.

| Env var | Default | Description |
|---------|---------|-------------|
| `DCGM_DIAG_LEVEL` | `2` | Diagnostic depth: 1 = short (approx 30 s, software deployment checks), 2 = medium (approx 2 min, adds PCIe and basic GPU stress), 3 = long (approx 15 min, adds Diagnostic plugin stress), 4 = xlong (1-2 hr, extended stress) |
| `DCGM_HOSTENGINE_ADDR` | `nvidia-dcgm-dra.gpu-operator.svc:5555,nvidia-dcgm.gpu-operator.svc:5555` | Comma-separated DCGM hostengine gRPC endpoints, tried in order. Only one exists per cluster (`nvidia-dcgm-dra` in GPU Operator GPUCluster mode) |
| `DCGM_DIAG_STATUS_RETRY_MAX_ATTEMPTS` | `10` | Maximum diagnostic attempts when DCGM returns a `DCGM_ST_*` status while starting/running diagnostics |
| `DCGM_DIAG_STATUS_RETRY_INTERVAL_SECONDS` | `10` | Delay between `DCGM_ST_*` retry attempts |

Example values override:

```yaml
initContainers:
  - name: preflight-dcgm-diag
    image:
      repository: ghcr.io/nvidia/nvsentinel/preflight-dcgm-diag
      tag: ""
    env:
      - name: DCGM_HOSTENGINE_ADDR
        value: "nvidia-dcgm-dra.gpu-operator.svc:5555,nvidia-dcgm.gpu-operator.svc:5555"
      - name: DCGM_DIAG_LEVEL
        value: "2"
      - name: DCGM_DIAG_STATUS_RETRY_MAX_ATTEMPTS
        value: "10"
      - name: DCGM_DIAG_STATUS_RETRY_INTERVAL_SECONDS
        value: "10"
    volumeMounts:
      - name: nvsentinel-socket
        mountPath: /var/run
```

If a `DCGM_ST_*` status still prevents the diagnostic from completing after
retries, `preflight-dcgm-diag` emits a non-fatal unhealthy HealthEvent with
`RecommendedAction=NONE` and exits successfully so the workload is not blocked by
a preflight infrastructure failure.

### preflight-nccl-loopback

Single-node NCCL all-reduce across all GPUs on the node. Validates intra-node interconnect (NVLink or PCIe).

| Env var | Default | Description |
|---------|---------|-------------|
| `BW_THRESHOLD_GBPS` | `150` | Minimum acceptable bus bandwidth in GB/s. NVLink interconnect typically sustains 150+ GB/s; set to approx 15 GB/s for PCIe interconnect |
| `TEST_SIZE_MB` | `256` | Message size in MB for the all-reduce benchmark |
| `SKIP_BANDWIDTH_CHECK` | `false` | When `true`, pass if the benchmark completes regardless of measured bandwidth |

Example values override:

```yaml
initContainers:
  - name: preflight-nccl-loopback
    image: ghcr.io/nvidia/nvsentinel/preflight-nccl-loopback:latest
    env:
      - name: BW_THRESHOLD_GBPS
        value: "15"           # PCIe interconnect
      - name: TEST_SIZE_MB
        value: "512"
```

### preflight-nccl-allreduce

Multi-node NCCL all-reduce across the entire gang. Requires `gangCoordination.enabled: true` and a gang-aware scheduler.

| Env var | Default | Description |
|---------|---------|-------------|
| `BW_THRESHOLD_GBPS` | `100` | Minimum acceptable bus bandwidth in GB/s |
| `MESSAGE_SIZES` | `4G` | Comma-separated message sizes for the benchmark (e.g. `"4G"`, `"4G,8G"`). Code default is `4G,8G`; Helm chart overrides to `4G` |
| `BENCHMARK_ITERS` | `20` | Number of timed iterations per message size |
| `WARMUP_ITERS` | `5` | Warmup iterations before timing begins |
| `NCCL_REDUCE_OP` | `sum` | Reduction operation (`sum`, `prod`, `min`, `max`, `avg`) |
| `SKIP_BANDWIDTH_CHECK` | `false` | Pass if benchmark completes regardless of bandwidth |
| `NCCL_DEBUG` | — | NCCL log verbosity (`INFO`, `WARN`, etc.) |
| `NCCL_DEBUG_SUBSYS` | — | NCCL subsystems to log (`INIT,NET`, etc.) |

The container also requires `IPC_LOCK` capability for RDMA memory registration:

```yaml
initContainers:
  - name: preflight-nccl-allreduce
    image: ghcr.io/nvidia/nvsentinel/preflight-nccl-allreduce:latest
    securityContext:
      capabilities:
        add: ["IPC_LOCK"]
    env:
      - name: BW_THRESHOLD_GBPS
        value: "100"
      - name: MESSAGE_SIZES
        value: "4G"
```

### Fabric-specific NCCL configuration

When a check has `inheritUserEnv` or `inheritUserVolumeMounts` enabled (the default for both), the webhook copies matching NCCL env vars and volume mounts from the pod's main containers using glob patterns:

```yaml
ncclEnvPatterns:    ["NCCL_*", "FI_*", "LD_LIBRARY_PATH", "UCX_*", "TORCH_NCCL_*", "CUDA_DEVICE_ORDER"]
volumeMountPatterns: ["host-opt-amazon*", "nvtcpxo-*", "nccl-*", "dev-shm"]
```

This means if your training container already has the correct `NCCL_TOPO_FILE`, `FI_PROVIDER`, or `LD_LIBRARY_PATH`, a preflight init container inherits them with no manual configuration.

Inheritance is per init container and is on by default. Chart-defined env wins on a name conflict, so an inherited value only fills a name the check does not set itself. Set the flags to `false` on a check whose environment must not depend on workload NCCL tuning:

```yaml
initContainers:
  - name: preflight-nccl-allreduce
    inheritUserEnv: false
    inheritUserVolumeMounts: false
```

For standalone testing (e.g. busybox main container), use `ncclAllreduceExtraEnv` and `gangCoordination.extraHostPathMounts` to provide fabric config explicitly.

## Gang discovery

Gang discovery identifies pods that belong to the same scheduling group so multi-node preflight checks (NCCL all-reduce) know their peers. A pod carries a "gang anchor"—a reference to a parent object—that holds gang metadata such as the minimum member count.

Two discovery mechanisms are supported:

### Native Kubernetes: schedulingGroup / workloadRef

The default when `gangDiscovery` is left empty (`{}`). Preflight first uses the Kubernetes 1.36 native PodGroup API when available, then falls back to the Kubernetes 1.35 native Workload API. Neither exists before Kubernetes 1.35, and preflight validates the discoverer at startup, so on an earlier cluster the pod fails to start — see [Single-node and single-GPU clusters](#single-node-and-single-gpu-clusters).

> The `PodGroup` resource (`scheduling.k8s.io/v1alpha2`) and `spec.schedulingGroup` are alpha in Kubernetes 1.36 and disabled by default. Enable the `GenericWorkload` feature gate on the API server and scheduler to use this path.

In Kubernetes 1.36, each pod links to a `PodGroup` resource via `spec.schedulingGroup`:

```yaml
spec:
  schedulingGroup:
    podGroupName: training-workers
```

The PodGroup object contains a gang policy with `minCount`:

```yaml
apiVersion: scheduling.k8s.io/v1alpha2
kind: PodGroup
metadata:
  name: training-workers
spec:
  schedulingPolicy:
    gang:
      minCount: 2
```

No `gangDiscovery` configuration is needed for this path.

In Kubernetes 1.35, each pod links to a native `Workload` resource via `spec.workloadRef`:

```yaml
spec:
  workloadRef:
    name: training-job-workload
    podGroup: workers
```

The Workload object contains pod groups and gang policy:

```yaml
apiVersion: scheduling.k8s.io/v1alpha1
kind: Workload
metadata:
  name: training-job-workload
spec:
  podGroups:
    - name: workers
      policy:
        gang:
          minCount: 2
```

No `gangDiscovery` configuration is needed for this fallback path either.

The default chart RBAC grants read access to both native resources:
`scheduling.k8s.io/podgroups` for Kubernetes 1.36 and
`scheduling.k8s.io/workloads` for Kubernetes 1.35.

### PodGroup-based schedulers (Volcano, Run:ai / OSMO, and similar)

For schedulers that use PodGroup CRDs, configure `gangDiscovery` with:

| Field | Purpose |
|-------|---------|
| `name` | Discoverer identifier, used in the gang ID prefix and logging (e.g. `"volcano"`) |
| `annotationKeys` | Pod annotation keys checked (in order) for the PodGroup name |
| `labelKeys` | Optional pod label keys checked as fallback |
| `podGroupGVR` | `group`, `version`, `resource` of the PodGroup CRD |
| `minCountExpr` | CEL expression to extract the minimum member count from the PodGroup object. Receives `podGroup` as the unstructured object. Default: `"podGroup.spec.minMember"` |

Volcano example:

```yaml
gangDiscovery:
  name: "volcano"
  annotationKeys:
    - "scheduling.k8s.io/group-name"
  podGroupGVR:
    group: "scheduling.volcano.sh"
    version: "v1beta1"
    resource: "podgroups"
  minCountExpr: "podGroup.spec.minMember"
```

Volcano sets the `scheduling.k8s.io/group-name` annotation on each pod. The discoverer reads that annotation, fetches the corresponding `PodGroup` CRD, and evaluates `minCountExpr` to determine expected gang size.

OSMO + Kai scheduler example:

```yaml
gangDiscovery:
  name: "osmo-with-kai"
  labelKeys:
    - "osmo.group_uuid"
  podGroupGVR:
    group: "scheduling.run.ai"
    version: "v2alpha2"
    resource: "podgroups"
  minCountExpr: "podGroup.spec.minMember"
```

Here membership is determined by a pod label instead of an annotation. The rest of the flow is the same: look up the PodGroup CRD and extract `minCount` via CEL.

### OSMO + KAI Scheduler

[KAI Scheduler](https://github.com/kai-scheduler/KAI-Scheduler) uses the `scheduling.run.ai/v2alpha2` PodGroup CRD. KAI Scheduler can run on its own, but in that setup the gang anchor is applied **after** admission, so preflight has no label or annotation to hook on when the webhook runs. This section documents the **OSMO + KAI** integration instead: OSMO creates the PodGroup and labels each pod with `osmo.group_uuid` at admission time. Preflight reads that label, fetches the PodGroup, and uses `spec.minMember` as the expected gang size for `preflight-nccl-allreduce`.

#### Step 1 — Enable preflight cluster-wide

```yaml
global:
  preflight:
    enabled: true

preflight:
  gangDiscovery:
    name: "osmo-with-kai"
    labelKeys:
      - "osmo.group_uuid"
    podGroupGVR:
      group: "scheduling.run.ai"
      version: "v2alpha2"
      resource: "podgroups"
    minCountExpr: "podGroup.spec.minMember"
```

The chart's built-in RBAC contributor role grants read access to `scheduling.run.ai/podgroups` when `gangDiscovery.podGroupGVR` is set in Helm values (see [RBAC (aggregated ClusterRole)](#rbac-aggregated-clusterrole) below).

#### Step 2 — Label namespaces for injection

```bash
kubectl label namespace {training-namespace} nvsentinel.nvidia.com/preflight=enabled
```

#### Step 3 — Verify image pull secrets

If preflight check images are in a private registry, set `injectedImagePullSecrets` in the preflight Helm values (`[charts/preflight/values.yaml](../../distros/kubernetes/nvsentinel/charts/preflight/values.yaml)`). The webhook injects those secrets into admitted pods as `spec.imagePullSecrets`.

Check whether any secrets are configured:

```bash
kubectl -n nvsentinel get configmap preflight -o jsonpath='{.data.config\.yaml}' | yq .imagePullSecrets
```

If `imagePullSecrets` is present, confirm each secret exists in your workload namespace:

```bash
kubectl -n {training-namespace} get secret {secret-name}
```

If `injectedImagePullSecrets` is empty (`[]`), no pull secrets are injected and this step can be skipped.

#### Step 4 — Submit a workload

Submit your OSMO gang-scheduled workload in the labeled namespace.

#### Step 5 — Verify injection

After a GPU pod is admitted, confirm the webhook injected preflight init containers and created the gang ConfigMap:

```bash
# Init containers injected (preflight-dcgm-diag, preflight-nccl-loopback, preflight-nccl-allreduce)
kubectl -n {training-namespace} get pod {pod-name} -o jsonpath='{range .spec.initContainers[*]}{.name}{": "}{.image}{"\n"}{end}'

# Gang ConfigMap created at admission
kubectl -n {training-namespace} get configmap -l nvsentinel.nvidia.com/managed-by=preflight -o yaml
```

The gang ID is `{gangDiscovery.name}-{namespace}-{podGroupName}`. For example, with `name: osmo-with-kai` in namespace `team-a` and PodGroup `myjob`, the gang ID is `osmo-with-kai-team-a-myjob`.

At admission, the ConfigMap contains `gang_id`, `expected_count`, `master_port`, `peers`, and `master_addr`. If the PodGroup is not present yet, `expected_count` is `"0"` and `peers` / `master_addr` are empty.

The ConfigMap name starts with `preflight-`, but long gang IDs are sanitized and truncated with a hash suffix, so use the label selector above instead of constructing the name by hand.

### Grove

[Grove](https://github.com/ai-dynamo/grove) is **not supported** for preflight gang coordination today. Its gang model is **hierarchical** — multiple nested scheduling layers — while preflight's `preflight-nccl-allreduce` coordination assumes a **single flat PodGroup** per gang.

Grove's hierarchy looks like this:

```text
PodCliqueSet
  └── PodClique / PodCliqueScalingGroup
        └── PodGang
              └── podgroups[]
                    └── podReferences[]
```

Tracking issue: [NVIDIA/NVSentinel#1354 — Parallelism-aware preflight checks](https://github.com/NVIDIA/NVSentinel/issues/1354).

### Per-namespace gang discovery

The Helm `gangDiscovery` value sets only the **cluster-wide default**. To make a specific namespace use a different gang-scheduling system (for example, Volcano for one team while everyone else uses native Kubernetes), create a **`PreflightConfig`** custom resource in that namespace. It is reconciled at runtime — no Helm upgrade and no controller restart.

A pod is resolved as follows:

1. If the pod's namespace has a `PreflightConfig` with a `gangDiscovery` block, that discoverer is used.
2. Otherwise, the cluster-wide `gangDiscovery` Helm value is used as the default.

`PreflightConfig` is namespaced (`preflight.nvsentinel.nvidia.com/v1alpha1`); the object's own namespace is its scope. Its `spec.gangDiscovery` block uses the same schema as the Helm `gangDiscovery` value — an empty block selects native Kubernetes discovery. (`spec` is intentionally a container for per-namespace preflight settings, so future options can be added alongside `gangDiscovery`.)

```yaml
# team-a uses Volcano; every other namespace uses the cluster-wide default.
apiVersion: preflight.nvsentinel.nvidia.com/v1alpha1
kind: PreflightConfig
metadata:
  name: default
  namespace: team-a
spec:
  gangDiscovery:
    name: "volcano"
    annotationKeys: ["scheduling.k8s.io/group-name"]
    podGroupGVR:
      group: "scheduling.volcano.sh"
      version: "v1beta1"
      resource: "podgroups"
    minCountExpr: "podGroup.spec.minMember"
```

At most one `PreflightConfig` should exist per namespace. If more than one is present, the **oldest** object (tie-broken by name) stays active so an existing working configuration is not disrupted; the additional objects are marked not ready as superseded. Check the resolved state via the object's status:

```console
$ kubectl -n team-a get preflightconfig
NAME      DISCOVERER   READY   AGE
default   volcano      True    10s
```

Readiness is reported via the `Ready` status condition (the `READY` column above is its status); its message explains why a config is not ready (invalid or superseded). Inspect it with `kubectl -n team-a get preflightconfig default -o yaml` under `.status.conditions`.

#### RBAC (aggregated ClusterRole)

The controller reads scheduler `PodGroup` resources cluster-wide through an **aggregated ClusterRole** (`{release}-gang-discovery`). The chart ships a built-in contributor role covering the native `scheduling.k8s.io` resources plus the default `gangDiscovery.podGroupGVR`. To let the controller read a scheduler CRD that isn't covered (e.g. a namespace registers Volcano but the default is native), apply a `ClusterRole` labeled for aggregation — it is merged in automatically, with no preflight change or restart:

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: preflight-gang-discovery-volcano
  labels:
    preflight.nvsentinel.nvidia.com/aggregate-to-gang-discovery: "true"
rules:
  - apiGroups: ["scheduling.volcano.sh"]
    resources: ["podgroups"]
    verbs: ["get", "list", "watch"]
```

Because `ClusterRole`s are cluster-scoped, creating one is a platform/cluster-admin action — a namespace tenant declares its scheduler via the `PreflightConfig`, while the platform grants the corresponding read access. Aggregation is eventually consistent, so a brief `Forbidden` window after adding a new contributor role is expected; the controller retries.

Each `PreflightConfig` is validated when reconciled: the `gangDiscovery.podGroupGVR` is resolved against the cluster's API RESTMapper (and native specs verify the `scheduling.k8s.io` resources). An invalid or unresolvable config does not disrupt admission — the namespace falls back to the default and the error is surfaced in the object's status.

#### Changing gang discovery configuration

`PreflightConfig` changes take effect on **newly-admitted gangs** and are applied per pod at admission. Avoid editing or deleting a namespace's active `PreflightConfig` (or deleting it so a different one becomes active) **while multi-node preflight gangs are being launched** in that namespace.

The gang ID embeds the discoverer name (`{discoverer}-{namespace}-{podGroup}`), and discovery is resolved per pod. If the effective discoverer for a namespace changes mid-flight, pods of the same gang admitted before and after the change can derive **different gang IDs** and fail to coordinate (peers never converge). Such a gang's `preflight-nccl-allreduce` check then waits until `gangCoordination.timeout` and fails — the pod stays in `Init:Error` and follows the normal NVSentinel quarantine path. This fails safe (no false "healthy" result) but causes a spurious preflight failure, so treat gang discovery config as a namespace setting to change during a quiet window. Adding a *second* `PreflightConfig` is safe — the active (oldest) one is unaffected (see above); the risk is specifically changing or removing the currently-active config.

#### Gang coordination Pod cache

When gang coordination is enabled, Preflight watches Pods cluster-wide because namespace-specific `PreflightConfig` resources can be added or removed while the process is running. The controller-runtime cache namespace set is fixed when the manager starts, so limiting it to the namespaces known at startup would miss Pods after a new namespace configuration is created.

To keep the cluster-wide cache small, Preflight stores only Pod identity and deletion metadata, annotations and labels used by configurable discoverers, the gang ConfigMap volume, node and scheduling-group references, and Pod IP and phase. Containers, init containers, unrelated volumes, conditions, managed fields, and other unused Pod data are discarded before caching.

## Gang coordination

When `gangCoordination.enabled` is true (default in the preflight chart), the controller coordinates multi-node checks through ConfigMaps:

1. At admission time the webhook creates a skeleton ConfigMap for the gang and injects it as a volume mount on the pod's preflight init containers.
2. As pods become ready the gang controller populates the ConfigMap with peer information (IP, rank).
3. Init containers read the ConfigMap at `gangCoordination.configMapMountPath` (default `/etc/preflight`) to discover the master address and peer list.

Each gang ConfigMap contains:

| Key | Value |
|-----|-------|
| `expected_count` | Minimum members needed (from the Workload / PodGroup CRD) |
| `peers` | Newline-separated list of `podName;podIP;rank` |
| `master_addr` | IP of the rank-0 pod |
| `master_port` | Port for PyTorch distributed TCP bootstrap (default `29500`) |
| `gang_id` | Unique gang identifier (discoverer prefix + namespace + group) |

ConfigMaps are labeled `nvsentinel.nvidia.com/managed-by: preflight` and named with a `preflight-` prefix.

### Key `gangCoordination` values

Set `enabled: false` when there is no gang to coordinate — a single node, or a scheduler that is not gang-aware. Drop `preflight-nccl-allreduce` from `initContainers` at the same time; see [Single-node and single-GPU clusters](#single-node-and-single-gpu-clusters).

```yaml
gangCoordination:
  enabled: true
  timeout: "10m"            # Max wait for all members to register
  masterPort: 29500         # PyTorch distributed bootstrap port
  configMapMountPath: "/etc/preflight"

  # Azure InfiniBand topology (required for NDv4/v5)
  ncclTopoConfigMap: ""     # Pre-existing ConfigMap name, or use ncclTopoShape
  ncclTopoShape: ""         # "ndv4" or "ndv5" to auto-create from bundled XML

  extraHostPathMounts: []   # Host paths for NCCL/OFI/CUDA libraries
  extraVolumeMounts: []     # Mount existing pod volumes (e.g. GCP TCPXO plugin)
  # mirrorResourceClaims: true  # Mirror DRA claims to init containers (default true)
```

For DRA / device claims mirrored into init containers, see [ADR-026 §DRA Integration](../designs/026-preflight-checks.md) and `mirrorResourceClaims` above.

## Publishing to the deployment platform connector

By default the checks report through the node-local platform connector socket that the webhook mounts into the pod. A cluster can also run the [deployment platform connector](/nvsentinel/v1.26.0/configuration/platform-connectors#deployment-platform-connector). Then set `preflight.publishTo: deployment`, and the checks publish directly to the deployment platform connector, like the health monitors do. Preflight then no longer needs the node-local DaemonSet. This setting needs `platformConnector.deployment.enabled: true` and platform-connector auth on. The chart refuses the other combinations.

What changes for an injected pod:

- Every check gets `HEALTH_PUBLISH_TARGET`, `HEALTH_PUBLISH_TOKEN_PATH` and `HEALTH_PUBLISH_TLS_CA_FILE`. The target is the Service of the deployment platform connector. The token is the same projected token as on the socket path, minted against the pod's own ServiceAccount. The socket env and mount stay as they are. The checks ignore them while the target is set.
- The CA bundle comes from a ConfigMap named `nvsentinel-platform-connector-ca`. The preflight controller keeps this ConfigMap in every namespace that it injects into, because a pod cannot mount a Secret from the NVSentinel namespace. The controller creates it when a pod is admitted and for every labelled namespace. It rewrites every copy when cert-manager rotates the CA. The pod mounts it at `/etc/nvsentinel/platform-connector-deployment-ca`.
- The pod gets the label `nvsentinel.nvidia.com/health-publisher: "true"`. The network policy of the deployment platform connector admits pods with this label in the namespaces that `preflight.namespaceSelector` selects. These are the namespaces that the webhook injects into. The policy cannot match workload pods on the host network. For such pods, set `preflight.hostNetworkWorkloads: true`. The chart then opens the gRPC rule to every source, and the caller token is the only gate.

A check waits at most five minutes in total for an unreachable deployment platform connector. This budget is `HEALTH_PUBLISH_RETRY_WINDOW`, a client default. One attempt lasts at most 30 seconds. After the budget is spent, each later report of the same run gets one attempt of 5 seconds. Thus a check that sends many reports, like `preflight-dcgm-diag`, waits through one outage at most. The deployment platform connector refuses a report for good when the report is invalid or names another node. The check then gives up that report at once.

An incomplete direct-mode environment is a configuration error. Examples are a target without a token path, no CA bundle without `HEALTH_PUBLISH_INSECURE`, or a bad `HEALTH_PUBLISH_RETRY_WINDOW`. A configuration error gives exit `1` for `preflight-dcgm-diag` and exit `2` for the two NCCL checks.

The [exit codes](#1-check-the-exit-code) below do not change:

- A report that the check cannot deliver still gives exit `3` for `preflight-nccl-loopback` and exit `4` for `preflight-nccl-allreduce`.
- `preflight-dcgm-diag` still keeps its diagnostic verdict.
- `STORE_ONLY` still turns a test failure into exit `0`. Configuration errors keep their exit code.

The events, and what Fault Quarantine does with them, are the same on both paths.

## Key Helm values (subchart)

| Area | Location |
|------|-----------|
| Webhook TLS, failure policy, cert provider | `preflight.webhook` |
| Where the checks publish (socket or deployment), host-network workloads | `preflight.publishTo`, `preflight.hostNetworkWorkloads` |
| Init container placement (append/prepend) | `preflight.initContainerPlacement` |
| Injected init container images and env | `preflight.initContainers` |
| GPU / network resource names | `preflight.gpuResourceNames`, `preflight.networkResourceNames` |
| Copy NCCL / fabric env and mounts from user containers | `preflight.ncclEnvPatterns`, `preflight.volumeMountPatterns` |
| Gang discovery | `preflight.gangDiscovery` |
| Gang coordination (timeouts, topology, mounts) | `preflight.gangCoordination` |
| Namespace selector for the webhook | `preflight.namespaceSelector` |
| Pod-level selector for the webhook | `preflight.objectSelector` |

## Webhook

TLS and admission behaviour for the mutating webhook.

```yaml
preflight:
  webhook:
    port: 8443
    failurePolicy: Fail
    timeoutSeconds: 10
    createIssuer: true
    certIssuer: ""
    caCertificateName: ""
    # certProvider: openshift-service-ca
```

### failurePolicy
What the API server does when the webhook does not answer. `Fail` rejects the pod, so a webhook outage blocks GPU pod creation in opted-in namespaces. `Ignore` admits the pod without preflight checks, trading the gate for availability.

### timeoutSeconds
How long the API server waits for the webhook, up to the Kubernetes maximum of 30. The default is `10`. Preflight resolves gang membership during admission, so a cluster with slow PodGroup lookups may need more; every GPU pod creation waits on this call, so raise it only as far as the lookups need.

### createIssuer
Creates the two-tier cert-manager CA for the webhook certificate. Set it to `false` to use a CA you already run, and then set `certIssuer` and `caCertificateName`.

### certIssuer
Name of the existing cert-manager issuer to sign the webhook certificate. Used only when `createIssuer` is `false`.

### caCertificateName
Name of the existing CA certificate whose bundle is injected into the webhook configuration. Used only when `createIssuer` is `false`. Without it the API server cannot verify the webhook, and admission fails closed under `failurePolicy: Fail`.

### certProvider
`cert-manager` by default. Set it to `openshift-service-ca` on OpenShift to issue the webhook certificate with the built-in service-ca-operator instead of cert-manager.

## Object selector (pod-level filtering)

By default the webhook intercepts all GPU pods in labeled namespaces. To further restrict which pods are intercepted, set `objectSelector` with standard Kubernetes label selectors. When empty (`{}`), no `objectSelector` is emitted and all pods in matching namespaces are intercepted.

Example — only intercept pods explicitly labeled for preflight:

```yaml
objectSelector:
  matchLabels:
    nvsentinel.nvidia.com/preflight: "enabled"
```

`matchExpressions` are also supported:

```yaml
objectSelector:
  matchExpressions:
    - key: nvsentinel.nvidia.com/preflight
      operator: In
      values: ["enabled", "true"]
```

This is useful when you want namespace-wide opt-in via `namespaceSelector` but only run preflight on specific workloads within those namespaces.

Full defaults and comments: `distros/kubernetes/nvsentinel/charts/preflight/values.yaml`.

Tilt development often trims init containers to DCGM-only; see `distros/kubernetes/nvsentinel/values-tilt.yaml`.

## Observability

- Webhook pod: liveness/readiness probes use `/healthz` on the webhook port.
- Prometheus metric names for check containers and the injector are specified in [ADR-026 § Metrics](../designs/026-preflight-checks.md#metrics); wire scrapers to your init container images and deployment as your environment allows.

## Debugging preflight failures

When a preflight check fails, the pod stays in `Init:Error` and the init container exits non-zero.

### 1. Check the exit code

```bash
kubectl -n {namespace} get pod {pod-name} -o jsonpath=\
'{range .status.initContainerStatuses[*]}{.name}{"\t"}{.state.terminated.exitCode}{"\n"}{end}'
```

Each check has its own codes:

| Exit code | `preflight-dcgm-diag` | `preflight-nccl-loopback` | `preflight-nccl-allreduce` |
|---|---|---|---|
| `0` | Passed, or reported a non-fatal finding | Passed | Passed |
| `1` | Fatal test failure, DCGM unreachable, or a configuration error | Test failed: bandwidth below threshold, or the benchmark did not run | Test failed: bandwidth degraded, all-reduce timeout, or NCCL init failure |
| `2` | — | Configuration error | Gang configuration error (`NCCL_GANG_CONFIG_ERROR`) |
| `3` | — | Could not send the health event | Gang formation timed out (`NCCL_GANG_TIMEOUT`) |
| `4` | — | — | Could not send the health event |

The exit code decides whether the pod starts. The health event decides what happens to the node: Fault Quarantine cordons on preflight events with `isFatal: true` only. Hardware findings are fatal, and coordination or configuration problems are not — so a gang timeout holds the pod in `Init:Error` without cordoning the node.

A check that fails before it can report sends no event at all. A `preflight-dcgm-diag` configuration error (exit `1`) and a `preflight-nccl-loopback` configuration error (exit `2`) both block the pod and leave no event to find.

When `processingStrategy` is `STORE_ONLY`, a check reports its result and then exits `0`, logging `Check failed (STORE_ONLY — not blocking pod)`, so the event is recorded and the workload still starts. The gang stage of `preflight-nccl-allreduce` runs before the benchmark and returns exit `2` or `3` directly, so `STORE_ONLY` does not unblock a gang failure.

#### A check can exit `0` and still report an unhealthy GPU

`preflight-dcgm-diag` sends one health event for each DCGM test result. Three outcomes report a problem without failing the gate:

- A test result of `warn`.
- A test result of `fail` whose error code has no actionable remediation — `DCGM_FR_XID_ERROR`, for example.
- A `DCGM_ST_*` status that still prevents the diagnostic from completing after `DCGM_DIAG_STATUS_RETRY_MAX_ATTEMPTS` attempts.

Each one emits an unhealthy event with `isFatal: false`, and the check exits `0`, so a warning or an infrastructure problem does not block the workload. The node then carries a preflight health event, and a node condition, while staying schedulable. A non-fatal event can still carry a recommended action, so read the event rather than inferring the finding from the exit code.

### 2. Check health events

All three checks (`preflight-dcgm-diag`, `preflight-nccl-loopback`, `preflight-nccl-allreduce`) report results as health events. From the NVSentinel repository root, query them with the MongoDB shell helper:

```bash
./scripts/mongodb-shell.sh
```

Once connected, filter for the failing node (replace `{NODE_NAME}`):

```javascript
db.HealthEvents.find({"healthevent.nodename": "{NODE_NAME}"}).pretty()
```

See [Connecting to the Datastore](/nvsentinel/v1.26.0/runbooks/datastore-connection) for more query examples.

Init container logs are not always available (for example if the pod was already cleaned up). In that case, rely on the health events and node conditions instead:

```bash
kubectl describe node {node-name} | grep -A3 Conditions
```

### 3. Act on the failure

- **`preflight-dcgm-diag` failed** — the GPU/node is in a bad state. The node needs to be fixed (reboot or terminate).
- **`preflight-nccl-loopback` or `preflight-nccl-allreduce` failed** — first confirm the `BW_THRESHOLD_GBPS` is realistic for the hardware. The default (`150` for loopback, `100` for all-reduce) assumes NVLink; it is too high for PCIe-only interconnect or lower-bandwidth GPUs (for example, L40S sustains ~20 GB/s), where the check would never pass. If the threshold is set too high for your GPUs, lower it to the expected value for that interconnect. If the threshold is already correct for the hardware, do **not** lower it to force a pass — re-run the check once to rule out a false positive, and if it fails again, check the health event and take the needful action such as rebooting or terminating the node.

## Related documentation

- [ADR-026: Preflight checks](../designs/026-preflight-checks.md)
- [ADR-035: Inline DCGM config](../designs/035-preflight-inline-dcgm-config.md) — design rationale for inline env var configuration
- [gRPC / TLS authentication](../designs/030-grpc-tls-authentication.md) (mentions preflight among webhooks)
- [Helm chart README](../../distros/kubernetes/README.md)
- E2E test entry point: `tests/preflight_test.go` (build tag `amd64_group`)