> This page is for NVSentinel, version v1.27.0.
> For other versions, use one of these documentation indexes:
> - Latest · v1.28.0 (default): https://docs.nvidia.com/nvsentinel/v1.28.0/llms.txt
> - v1.27.0: https://docs.nvidia.com/nvsentinel/v1.27.0/llms.txt
> - v1.26.0: https://docs.nvidia.com/nvsentinel/v1.26.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nvsentinel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nvsentinel/_mcp/server.

# Node Drainer Configuration

## Overview

The Node Drainer module evacuates workloads from quarantined nodes, safely moving pods to healthy nodes. This document covers all Helm configuration options available for system administrators.

## Configuration Reference

### Module Enable/Disable

Controls whether the node-drainer module is deployed in the cluster.

```yaml
global:
  nodeDrainer:
    enabled: true
```

### Resources

Defines CPU and memory resource requests and limits for the node-drainer pod.

```yaml
node-drainer:
  resources:
    limits:
      cpu: "200m"
      memory: "300Mi"
    requests:
      cpu: "200m"
      memory: "300Mi"
```

### Logging

Sets the verbosity level for node-drainer logs.

```yaml
node-drainer:
  logLevel: info  # Options: debug, info, warn, error
```

### Kubernetes API Rate Limits

Node Drainer inherits the Kubernetes client limits from `global.qps` and `global.burst` (defaults: `5` and `10`). Set component values only when Node Drainer needs different limits:

```yaml
node-drainer:
  qps: 40
  burst: 80
```

Positive `qps` values enable client-side throttling, `0` uses the client-go default, and a negative value disables client-side throttling. `burst` must be non-negative; `0` uses the client-go default.

> Note: This module depends on the results from fault-quarantine. It also depends on the datastore being enabled. Therefore, ensure the datastore and the other modules are also enabled.

### Change Stream Resume Token

To make node-drainer skip accumulated events and start from the current stream head, scale it to zero, set its key in the shared resume-control ConfigMap to `CREATE`, then restore its replicas. Node-drainer deletes only its own resume token, records a cold-start cutoff timestamp, skips startup cold-start recovery for that run, and resets the key back to `RESUME` during startup. Future restarts still run cold-start recovery, but only for records newer than the recorded cutoff.

```bash
REPLICAS=$(kubectl -n nvsentinel get deployment node-drainer -o jsonpath='{.spec.replicas}')
kubectl -n nvsentinel scale deployment/node-drainer --replicas=0
kubectl -n nvsentinel rollout status deployment/node-drainer --timeout=180s
kubectl -n nvsentinel get configmap resume-control >/dev/null 2>&1 || \
  kubectl -n nvsentinel create configmap resume-control
kubectl -n nvsentinel patch configmap resume-control \
  --type merge \
  -p '{"data":{"node-drainer":"CREATE"}}'
kubectl -n nvsentinel scale deployment/node-drainer --replicas="${REPLICAS:-1}"
kubectl -n nvsentinel rollout status deployment/node-drainer --timeout=180s
```

### Partial Drain

If enabled, the node-drainer will only drain pods which are leveraging the GPU_UUID impacted entity in COMPONENT_RESET HealthEvents. If disabled, the node-drainer will drain all eligible pods on the impacted node for the configured namespaces regardless of the remediation action. HealthEvents with the COMPONENT_RESET remediation action must include an impacted entity for the unhealthy GPU_UUID or else the drain will fail. 

A partial drain also fails if a pod requests GPUs but has no `dgxc.nvidia.com/devices` annotation, because the metadata-collector is then not working. A pod requests GPUs through `nvidia.com/gpu` or `nvidia.com/pgpu` container limits, or through a DRA ResourceClaim for the `gpu.nvidia.com` DeviceClass. The node-drainer reads ResourceClaims only for pods that have DRA claims and no device annotation.

IMPORTANT: If this setting is enabled, the COMPONENT_RESET action in fault-remediation must map to a custom resource which takes action only against the GPU_UUID. If partial drain was enabled in node-drainer but fault-remediation mapped COMPONENT_RESET to a reboot action, pods which weren't drained would be restarted as part of the reboot.
```yaml
node-drainer:
  partialDrainEnabled: true
```

Post-remediation validation in fault-quarantine needs to know whether an event was drained partially. `fault-quarantine.validation.partialDrainEnabled` defaults to `false`, the same as this setting. If you enable partial drain here, enable it there too so a partial drain never counts as a full drain, or set `global.partialDrainEnabled`, which overrides both subchart values:

```yaml
global:
  partialDrainEnabled: true
```

### Partial Drain Entity Metric

Registers `node_drainer_partial_drains_total{node, entity_type, entity_value}`.

```yaml
node-drainer:
  partialDrainEntityMetricEnabled: false
```

Off by default because `entity_value` holds a GPU UUID, which adds one time series per GPU in the fleet. The `drain_scope` label on `node_drainer_events_processed_total` already separates partial drains from full ones without that cardinality. Enable this only when you need per-GPU attribution and your Prometheus can carry the extra series.

### Eviction Timeout

Grace period in seconds applied to pod eviction requests in Immediate mode only.

```yaml
node-drainer:
  evictionTimeoutInSeconds: "60"
```

This timeout is passed as the `GracePeriodSeconds` in the Kubernetes eviction API call. Only used for `Immediate` eviction mode. Other modes respect the pod's configured `terminationGracePeriodSeconds`.

### Requeue Backoff Base

Base duration for exponential backoff on drain requeues in `Immediate` mode.

```yaml
node-drainer:
  requeueBackoffBase: 10s
```

When pods cannot be evicted immediately, the node drainer requeues the drain event. Retries use exponential backoff starting at `requeueBackoffBase` and double up to a two-minute maximum.

Configure this setting based on the workload pod termination time:
- For zero-grace pods (`terminationGracePeriodSeconds: 0`), set a lower base such as `2s` to decrease drain latency.
- For workloads with standard grace periods (`terminationGracePeriodSeconds: 30`), pod termination dominates drain latency. A short retry interval increases redundant eviction calls.

### System Namespaces

Regular expression pattern matching system namespaces that are skipped during node drain operations.

```yaml
node-drainer:
  systemNamespaces: "^(nvsentinel|kube-system|gpu-operator|gmp-system|network-operator|skyhook)$"
```

Pods in namespaces matching this regex are not evicted during drain operations.

### Delete After Timeout

Time in minutes from the health event creation after which pods will be force deleted if still running.

```yaml
node-drainer:
  deleteAfterTimeoutMinutes: 60
```

Used with `DeleteAfterTimeout` eviction mode. When the timeout expires, remaining pods are force deleted regardless of their state.

### Not Ready Timeout

Time in minutes after which a pod in NotReady state is skipped from eviction operations.

```yaml
node-drainer:
  notReadyTimeoutMinutes: 5
```

When a pod has been in NotReady state for longer than this timeout, it is excluded from the list of pods to evict. This prevents attempting to evict pods that are already unhealthy and unlikely to respond to eviction requests.

### GPU-Only Draining

If enabled, the node-drainer filters pod eviction to only target workloads that request GPU resources.

```yaml
node-drainer:
  drainGPUPods: false
```

The node-drainer detects GPU resource requests through device annotations added to pods by the metadata-collector. Pods with device annotations are identified as GPU workloads and eligible for eviction. A pod without the annotation is also a GPU workload if it requests GPUs through `nvidia.com/gpu` or `nvidia.com/pgpu` container limits, or through a DRA ResourceClaim for the `gpu.nvidia.com` DeviceClass. If the node-drainer cannot read a ResourceClaim, the drain stops and retries.

Device annotations are added to pods requesting GPU resources by metadata-collector with the format:
```yaml
annotations:
	  dgxc.nvidia.com/devices: '{"devices":{"nvidia.com/gpu":["GPU-123"]}}'
```

#### Behavior

- **When enabled (`true`)**: Only pods with GPU device annotations, or that request GPUs through container limits or DRA claims, are evicted during drain operations
- **When disabled (`false`)**: All eligible pods in configured namespaces are evicted (default behavior)
- Pods without GPU requests are preserved, maintaining critical infrastructure services

## Pod Drain Policies

Use `podDrainPolicies` when workloads in the same namespace need different drain modes. Policies match pod labels using standard [Kubernetes label selector syntax](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors). The first matching policy wins. `podDrainPolicies` and `userNamespaces` are mutually exclusive; configuration with both non-empty is rejected. Set `userNamespaces: []` to clear the Helm chart's default namespace rule when enabling policies. Pods that match no policy are outside the drain scope and do not block completion.

```yaml
node-drainer:
  userNamespaces: []
  podDrainPolicies:
    - name: finish-training
      namespace: training-*
      podSelector: "example.com/drain-mode=finish"
      mode: AllowCompletion
    - name: replaceable-workers
      podSelector: "example.com/drain-mode in (immediate,replaceable)"
      mode: Immediate
    - name: bounded-shutdown
      podSelector: "example.com/drain-mode=timeout"
      mode: DeleteAfterTimeout
```

Each policy has a unique, non-empty `name`, a non-empty `podSelector`, and a `mode` (`Immediate`, `AllowCompletion`, or `DeleteAfterTimeout`). `namespace` is an optional namespace-name glob; omitting it matches all user namespaces. Selectors support equality, set membership, and key existence. As in Kubernetes, `key!=value` and `key notin (value)` also match pods without that key; add a key-existence requirement if absent labels should not match.

Policy order takes precedence over mode: an early `AllowCompletion` match cannot be overridden by a later `Immediate` match. There is no namespace fallback in policy mode. Ensure the selectors cover every workload that should participate in draining. To retain namespace-based draining instead, leave `podDrainPolicies` empty and configure `userNamespaces` as before.

Policies narrow eligible workloads; system namespace exclusions, DaemonSet exclusions, GPU-only filtering and partial GPU drain scope still apply. `DrainOverrides.Force` changes the selected pods' mode to `Immediate` without widening that scope. `DrainOverrides.Skip` retains its existing behavior. Custom drain configuration cannot be combined with pod drain policies unless `customDrain.nodeSelector` scopes it to part of the cluster; see [Scoping custom drain to part of the cluster](#scoping-custom-drain-to-part-of-the-cluster).

Pod labels are read from the informer cache on each reconciliation and before eviction or timeout deletion. Changing a relevant label changes the policy on a subsequent observation. API deletion preconditions prevent a stale observation from deleting a relabelled or replaced pod. Configuration changes require restarting node-drainer. Only label keys referenced by policies are retained in its pod cache.

`DeleteAfterTimeout` continues to use `deleteAfterTimeoutMinutes` measured from the health event's creation, including after a restart. Policies do not introduce a new timeout or automatically choose a mode based on workload kind; workload owners opt in through labels.

## Scoping custom drain to part of the cluster

`customDrain.enabled: true` replaces node-drainer's own eviction with a
[drain plugin](/nvsentinel/v1.27.0/tutorials/writing-a-drain-plugin), for the whole cluster. Set
`customDrain.nodeSelector` when only some nodes are drained that way — a Slurm or LSF partition
alongside nodes running ordinary Kubernetes workloads, for example. Nodes matching the selector go
through the plugin; every other node keeps the built-in eviction path, so `userNamespaces` or
`podDrainPolicies` must stay configured for them. The selector uses standard
[Kubernetes label selector syntax](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors).

```yaml
node-drainer:
  customDrain:
    enabled: true
    nodeSelector: "nvsentinel.example.com/scheduler=slurm"
    # ... CR template and status condition settings
  userNamespaces:
    - name: "*"
      mode: "AllowCompletion"
```

An empty or omitted `nodeSelector` keeps the original behavior: every node goes through custom
drain, and configuring `userNamespaces` or `podDrainPolicies` alongside it is rejected at startup.

A `nodeSelector` with neither `userNamespaces` nor `podDrainPolicies` is rejected at startup as
well: the unmatched nodes would reach the built-in eviction path with no namespace and no policy to
act on, and be marked drained while their pods keep running. An invalid selector is also rejected at
startup.

The drain path is chosen per health event from the node's labels as the informer cache holds them,
so relabelling a node changes the path taken by subsequent events. A node the cache cannot resolve
is retried rather than assigned a path. Only the label keys the selector references are retained in
the node cache.

## User Namespaces

Defines eviction behavior for user workloads based on namespace patterns. Cannot be combined with a non-empty `podDrainPolicies` list.

### Configuration Structure

```yaml
node-drainer:
  userNamespaces:
    - name: "namespace-pattern"
      mode: "EvictionMode"
```

### Parameters

#### name
Namespace name or pattern to match. Use `*` to match all user namespaces not explicitly configured.

#### mode
Eviction strategy to apply for pods in matching namespaces. Valid values: `Immediate`, `AllowCompletion`, or `DeleteAfterTimeout`.

### Eviction Modes

#### Immediate

Evicts pods with minimal grace period without waiting for completion.

Best for: Stateless applications that can be quickly restarted.

```yaml
userNamespaces:
  - name: "web-frontend"
    mode: "Immediate"
```

#### AllowCompletion

Waits for pods to terminate gracefully, respecting their `terminationGracePeriodSeconds`.

Best for: Most workloads including stateful applications and services.

```yaml
userNamespaces:
  - name: "production"
    mode: "AllowCompletion"
```

#### DeleteAfterTimeout

Waits up to `deleteAfterTimeoutMinutes` (from health event creation), then force deletes remaining pods.

Best for: Long-running training jobs that need time to checkpoint and save state.

```yaml
userNamespaces:
  - name: "ml-training"
    mode: "DeleteAfterTimeout"
```

### Configuration Examples

#### Example 1: Default Configuration for All Namespaces

```yaml
userNamespaces:
  - name: "*"
    mode: "AllowCompletion"
```

#### Example 2: Different Modes for Different Workloads

```yaml
userNamespaces:
  - name: "training"
    mode: "DeleteAfterTimeout"
  - name: "inference-*"
    mode: "Immediate"
```

#### Example 3: Multi-Tier Application

```yaml
userNamespaces:
  - name: "frontend"
    mode: "Immediate"
  - name: "batch-jobs"
    mode: "DeleteAfterTimeout"
  - name: "database"
    mode: "AllowCompletion"
  - name: "*"
    mode: "AllowCompletion"
```

### Observe a Drain

On a quarantined node with workloads still scheduled:

```bash
NODE={cordoned-node}

kubectl get pods -A --field-selector "spec.nodeName=$NODE" -w
kubectl get node "$NODE" -L dgxc.nvidia.com/nvsentinel-state
kubectl get events -n default \
  --field-selector "involvedObject.kind=Node,involvedObject.name=$NODE"
```

Node Drainer events are created in the **`default`** namespace (`Type=Normal`,
`Source=nvsentinel-node-drainer`). Example output for a pod `training-job-0` in namespace
`batch-jobs` on node `gpu-node-42`:

#### `Immediate`

Pods are evicted via the Eviction API — no Node Drainer event is emitted.

```text
# kubectl get pods -A --field-selector "spec.nodeName=gpu-node-42"
NAMESPACE   NAME              READY   STATUS        RESTARTS   AGE
frontend    inference-api-0   1/1     Terminating   0          2h

# (pod disappears within evictionTimeoutInSeconds)

# kubectl get node gpu-node-42 -L dgxc.nvidia.com/nvsentinel-state
NAME         STATUS                     ROLES    AGE   VERSION   NVSENTINEL-STATE
gpu-node-42  Ready,SchedulingDisabled   <none>   30d   v1.29.0   draining
```

#### `AllowCompletion`

```text
# kubectl get pods -A --field-selector "spec.nodeName=gpu-node-42"
NAMESPACE   NAME              READY   STATUS    RESTARTS   AGE
database    postgres-0        1/1     Running   0          2h

# kubectl get events -n default --field-selector "involvedObject.name=gpu-node-42"
LAST SEEN   TYPE          REASON                  OBJECT           MESSAGE
2m          Normal        AwaitingPodCompletion   Node/gpu-node-42 Waiting for following pods to finish: [database/postgres-0]
```

#### `DeleteAfterTimeout`

```text
# kubectl get pods -A --field-selector "spec.nodeName=gpu-node-42"
NAMESPACE   NAME              READY   STATUS    RESTARTS   AGE
batch-jobs  training-job-0    1/1     Running   0          2h

# kubectl get events -n default --field-selector "involvedObject.name=gpu-node-42"
LAST SEEN   TYPE          REASON                    OBJECT           MESSAGE
1m          Normal        WaitingBeforeForceDelete  Node/gpu-node-42 Waiting for following pods to finish: [training-job-0] in namespace: [batch-jobs] or they will be force deleted on: 2026-07-01 18:30:00 +0000 UTC
```

After the deadline, the pod is force-deleted and `kubectl get pods` shows it gone.

### One-shot AI prompt (per-namespace drain modes)

Paste to an AI coding agent. Replace bracketed parts.

```text
Create a Helm values overlay for NVSentinel node-drainer per-namespace drain modes.

Modes (exact strings): Immediate, AllowCompletion, DeleteAfterTimeout.
- Immediate: Eviction API; evictionTimeoutInSeconds as a quoted string (e.g. "60")
- AllowCompletion: wait for pod exit (terminationGracePeriodSeconds)
- DeleteAfterTimeout: force-delete after deleteAfterTimeoutMinutes from health-event createdAt

YAML under global.nodeDrainer and node-drainer:
  global.nodeDrainer.enabled: true
  node-drainer.evictionTimeoutInSeconds: "60"
  node-drainer.deleteAfterTimeoutMinutes: [minutes, default 60]
  node-drainer.systemNamespaces: "^(nvsentinel|kube-system|gpu-operator|gmp-system|network-operator|skyhook)$"
  node-drainer.userNamespaces (specific globs before "*"; omit customDrain):
    [ns1] → Immediate     (default: frontend)
    [ns2] → DeleteAfterTimeout (default: batch-jobs)
    [ns3] → AllowCompletion  (default: database)
    *     → AllowCompletion

Output only the values YAML. User applies it with their existing NVSentinel Helm release.
```