Configuration

Node Drainer Configuration

View as Markdown

Overview

The Node Drainer module evacuates workloads from quarantined nodes, safely moving pods to healthy nodes. This document covers all Helm configuration options available for system administrators.

Configuration Reference

Module Enable/Disable

Controls whether the node-drainer module is deployed in the cluster.

global:
  nodeDrainer:
    enabled: true

Resources

Defines CPU and memory resource requests and limits for the node-drainer pod.

node-drainer:
  resources:
    limits:
      cpu: "200m"
      memory: "300Mi"
    requests:
      cpu: "200m"
      memory: "300Mi"

Logging

Sets the verbosity level for node-drainer logs.

node-drainer:
  logLevel: info  # Options: debug, info, warn, error

Kubernetes API Rate Limits

Node Drainer inherits the Kubernetes client limits from global.qps and global.burst (defaults: 5 and 10). Set component values only when Node Drainer needs different limits:

node-drainer:
  qps: 40
  burst: 80

Positive qps values enable client-side throttling, 0 uses the client-go default, and a negative value disables client-side throttling. burst must be non-negative; 0 uses the client-go default.

Note: This module depends on the results from fault-quarantine. It also depends on the datastore being enabled. Therefore, ensure the datastore and the other modules are also enabled.

Change Stream Resume Token

To make node-drainer skip accumulated events and start from the current stream head, scale it to zero, set its key in the shared resume-control ConfigMap to CREATE, then restore its replicas. Node-drainer deletes only its own resume token, records a cold-start cutoff timestamp, skips startup cold-start recovery for that run, and resets the key back to RESUME during startup. Future restarts still run cold-start recovery, but only for records newer than the recorded cutoff.

REPLICAS=$(kubectl -n nvsentinel get deployment node-drainer -o jsonpath='{.spec.replicas}')
kubectl -n nvsentinel scale deployment/node-drainer --replicas=0
kubectl -n nvsentinel rollout status deployment/node-drainer --timeout=180s
kubectl -n nvsentinel get configmap resume-control >/dev/null 2>&1 || \
  kubectl -n nvsentinel create configmap resume-control
kubectl -n nvsentinel patch configmap resume-control \
  --type merge \
  -p '{"data":{"node-drainer":"CREATE"}}'
kubectl -n nvsentinel scale deployment/node-drainer --replicas="${REPLICAS:-1}"
kubectl -n nvsentinel rollout status deployment/node-drainer --timeout=180s

Partial Drain

If enabled, the node-drainer will only drain pods which are leveraging the GPU_UUID impacted entity in COMPONENT_RESET HealthEvents. If disabled, the node-drainer will drain all eligible pods on the impacted node for the configured namespaces regardless of the remediation action. HealthEvents with the COMPONENT_RESET remediation action must include an impacted entity for the unhealthy GPU_UUID or else the drain will fail.

IMPORTANT: If this setting is enabled, the COMPONENT_RESET action in fault-remediation must map to a custom resource which takes action only against the GPU_UUID. If partial drain was enabled in node-drainer but fault-remediation mapped COMPONENT_RESET to a reboot action, pods which weren’t drained would be restarted as part of the reboot.

node-drainer:
  partialDrainEnabled: true

Eviction Timeout

Grace period in seconds applied to pod eviction requests in Immediate mode only.

node-drainer:
  evictionTimeoutInSeconds: "60"

This timeout is passed as the GracePeriodSeconds in the Kubernetes eviction API call. Only used for Immediate eviction mode. Other modes respect the pod’s configured terminationGracePeriodSeconds.

Requeue Backoff Base

Base duration for exponential backoff on drain requeues in Immediate mode.

node-drainer:
  requeueBackoffBase: 10s

When pods cannot be evicted immediately, the node drainer requeues the drain event. Retries use exponential backoff starting at requeueBackoffBase and double up to a two-minute maximum.

Configure this setting based on the workload pod termination time:

  • For zero-grace pods (terminationGracePeriodSeconds: 0), set a lower base such as 2s to decrease drain latency.
  • For workloads with standard grace periods (terminationGracePeriodSeconds: 30), pod termination dominates drain latency. A short retry interval increases redundant eviction calls.

System Namespaces

Regular expression pattern matching system namespaces that are skipped during node drain operations.

node-drainer:
  systemNamespaces: "^(nvsentinel|kube-system|gpu-operator|gmp-system|network-operator|skyhook)$"

Pods in namespaces matching this regex are not evicted during drain operations.

Delete After Timeout

Time in minutes from the health event creation after which pods will be force deleted if still running.

node-drainer:
  deleteAfterTimeoutMinutes: 60

Used with DeleteAfterTimeout eviction mode. When the timeout expires, remaining pods are force deleted regardless of their state.

Not Ready Timeout

Time in minutes after which a pod in NotReady state is skipped from eviction operations.

node-drainer:
  notReadyTimeoutMinutes: 5

When a pod has been in NotReady state for longer than this timeout, it is excluded from the list of pods to evict. This prevents attempting to evict pods that are already unhealthy and unlikely to respond to eviction requests.

GPU-Only Draining

If enabled, the node-drainer filters pod eviction to only target workloads that request GPU resources.

node-drainer:
  drainGPUPods: false

The node-drainer detects GPU resource requests through device annotations added to pods by the metadata-collector. Pods with device annotations are identified as GPU workloads and eligible for eviction.

Device annotations are added to pods requesting GPU resources by metadata-collector with the format:

annotations:
	  dgxc.nvidia.com/devices: '{"devices":{"nvidia.com/gpu":["GPU-123"]}}'

Behavior

  • When enabled (true): Only pods with GPU device annotations are evicted during drain operations
  • When disabled (false): All eligible pods in configured namespaces are evicted (default behavior)
  • Pods without GPU requests are preserved, maintaining critical infrastructure services

Pod Drain Policies

Use podDrainPolicies when workloads in the same namespace need different drain modes. Policies match pod labels using standard Kubernetes label selector syntax. The first matching policy wins. podDrainPolicies and userNamespaces are mutually exclusive; configuration with both non-empty is rejected. Set userNamespaces: [] to clear the Helm chart’s default namespace rule when enabling policies. Pods that match no policy are outside the drain scope and do not block completion.

node-drainer:
  userNamespaces: []
  podDrainPolicies:
    - name: finish-training
      namespace: training-*
      podSelector: "example.com/drain-mode=finish"
      mode: AllowCompletion
    - name: replaceable-workers
      podSelector: "example.com/drain-mode in (immediate,replaceable)"
      mode: Immediate
    - name: bounded-shutdown
      podSelector: "example.com/drain-mode=timeout"
      mode: DeleteAfterTimeout

Each policy has a unique, non-empty name, a non-empty podSelector, and a mode (Immediate, AllowCompletion, or DeleteAfterTimeout). namespace is an optional namespace-name glob; omitting it matches all user namespaces. Selectors support equality, set membership, and key existence. As in Kubernetes, key!=value and key notin (value) also match pods without that key; add a key-existence requirement if absent labels should not match.

Policy order takes precedence over mode: an early AllowCompletion match cannot be overridden by a later Immediate match. There is no namespace fallback in policy mode. Ensure the selectors cover every workload that should participate in draining. To retain namespace-based draining instead, leave podDrainPolicies empty and configure userNamespaces as before.

Policies narrow eligible workloads; system namespace exclusions, DaemonSet exclusions, GPU-only filtering and partial GPU drain scope still apply. DrainOverrides.Force changes the selected pods’ mode to Immediate without widening that scope. DrainOverrides.Skip retains its existing behavior. Custom drain configuration cannot be combined with pod drain policies.

Pod labels are read from the informer cache on each reconciliation and before eviction or timeout deletion. Changing a relevant label changes the policy on a subsequent observation. API deletion preconditions prevent a stale observation from deleting a relabelled or replaced pod. Configuration changes require restarting node-drainer. Only label keys referenced by policies are retained in its pod cache.

DeleteAfterTimeout continues to use deleteAfterTimeoutMinutes measured from the health event’s creation, including after a restart. Policies do not introduce a new timeout or automatically choose a mode based on workload kind; workload owners opt in through labels.

User Namespaces

Defines eviction behavior for user workloads based on namespace patterns. Cannot be combined with a non-empty podDrainPolicies list.

Configuration Structure

node-drainer:
  userNamespaces:
    - name: "namespace-pattern"
      mode: "EvictionMode"

Parameters

name

Namespace name or pattern to match. Use * to match all user namespaces not explicitly configured.

mode

Eviction strategy to apply for pods in matching namespaces. Valid values: Immediate, AllowCompletion, or DeleteAfterTimeout.

Eviction Modes

Immediate

Evicts pods with minimal grace period without waiting for completion.

Best for: Stateless applications that can be quickly restarted.

userNamespaces:
  - name: "web-frontend"
    mode: "Immediate"

AllowCompletion

Waits for pods to terminate gracefully, respecting their terminationGracePeriodSeconds.

Best for: Most workloads including stateful applications and services.

userNamespaces:
  - name: "production"
    mode: "AllowCompletion"

DeleteAfterTimeout

Waits up to deleteAfterTimeoutMinutes (from health event creation), then force deletes remaining pods.

Best for: Long-running training jobs that need time to checkpoint and save state.

userNamespaces:
  - name: "ml-training"
    mode: "DeleteAfterTimeout"

Configuration Examples

Example 1: Default Configuration for All Namespaces

userNamespaces:
  - name: "*"
    mode: "AllowCompletion"

Example 2: Different Modes for Different Workloads

userNamespaces:
  - name: "training"
    mode: "DeleteAfterTimeout"
  - name: "inference-*"
    mode: "Immediate"

Example 3: Multi-Tier Application

userNamespaces:
  - name: "frontend"
    mode: "Immediate"
  - name: "batch-jobs"
    mode: "DeleteAfterTimeout"
  - name: "database"
    mode: "AllowCompletion"
  - name: "*"
    mode: "AllowCompletion"

Observe a Drain

On a quarantined node with workloads still scheduled:

NODE={cordoned-node}
kubectl get pods -A --field-selector "spec.nodeName=$NODE" -w
kubectl get node "$NODE" -L dgxc.nvidia.com/nvsentinel-state
kubectl get events -n default \
  --field-selector "involvedObject.kind=Node,involvedObject.name=$NODE"

Node Drainer events are created in the default namespace (Type=Normal, Source=nvsentinel-node-drainer). Example output for a pod training-job-0 in namespace batch-jobs on node gpu-node-42:

Immediate

Pods are evicted via the Eviction API — no Node Drainer event is emitted.

# kubectl get pods -A --field-selector "spec.nodeName=gpu-node-42"
NAMESPACE   NAME              READY   STATUS        RESTARTS   AGE
frontend    inference-api-0   1/1     Terminating   0          2h
# (pod disappears within evictionTimeoutInSeconds)
# kubectl get node gpu-node-42 -L dgxc.nvidia.com/nvsentinel-state
NAME         STATUS                     ROLES    AGE   VERSION   NVSENTINEL-STATE
gpu-node-42  Ready,SchedulingDisabled   <none>   30d   v1.29.0   draining

AllowCompletion

# kubectl get pods -A --field-selector "spec.nodeName=gpu-node-42"
NAMESPACE   NAME              READY   STATUS    RESTARTS   AGE
database    postgres-0        1/1     Running   0          2h
# kubectl get events -n default --field-selector "involvedObject.name=gpu-node-42"
LAST SEEN   TYPE          REASON                  OBJECT           MESSAGE
2m          Normal        AwaitingPodCompletion   Node/gpu-node-42 Waiting for following pods to finish: [database/postgres-0]

DeleteAfterTimeout

# kubectl get pods -A --field-selector "spec.nodeName=gpu-node-42"
NAMESPACE   NAME              READY   STATUS    RESTARTS   AGE
batch-jobs  training-job-0    1/1     Running   0          2h
# kubectl get events -n default --field-selector "involvedObject.name=gpu-node-42"
LAST SEEN   TYPE          REASON                    OBJECT           MESSAGE
1m          Normal        WaitingBeforeForceDelete  Node/gpu-node-42 Waiting for following pods to finish: [training-job-0] in namespace: [batch-jobs] or they will be force deleted on: 2026-07-01 18:30:00 +0000 UTC

After the deadline, the pod is force-deleted and kubectl get pods shows it gone.

One-shot AI prompt (per-namespace drain modes)

Paste to an AI coding agent. Replace bracketed parts.

Create a Helm values overlay for NVSentinel node-drainer per-namespace drain modes.
Modes (exact strings): Immediate, AllowCompletion, DeleteAfterTimeout.
- Immediate: Eviction API; evictionTimeoutInSeconds as a quoted string (e.g. "60")
- AllowCompletion: wait for pod exit (terminationGracePeriodSeconds)
- DeleteAfterTimeout: force-delete after deleteAfterTimeoutMinutes from health-event createdAt
YAML under global.nodeDrainer and node-drainer:
  global.nodeDrainer.enabled: true
  node-drainer.evictionTimeoutInSeconds: "60"
  node-drainer.deleteAfterTimeoutMinutes: [minutes, default 60]
  node-drainer.systemNamespaces: "^(nvsentinel|kube-system|gpu-operator|gmp-system|network-operator|skyhook)$"
  node-drainer.userNamespaces (specific globs before "*"; omit customDrain):
    [ns1] → Immediate     (default: frontend)
    [ns2] → DeleteAfterTimeout (default: batch-jobs)
    [ns3] → AllowCompletion  (default: database)
    *     → AllowCompletion
Output only the values YAML. User applies it with their existing NVSentinel Helm release.