Labeler Configuration

Overview

The Labeler module automatically applies labels to Kubernetes nodes based on GPU runtime components. It watches DCGM and driver pods deployed by GPU Operator and detects Kata Containers runtime. This document covers all Helm configuration options for system administrators.

Labels Applied

The labeler automatically manages these node labels:

Configuration Reference

Module Enable/Disable

Controls whether the labeler module is deployed in the cluster.

1
global:
2
  labeler:
3
    enabled: true

Resources

Defines CPU and memory resource requests and limits for the labeler pod.

1
labeler:
2
  resources:
3
    requests:
4
      cpu: 100m
5
      memory: 128Mi
6
    limits:
7
      cpu: 500m
8
      memory: 256Mi

Logging

Sets the verbosity level for labeler logs.

1
labeler:
2
  logLevel: info  # Options: debug, info, warn, error

DCGM Bootstrap Gating

Controls whether the DCGM pod must be ready before the DCGM version label is set for the first time on a node.

1
labeler:
2
  requireDCGMReadyForBootstrap: true

Kata Containers Detection

Configures detection of Kata Containers runtime on nodes.

1
labeler:
2
  kataLabelOverride: ""

Parameters

kataLabelOverride

Optional custom node label to check for Kata Containers detection, in addition to the default label.

Default Label: katacontainers.io/kata-runtime

When empty, only the default label is checked. When set, both default and custom labels are checked.

Truthy Values

The following label values (case-insensitive) are considered truthy for Kata detection:

• "true"
• "enabled"
• "1"
• "yes"

Any other value or missing label results in kata.enabled=false.

Expected Device Counts

Expected device-count labeling is disabled by default. When enabled, the labeler evaluates enabled classes and writes current/expected count labels only when the configured CEL expression returns a valid non-negative integer.

The Helm chart renders this values block into a TOML ConfigMap entry and mounts it into the labeler pod. Because expressions are compiled at startup, Helm also annotates the pod template with a checksum so changes to the ConfigMap roll the Deployment.

1
labeler:
2
  expectedDeviceCounts:
3
    enabled: true
4
    classes:
5
      - name: gpu
6
        enabled: true
7
        labels:
8
          current: nvsentinel.dgxc.nvidia.com/gpu.count.current
9
          expected: nvsentinel.dgxc.nvidia.com/gpu.count.expected
10
        groupingLabels:
11
          - node.kubernetes.io/instance-type
12
          - nvidia.com/gpu.product
13
        expectedCountOverrides:
14
          - matchLabels:
15
              nvidia.com/gpu.product: NVIDIA-GB200
16
            count: 8
17
        currentExpression: |
18
          int(node.metadata.labels['nvidia.com/gpu.count'])

The CEL context exposes:

• node: the Kubernetes Node object being reconciled.
• resourceSlices: ResourceSlice objects associated with the node.
• sum(list<int>): helper that returns the sum of a list of integers.

For classes without a matching override, the expected value is learned as the maximum current or existing expected count among nodes with the same configured grouping-label values. Learned expected counts can rise automatically, but do not fall automatically when a node reports fewer devices.

Kata Detection Examples

Example 1: Default Detection

1
labeler:
2
  kataLabelOverride: ""

Checks only katacontainers.io/kata-runtime label on nodes.

Example 2: Custom Kata Label

1
labeler:
2
  kataLabelOverride: "io.katacontainers.config.runtime.oci_runtime"

Checks both katacontainers.io/kata-runtime and io.katacontainers.config.runtime.oci_runtime. Kata is enabled if either label has a truthy value.

GPU Operator Integration

The labeler watches for specific pod labels to detect DCGM and driver status.

Expected Pod Labels

DCGM Pods:

1
metadata:
2
  labels:
3
    app: nvidia-dcgm

Driver Pods:

1
metadata:
2
  labels:
3
    app: nvidia-driver-daemonset

If your GPU Operator configures its operands with different labels, the labeler will not detect the components.