> This page is for NVSentinel, version v1.27.0. > For other versions, use one of these documentation indexes: > - Latest · v1.28.0 (default): https://docs.nvidia.com/nvsentinel/v1.28.0/llms.txt > - v1.27.0: https://docs.nvidia.com/nvsentinel/v1.27.0/llms.txt > - v1.26.0: https://docs.nvidia.com/nvsentinel/v1.26.0/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nvsentinel/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nvsentinel/_mcp/server. # Labeler Configuration ## Overview The Labeler module automatically applies labels to Kubernetes nodes based on GPU runtime components. It watches DCGM and driver pods deployed by GPU Operator and detects Kata Containers runtime. This document covers all Helm configuration options for system administrators. ## Labels Applied The labeler automatically manages these node labels: | Label | Values | Purpose | |-------|--------|---------| | `nvsentinel.dgxc.nvidia.com/dcgm.version` | `3.x`, `4.x` | DCGM major version detected from DCGM pods | | `nvsentinel.dgxc.nvidia.com/driver.installed` | `true`, `false` | NVIDIA driver pod status on node | | `nvsentinel.dgxc.nvidia.com/kata.enabled` | `true`, `false` | Kata Containers runtime presence | | `nvsentinel.dgxc.nvidia.com/gpu.count.current` | non-negative integer | Current GPU count from the configured class expression | | `nvsentinel.dgxc.nvidia.com/gpu.count.expected` | non-negative integer | Expected GPU count from override or learned hardware-class baseline | | `nvsentinel.dgxc.nvidia.com/nic.count.current` | non-negative integer | Current NIC count from the configured class expression | | `nvsentinel.dgxc.nvidia.com/nic.count.expected` | non-negative integer | Expected NIC count from override or learned hardware-class baseline | ## Configuration Reference ### Module Enable/Disable Controls whether the labeler module is deployed in the cluster. ```yaml global: labeler: enabled: true ``` ### Resources Defines CPU and memory resource requests and limits for the labeler pod. ```yaml labeler: resources: requests: cpu: 100m memory: 128Mi limits: cpu: 500m memory: 256Mi ``` ### Logging Sets the verbosity level for labeler logs. ```yaml labeler: logLevel: info # Options: debug, info, warn, error ``` ### Kubernetes API Rate Limits The labeler inherits the Kubernetes client limits from `global.qps` and `global.burst` (defaults: `5` and `10`). Set component values only when the labeler needs different limits: ```yaml labeler: qps: 40 burst: 80 ``` Positive `qps` values enable client-side throttling, `0` uses the client-go default, and a negative value disables client-side throttling. `burst` must be non-negative; `0` uses the client-go default. ## Pre-Installed Drivers Assumes NVIDIA drivers are installed directly on the host rather than via GPU Operator driver containers. When enabled, the labeler sets `nvsentinel.dgxc.nvidia.com/driver.installed=true` on all GPU nodes it manages (`nvidia.com/gpu.present=true`; nodes opted out with `nvsentinel.dgxc.nvidia.com/managed=false` — for example during external remediation — are excluded), skipping driver pod detection. ```yaml labeler: assumeDriverInstalled: false ``` ## DCGM Pod Discovery Comma-separated values of the `app` pod label used to find the DCGM pod on each node. The default matches both the classic GPU Operator DCGM DaemonSet and the GPUCluster (DRA) mode one, so the correct pod is found in either mode without per-cluster configuration. ```yaml labeler: dcgmAppLabel: "nvidia-dcgm,nvidia-dcgm-dra" ``` ## Driver Pod Discovery Pod label values used to find the driver pod on each node: the `app` label of the GPU Operator driver pod, and the `k8s-app` label of the GKE driver installer pod. The defaults match GPU Operator and GKE. ```yaml labeler: driverAppLabel: "nvidia-driver-daemonset" gkeInstallerAppLabel: "nvidia-driver-installer" ``` ## DCGM Bootstrap Gating Controls whether the DCGM pod must be ready before the DCGM version label is set for the first time on a node. ```yaml labeler: requireDCGMReadyForBootstrap: true ``` ## Kata Containers Detection Configures detection of Kata Containers runtime on nodes. ```yaml labeler: kataLabelOverride: "" ``` ### Parameters #### kataLabelOverride Optional custom node label to check for Kata Containers detection, in addition to the default label. **Default Label:** `katacontainers.io/kata-runtime` When empty, only the default label is checked. When set, both default and custom labels are checked. ### Truthy Values The following label values (case-insensitive) are considered truthy for Kata detection: - `"true"` - `"enabled"` - `"1"` - `"yes"` Any other value or missing label results in `kata.enabled=false`. ## Expected Device Counts Expected device-count labeling is disabled by default. When enabled, the labeler evaluates enabled classes and writes current/expected count labels only when the configured CEL expression returns a valid non-negative integer. The Helm chart renders this values block into a TOML ConfigMap entry and mounts it into the labeler pod. Because expressions are compiled at startup, Helm also annotates the pod template with a checksum so changes to the ConfigMap roll the Deployment. ```yaml labeler: expectedDeviceCounts: enabled: true classes: - name: gpu enabled: true labels: current: nvsentinel.dgxc.nvidia.com/gpu.count.current expected: nvsentinel.dgxc.nvidia.com/gpu.count.expected groupingLabels: - node.kubernetes.io/instance-type - nvidia.com/gpu.product expectedCountOverrides: - matchLabels: nvidia.com/gpu.product: NVIDIA-GB200 count: 8 currentExpression: | int(node.metadata.labels['nvidia.com/gpu.count']) ``` The CEL context exposes: - `node`: the cached projection of the Kubernetes Node being reconciled. - `resourceSlices`: ResourceSlice objects associated with the node. - `sum(list)`: helper that returns the sum of a list of integers. #### Node fields available to expressions To limit informer memory use, the Labeler does not cache complete Node objects. The following fields are always retained: - `metadata.name`, `metadata.uid`, and `metadata.resourceVersion` - all `metadata.labels` - the `nvsentinel.dgxc.nvidia.com/dcgm-bootstrap-completed` annotation, when present When expected device counts have at least one enabled class, the Labeler also retains `status.allocatable` and `status.capacity`. Device-count expressions that read Node data must use `node.metadata.labels`, `node.status.allocatable`, or `node.status.capacity`. All other Node fields are discarded before caching, including `spec`, other annotations, `status.conditions`, addresses, images, and node information. When expected device counts are disabled, all of `status` is discarded. Expressions that reference discarded fields are unsupported and receive only the field's empty or absent value. For classes without a matching override, the expected value is learned as the maximum current or existing expected count among nodes with the same configured grouping-label values. Learned expected counts can rise automatically, but do not fall automatically when a node reports fewer devices. ### Kata Detection Examples #### Example 1: Default Detection ```yaml labeler: kataLabelOverride: "" ``` Checks only `katacontainers.io/kata-runtime` label on nodes. #### Example 2: Custom Kata Label ```yaml labeler: kataLabelOverride: "io.katacontainers.config.runtime.oci_runtime" ``` Checks both `katacontainers.io/kata-runtime` and `io.katacontainers.config.runtime.oci_runtime`. Kata is enabled if either label has a truthy value. ## GPU Operator Integration The labeler watches for specific pod labels to detect DCGM and driver status. ### Expected Pod Labels **DCGM Pods:** ```yaml metadata: labels: app: nvidia-dcgm ``` **Driver Pods:** ```yaml metadata: labels: app: nvidia-driver-daemonset ``` If your GPU Operator configures its operands with different labels, the labeler will not detect the components.