> This page is for NVSentinel, version v1.30.0. > For other versions, use one of these documentation indexes: > - Latest · v1.31.0 (default): https://docs.nvidia.com/nvsentinel/v1.31.0/llms.txt > - v1.30.0: https://docs.nvidia.com/nvsentinel/v1.30.0/llms.txt > - v1.29.0: https://docs.nvidia.com/nvsentinel/v1.29.0/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nvsentinel/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nvsentinel/_mcp/server. # Janitor Configuration ## Overview The Janitor module watches for Kubernetes Custom Resources (`RebootNode`, `TerminateNode`, `GPUReset`) created by fault-remediation and carries out the actual node operations by calling the Janitor Provider over gRPC. This document covers all Helm configuration options for system administrators. ## Configuration Reference ### Module Enable/Disable Controls whether the janitor module is deployed in the cluster. ```yaml global: janitor: enabled: true ``` ### Replica Count ```yaml janitor: replicaCount: 1 ``` The Janitor processes one CR at a time per controller. A single replica is sufficient for most deployments. ### Resources Defines CPU and memory resource requests and limits for the janitor pod. No defaults are set; configure for production use. ```yaml janitor: resources: {} ``` ### Logging Sets the verbosity level for janitor logs. ```yaml janitor: logLevel: info # Options: debug, info, warn, error ``` ## Operation Timeout Global default timeout applied to all node operations when no controller-specific timeout is set. ```yaml janitor: config: timeout: "25m" ``` Choose a value that covers the slowest expected operation for your cloud provider. AWS, Azure, and OCI node reboots typically complete within 25 minutes. GCP can be shorter. For `kind` clusters used in development, a much shorter value (e.g. `"5m"`) is sufficient. ## CSP Provider Connection The Janitor connects to the Janitor Provider over gRPC. Configure the endpoint and connection security here. ```yaml janitor: config: cspProviderHost: "janitor-provider.nvsentinel.svc.cluster.local:50051" ``` `cspProviderHost` is the gRPC address of the Janitor Provider. Change this only when the Janitor Provider is deployed in a different namespace or under a custom service name. ### TLS ```yaml janitor: config: cspProvider: tls: enabled: true caSecretName: "janitor-provider-grpc-cert" insecure: false ``` ### enabled When `true`, the Janitor uses TLS for the gRPC connection to the Janitor Provider. Disable only in isolated test environments. ### caSecretName Name of the Kubernetes Secret containing `ca.crt` used to verify the Janitor Provider's TLS certificate. The self-signed certificate is managed by cert-manager, which is a required dependency. ### insecure Set to `true` to skip TLS certificate verification. For development use only; never enable in production. ### Service Account Token Auth ```yaml janitor: config: cspProvider: auth: enabled: true audience: "nvsentinel-csp-provider" expirationSeconds: 3600 ``` ### enabled When `true`, the Janitor mounts a projected ServiceAccount token and sends it to the Janitor Provider for authentication. ### audience The token audience must match the `auth.audiences` value configured on the Janitor Provider. ### expirationSeconds Requested lifetime of the projected ServiceAccount token in seconds. Kubernetes automatically rotates the token before expiry. ## Manual Mode ```yaml janitor: config: manualMode: false ``` When `true`, the Janitor creates the Custom Resource but does not call the Janitor Provider to execute any node operation. Use this to test the fault-remediation → janitor CR creation pipeline without triggering actual node reboots, terminations, or GPU resets. ## HTTP Port ```yaml janitor: config: httpPort: 8082 ``` Port for the Janitor's internal HTTP server (health and readiness endpoints). ## Node Exclusions Stops new `RebootNode`, `TerminateNode`, and `GPUReset` CRs from being created for the matching nodes. ```yaml janitor: config: nodes: exclusions: [] ``` Each entry is a Kubernetes [label selector](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors), not a node name. Use `matchLabels`, `matchExpressions`, or both. A node that matches **any** entry is excluded. ```yaml janitor: config: nodes: exclusions: # Every control-plane node - matchLabels: node-role.kubernetes.io/control-plane: "" # One specific node, selected through its hostname label - matchLabels: kubernetes.io/hostname: gpu-node-07 # Any node in a critical or production tier - matchExpressions: - key: workload-tier operator: In values: - critical - production ``` To exclude one node, select it through the `kubernetes.io/hostname` label that kubelet sets on every node. Read the value first, because it does not always match the node's object name: ```bash kubectl get node gpu-node-07 -o jsonpath='{.metadata.labels.kubernetes\.io/hostname}' ``` One list covers all three controllers: RebootNode, TerminateNode, and GPUReset. ### How exclusions are enforced A validating admission webhook compares the node's labels against every exclusion when a Janitor CR is created, and rejects the request on the first match: ```text node 'control-plane-1' is excluded from janitor operations due to a label on the node matching the label exclusion 'node-role.kubernetes.io/control-plane=' from config value global.nodes.exclusions ``` The check runs server-side on the API request, so it applies to every client, and `force: true` does not bypass it. Because it matches on labels read at admission time, relabelling a node changes what is excluded without a Janitor restart. Admission is the only place exclusions are enforced. The controllers do not re-check them while reconciling, so a CR admitted before you added the exclusion, or before you labelled the node, still runs to completion. After you add an exclusion, check for CRs that are already in flight for those nodes: ```bash kubectl get rebootnodes,terminatenodes,gpuresets -A ``` Use this for control-plane nodes, infrastructure nodes, or any node that NVSentinel should not reboot or terminate. ## Controllers Each controller handles one CR type. All three are enabled by default. ### RebootNode Controller ```yaml janitor: config: controllers: rebootNode: enabled: true timeout: "25m" ``` Handles `RebootNode` CRs. `timeout` overrides `config.timeout` for reboot operations. ### TerminateNode Controller ```yaml janitor: config: controllers: terminateNode: enabled: true timeout: "25m" ``` Handles `TerminateNode` CRs. `timeout` overrides `config.timeout` for terminate operations. ### GPUReset Controller ```yaml janitor: config: controllers: gpuReset: enabled: true timeout: "25m" serviceManager: name: "gpu-operator" resetJob: writeSysLogEvent: true runtimeClassName: "nvidia" hostDriverRootPath: "/run/nvidia/driver" driverRoot: "/run/nvidia/driver" driverBinPath: "" image: repository: ghcr.io/nvidia/nvsentinel/gpu-reset tag: "" resources: requests: cpu: "50m" memory: "64Mi" limits: cpu: "100m" memory: "128Mi" ``` Handles `GPUReset` CRs. Before issuing the GPU reset, the controller removes the GPU Operator operands that hold device handles from the node by setting their `nvidia.com/gpu.deploy.*` node labels to `false`, waits for the pods to leave, runs the reset Job, then sets the labels back to `true` and waits for the pods to be Ready. ### serviceManager.name Name of the built-in GPU services manager. Use `gpu-operator` for the NVIDIA GPU Operator. #### GPU Operator modes The `gpu-operator` service manager supports both GPU Operator modes: device plugin and DRA. You do not set the mode. The controller finds the GPU Operator operands that run on each node. Each managed operand has a deploy node label. To stop an operand on the node, the controller sets its label to `false`. After the reset, the controller sets the label back to `true` and waits until the operand pods are ready. | GPU Operator mode | Pod labels | Node label | |---|---|---| | device plugin | `app=nvidia-device-plugin-daemonset`, `app.kubernetes.io/managed-by=gpu-operator` | `nvidia.com/gpu.deploy.device-plugin` | | device plugin | `app=nvidia-dcgm`, `app.kubernetes.io/managed-by=gpu-operator` | `nvidia.com/gpu.deploy.dcgm` | | device plugin | `app=nvidia-dcgm-exporter`, `app.kubernetes.io/managed-by=gpu-operator` | `nvidia.com/gpu.deploy.dcgm-exporter` | | device plugin | `app=gpu-feature-discovery`, `app.kubernetes.io/managed-by=gpu-operator` | `nvidia.com/gpu.deploy.gpu-feature-discovery` | | DRA | `app=nvidia-dcgm-dra` | `nvidia.com/gpu.deploy.dcgm-dra` | | DRA | `app=nvidia-dcgm-exporter-dra` | `nvidia.com/gpu.deploy.dcgm-exporter-dra` | When the teardown starts, the controller lists the pods of each operand on the node. A pod in any phase counts. The controller stops only the operands that have a pod on the node. It records their node labels in the GPUReset status, in `status.managedServices.nodeLabels`. The restore and the GPUReset finalizer use this record. The controller does not change the node labels of the other operands. The controller changes a node label only when it holds the enabled or the disabled value. A label that is absent is not held: the teardown sets it to the disabled value and the restore to the enabled value. Another controller can hold a label at a different value. For example, the GPU Operator upgrade controller sets `nvidia.com/gpu.deploy.dcgm=paused-for-driver-upgrade` during a driver upgrade. The controller then waits and retries. Before the teardown starts, it checks the labels of every operand of the service manager, also the operands that have no pod on the node, because a driver upgrade removes the operand pods. This wait is bounded by the `gpuReset.timeout` value. The same check runs before the controller creates the reset Job. During the teardown, the wait is bounded by the service manager `teardownTimeout`, and during the restore by `restoreTimeout`. When the wait times out, the GPUReset fails with the reason `ServiceTeardownTimeoutExceeded`, `ResetJobCreationFailed` or `RestoreTimeoutExceeded`. A GPUReset that fails after it stopped services, for example on the restore timeout or on a held label before the reset Job, keeps the node lock and keeps restoring the node labels, without waiting for the pods, for one more `gpuReset.timeout`. Only the controller writes those labels back; the GPU Operator does not touch a label that holds the disabled value. A label that stays held for that long is left as it is, the other labels are restored, the `ServicesRestored` condition names the held one, and the lock is released. Deletion waits in the same way. When no operand pod runs on the node, the controller skips the teardown and the restore. The `ServicesTornDown` and `ServicesRestored` conditions then have the reason `Skipped`. A GPUReset that an earlier janitor version started has no record. For such a GPUReset, the controller treats every operand whose node label currently holds its disabled value as stopped, and restores those. Before the controller creates the reset Job, it makes sure that no pod of a stopped operand runs on the node. The janitor Pod cache holds only the pods in the service manager namespace that have one of the `app` labels of the service manager. The controller finds the operands only one time, when the teardown starts. An operand pod that is not on the node at that time is not stopped. For example, a DaemonSet rolling update can remove the pod for a short time. The pod can then start again during the reset and keep the GPU open, and the reset fails. #### Values per GPU Operator mode The built-in `gpu-operator` service manager and the `/run/nvidia/driver` driver paths work unchanged on ClusterPolicy and `GPUCluster` (DRA) clusters. The one value that differs is `resetJob.runtimeClassName`, which must be `""` together with `global.gpuDraEnabled: true`. Set `serviceManager.spec` only for a GPU services manager other than the GPU Operator, or to stop a GPU Operator operand that the built-in entry does not cover, as in the [host-installed driver example](#example-host-installed-driver-on-gke). The spec replaces the built-in entry completely, including its mode detection, so list every operand to stop. ### resetJob.writeSysLogEvent When `true`, the reset job writes a kernel syslog message on reset completion. Useful for correlating reset events with node-level logs. ### resetJob.runtimeClassName RuntimeClass name for the GPU reset Job. Defaults to `nvidia`. GPU Operator `GPUCluster` (DRA) mode creates no `nvidia` RuntimeClass, so with `global.gpuDraEnabled: true` this value must be `""`; the chart refuses to render otherwise. The Job is privileged and reaches the driver through `resetJob.hostDriverRootPath`, so it does not need a RuntimeClass in either mode; on ClusterPolicy clusters the default is kept for compatibility. Also set it to `""` on a ClusterPolicy cluster that has no `nvidia` RuntimeClass, for example with the GPU Operator NRI plugin enabled. ### resetJob.hostDriverRootPath Host path containing the NVIDIA driver filesystem. The reset Job mounts it at `resetJob.driverRoot` inside the reset container. Keep the default `/run/nvidia/driver` for containerized driver installations. Set it to `/` when the driver is installed directly into the host filesystem and `nvidia-smi` is available at a path such as `/usr/bin/nvidia-smi`. ### resetJob.driverRoot Path inside the reset container that the reset command chroots into. The reset command runs `chroot nvidia-smi` for every `nvidia-smi` call, so this path must hold a complete filesystem: the `nvidia-smi` binary, its shared libraries, and a dynamic loader. The reset Job also mounts the host `/sys` at `/sys`, so the chroot can read sysfs. Keep the default `/run/nvidia/driver`. The GPU Operator driver container bind-mounts its full container root to that host path, which satisfies the chroot. This default also works with `hostDriverRootPath: "/"`, because the host root filesystem is complete as well. Set `driverRoot` to `/` when the driver root holds only driver files and not a complete filesystem. A chroot into a partial driver root fails before `nvidia-smi` starts. With `/`, the chroot is a no-op and the reset command uses the `nvidia-smi` in the reset container image. The Job then does not mount `hostDriverRootPath`, and it mounts the host `/sys` at `/sys`. The value must be an absolute, clean path. The janitor rejects any other value at startup. ### resetJob.driverBinPath Directory inside `resetJob.driverRoot` that holds `nvidia-smi` and `nvidia-bug-report.sh`. The reset command adds it to the front of `PATH` for every command it runs in the chroot. Leave it empty when these files are on the default `PATH`, for example with the GPU Operator driver container. The value is empty by default. A value that is set must be an absolute, clean path without `:`. The janitor rejects any other value at startup. #### Example: host-installed driver on GKE The GKE driver installer on Container-Optimized OS (COS) nodes puts the driver in `/home/kubernetes/bin/nvidia` on the host, and no GPU Operator driver container publishes `/run/nvidia/driver`. The GPU Operator then runs with `driver.enabled: false` and its container toolkit installed in the same directory. This setup needs three changes: - Chroot into the host root, and point the reset command at the driver binaries with `driverBinPath`. The installer adds `/home/kubernetes/bin/nvidia/lib64` to the host's dynamic loader configuration, so the chroot finds the driver libraries without further settings. - Stop the GPU Operator container toolkit during the reset. In this setup, the toolkit process keeps every GPU device open, and the reset fails with `In use by another client`. The built-in `gpu-operator` entry does not stop the toolkit, so set `serviceManager.spec` with the device plugin mode operands and the toolkit. - Set `runtimeClassName` to `""`. When the toolkit stops, it removes the `nvidia` runtime from containerd, so a reset Job that requests that RuntimeClass cannot start. ```yaml janitor: config: controllers: gpuReset: serviceManager: name: "gpu-operator" spec: namespace: "gpu-operator" teardownTimeout: "5m" restoreTimeout: "10m" apps: - appSelector: {app: "nvidia-device-plugin-daemonset", app.kubernetes.io/managed-by: "gpu-operator"} nodeLabel: "nvidia.com/gpu.deploy.device-plugin" enabledValue: "true" disabledValue: "false" - appSelector: {app: "nvidia-dcgm", app.kubernetes.io/managed-by: "gpu-operator"} nodeLabel: "nvidia.com/gpu.deploy.dcgm" enabledValue: "true" disabledValue: "false" - appSelector: {app: "nvidia-dcgm-exporter", app.kubernetes.io/managed-by: "gpu-operator"} nodeLabel: "nvidia.com/gpu.deploy.dcgm-exporter" enabledValue: "true" disabledValue: "false" - appSelector: {app: "gpu-feature-discovery", app.kubernetes.io/managed-by: "gpu-operator"} nodeLabel: "nvidia.com/gpu.deploy.gpu-feature-discovery" enabledValue: "true" disabledValue: "false" - appSelector: {app: "nvidia-container-toolkit-daemonset", app.kubernetes.io/managed-by: "gpu-operator"} nodeLabel: "nvidia.com/gpu.deploy.container-toolkit" enabledValue: "true" disabledValue: "false" resetJob: runtimeClassName: "" hostDriverRootPath: "/" driverRoot: "/run/nvidia/driver" driverBinPath: "/home/kubernetes/bin/nvidia/bin" ``` The reset fails while any process holds the GPU open. If another GPU client runs on the node, for example a DRA kubelet plugin, add its operand to the spec as well. ### resetJob.image Container image for the GPU reset Job. Leave `tag` empty to use the chart default. ### resetJob.resources Resource requests and limits for the GPU reset Job container. ## TTL-Based CR Cleanup Completed CRs are automatically deleted after the TTL expires. ```yaml janitor: ttl: enabled: true defaultTTL: "336h" ``` ### enabled When `true`, the TTL controller deletes completed `RebootNode`, `TerminateNode`, and `GPUReset` CRs after `defaultTTL` has elapsed since completion. ### defaultTTL Duration after which completed CRs are deleted. Default is `336h` (14 days). Use a shorter value (e.g. `"24h"`) in test environments to keep the CR list clean. ## Webhook The Janitor uses an admission webhook to validate CRs before they are persisted. ```yaml janitor: webhook: port: 9443 certIssuer: "janitor-selfsigned-issuer" certProvider: cert-manager ``` ### certProvider cert-manager is a required dependency. The webhook certificate is issued by the `certIssuer` ClusterIssuer and renewed automatically. ## Metrics TLS Serves the Prometheus metrics endpoint over TLS, with a cert-manager issued certificate. ```yaml janitor: metrics: tls: enabled: false certDir: "/tmp/k8s-metrics-server/metrics-certs" issuerName: "" issuerKind: "Issuer" issuerGroup: "" duration: "2160h" renewBefore: "720h" organization: "NVIDIA" ``` ### enabled Turns on TLS for the metrics endpoint. Off by default. Your scrape configuration must then trust the issuing CA, or Prometheus stops collecting janitor metrics. ### certDir Directory where the metrics server reads its certificate and key. ### issuerName cert-manager issuer that signs the certificate. Falls back to `webhook.certIssuer` when empty. ### issuerKind `Issuer` for a namespaced issuer, or `ClusterIssuer` for a cluster-scoped one. Defaults to `Issuer`, so set it to `ClusterIssuer` when `issuerName` names one. ### issuerGroup API group of the issuer. Leave empty for cert-manager's own issuers; set it only for an external issuer implementation. ### duration Certificate lifetime. Defaults to `2160h`, which is 90 days. ### renewBefore How long before expiry cert-manager renews the certificate. Defaults to `720h`, which is 30 days. It must stay shorter than `duration`. ### organization Organization name in the certificate subject. ## Inactive Values The janitor chart carries `autoscaling`, `livenessProbe`, `readinessProbe`, `podSecurityContext` and `securityContext` in `values.yaml`, but no template reads them. They are Helm scaffold left from the chart's creation: - `autoscaling` renders no HorizontalPodAutoscaler. The Deployment always uses `replicaCount`. - `livenessProbe` and `readinessProbe` add no probes to the pod. - `podSecurityContext` and `securityContext` are not applied. The only pod security context the chart sets is `fsGroup: 65532`, added automatically when `global.auditLogging.enabled` is true. Setting any of them changes nothing. Do not rely on them.