Janitor Configuration

Overview

The Janitor module watches for Kubernetes Custom Resources (RebootNode, TerminateNode, GPUReset) created by fault-remediation and carries out the actual node operations by calling the Janitor Provider over gRPC. This document covers all Helm configuration options for system administrators.

Configuration Reference

Module Enable/Disable

Controls whether the janitor module is deployed in the cluster.

global:
  janitor:
    enabled: true

Replica Count

janitor:
  replicaCount: 1

The Janitor processes one CR at a time per controller. A single replica is sufficient for most deployments.

Resources

Defines CPU and memory resource requests and limits for the janitor pod. No defaults are set; configure for production use.

janitor:
  resources: {}

Logging

Sets the verbosity level for janitor logs. Inherited from global.logLevel when not overridden.

janitor:
  logLevel: info  # Options: debug, info, warn, error

Operation Timeout

Global default timeout applied to all node operations when no controller-specific timeout is set.

janitor:
  config:
    timeout: "25m"

Choose a value that covers the slowest expected operation for your cloud provider. AWS, Azure, and OCI node reboots typically complete within 25 minutes. GCP can be shorter. For kind clusters used in development, a much shorter value (e.g. "5m") is sufficient.

CSP Provider Connection

The Janitor connects to the Janitor Provider over gRPC. Configure the endpoint and connection security here.

janitor:
  config:
    cspProviderHost: "janitor-provider.nvsentinel.svc.cluster.local:50051"

cspProviderHost is the gRPC address of the Janitor Provider. Change this only when the Janitor Provider is deployed in a different namespace or under a custom service name.

TLS

janitor:
  config:
    cspProvider:
      tls:
        enabled: true
        caSecretName: "janitor-provider-grpc-cert"
        insecure: false

enabled

When true, the Janitor uses TLS for the gRPC connection to the Janitor Provider. Disable only in isolated test environments.

caSecretName

Name of the Kubernetes Secret containing ca.crt used to verify the Janitor Provider’s TLS certificate. The self-signed certificate is managed by cert-manager, which is a required dependency.

insecure

Set to true to skip TLS certificate verification. For development use only; never enable in production.

Service Account Token Auth

janitor:
  config:
    cspProvider:
      auth:
        enabled: true
        audience: "nvsentinel-csp-provider"
        expirationSeconds: 3600

enabled

When true, the Janitor mounts a projected ServiceAccount token and sends it to the Janitor Provider for authentication.

audience

The token audience must match the auth.audiences value configured on the Janitor Provider.

expirationSeconds

Requested lifetime of the projected ServiceAccount token in seconds. Kubernetes automatically rotates the token before expiry.

Manual Mode

janitor:
  config:
    manualMode: false

When true, the Janitor creates the Custom Resource but does not call the Janitor Provider to execute any node operation. Use this to test the fault-remediation → janitor CR creation pipeline without triggering actual node reboots, terminations, or GPU resets.

HTTP Port

janitor:
  config:
    httpPort: 8082

Port for the Janitor’s internal HTTP server (health and readiness endpoints).

Node Exclusions

Stops new RebootNode, TerminateNode, and GPUReset CRs from being created for the matching nodes.

janitor:
  config:
    nodes:
      exclusions: []

Each entry is a Kubernetes label selector, not a node name. Use matchLabels, matchExpressions, or both. A node that matches any entry is excluded.

janitor:
  config:
    nodes:
      exclusions:
        # Every control-plane node
        - matchLabels:
            node-role.kubernetes.io/control-plane: ""
        # One specific node, selected through its hostname label
        - matchLabels:
            kubernetes.io/hostname: gpu-node-07
        # Any node in a critical or production tier
        - matchExpressions:
            - key: workload-tier
              operator: In
              values:
                - critical
                - production

To exclude one node, select it through the kubernetes.io/hostname label that kubelet sets on every node. Read the value first, because it does not always match the node’s object name:

kubectl get node gpu-node-07 -o jsonpath='{.metadata.labels.kubernetes\.io/hostname}'

One list covers all three controllers: RebootNode, TerminateNode, and GPUReset.

How exclusions are enforced

A validating admission webhook compares the node’s labels against every exclusion when a Janitor CR is created, and rejects the request on the first match:

node 'control-plane-1' is excluded from janitor operations due to a label on the node matching the label exclusion 'node-role.kubernetes.io/control-plane=' from config value global.nodes.exclusions

The check runs server-side on the API request, so it applies to every client, and force: true does not bypass it. Because it matches on labels read at admission time, relabelling a node changes what is excluded without a Janitor restart.

Admission is the only place exclusions are enforced. The controllers do not re-check them while reconciling, so a CR admitted before you added the exclusion, or before you labelled the node, still runs to completion. After you add an exclusion, check for CRs that are already in flight for those nodes:

kubectl get rebootnodes,terminatenodes,gpuresets -A

Use this for control-plane nodes, infrastructure nodes, or any node that NVSentinel should not reboot or terminate.

Controllers

Each controller handles one CR type. All three are enabled by default.

RebootNode Controller

janitor:
  config:
    controllers:
      rebootNode:
        enabled: true
        timeout: "25m"

Handles RebootNode CRs. timeout overrides config.timeout for reboot operations.

TerminateNode Controller

janitor:
  config:
    controllers:
      terminateNode:
        enabled: true
        timeout: "25m"

Handles TerminateNode CRs. timeout overrides config.timeout for terminate operations.

GPUReset Controller

janitor:
  config:
    controllers:
      gpuReset:
        enabled: true
        timeout: "25m"
        serviceManager:
          name: "gpu-operator"
        resetJob:
          writeSysLogEvent: true
          runtimeClassName: "nvidia"
          hostDriverRootPath: "/run/nvidia/driver"
          driverRoot: "/run/nvidia/driver"
          image:
            repository: ghcr.io/nvidia/nvsentinel/gpu-reset
            tag: ""
          resources:
            requests:
              cpu: "50m"
              memory: "64Mi"
            limits:
              cpu: "100m"
              memory: "128Mi"

Handles GPUReset CRs. Before issuing the GPU reset, the controller removes the GPU Operator operands that hold device handles from the node by setting their nvidia.com/gpu.deploy.* node labels to false, waits for the pods to leave, runs the reset Job, then sets the labels back to true and waits for the pods to be Ready.

serviceManager.name

Name of the built-in GPU services manager. Use gpu-operator for the NVIDIA GPU Operator.

GPU Operator modes

The gpu-operator service manager supports both GPU Operator modes: device plugin and DRA. You do not set the mode. The controller finds the GPU Operator operands that run on each node.

Each managed operand has a deploy node label. To stop an operand on the node, the controller sets its label to false. After the reset, the controller sets the label back to true and waits until the operand pods are ready.

When the teardown starts, the controller lists the pods of each operand on the node. A pod in any phase counts. The controller stops only the operands that have a pod on the node. It records their node labels in the GPUReset status, in status.managedServices.nodeLabels. The restore and the GPUReset finalizer use this record. The controller does not change the node labels of the other operands.

When no operand pod runs on the node, the controller skips the teardown and the restore. The ServicesTornDown and ServicesRestored conditions then have the reason Skipped.

A GPUReset that an earlier janitor version started has no record. For such a GPUReset, the controller treats every operand whose node label currently holds its disabled value as stopped, and restores those.

Before the controller creates the reset Job, it makes sure that no pod of a stopped operand runs on the node.

The janitor Pod cache holds only the pods in the service manager namespace that have one of the app labels of the service manager.

The controller finds the operands only one time, when the teardown starts. An operand pod that is not on the node at that time is not stopped. For example, a DaemonSet rolling update can remove the pod for a short time. The pod can then start again during the reset and keep the GPU open, and the reset fails.

Values per GPU Operator mode

The built-in gpu-operator service manager and the /run/nvidia/driver driver paths work unchanged on ClusterPolicy and GPUCluster (DRA) clusters. The one value that differs is resetJob.runtimeClassName, which must be "" together with global.gpuDraEnabled: true. Set serviceManager.spec only for a GPU services manager other than the GPU Operator; it replaces the built-in entry completely, including its mode detection.

resetJob.writeSysLogEvent

When true, the reset job writes a kernel syslog message on reset completion. Useful for correlating reset events with node-level logs.

resetJob.runtimeClassName

RuntimeClass name for the GPU reset Job. Defaults to nvidia. GPU Operator GPUCluster (DRA) mode creates no nvidia RuntimeClass, so with global.gpuDraEnabled: true this value must be ""; the chart refuses to render otherwise. The Job is privileged and reaches the driver through resetJob.hostDriverRootPath, so it does not need a RuntimeClass in either mode; on ClusterPolicy clusters the default is kept for compatibility. Also set it to "" on a ClusterPolicy cluster that has no nvidia RuntimeClass, for example with the GPU Operator NRI plugin enabled.

resetJob.hostDriverRootPath

Host path containing the NVIDIA driver filesystem. The reset Job mounts it at resetJob.driverRoot inside the reset container.

Keep the default /run/nvidia/driver for containerized driver installations. Set it to / when the driver is installed directly into the host filesystem and nvidia-smi is available at a path such as /usr/bin/nvidia-smi.

resetJob.driverRoot

Path inside the reset container that the reset command chroots into. The reset command runs chroot <driverRoot> nvidia-smi for every nvidia-smi call, so this path must hold a complete filesystem: the nvidia-smi binary, its shared libraries, and a dynamic loader. The reset Job also mounts the host /sys at <driverRoot>/sys, so the chroot can read sysfs.

Keep the default /run/nvidia/driver. The GPU Operator driver container bind-mounts its full container root to that host path, which satisfies the chroot. This default also works with hostDriverRootPath: "/", because the host root filesystem is complete as well.

Set driverRoot to / when the driver root holds only driver files and not a complete filesystem. A chroot into a partial driver root fails before nvidia-smi starts. With /, the chroot is a no-op and the reset command uses the nvidia-smi in the reset container image. The Job then does not mount hostDriverRootPath, and it mounts the host /sys at /sys.

The value must be an absolute, clean path. The janitor rejects any other value at startup.

resetJob.image

Container image for the GPU reset Job. Leave tag empty to use the chart default.

resetJob.resources

Resource requests and limits for the GPU reset Job container.

TTL-Based CR Cleanup

Completed CRs are automatically deleted after the TTL expires.

janitor:
  ttl:
    enabled: true
    defaultTTL: "336h"

enabled

When true, the TTL controller deletes completed RebootNode, TerminateNode, and GPUReset CRs after defaultTTL has elapsed since completion.

defaultTTL

Duration after which completed CRs are deleted. Default is 336h (14 days). Use a shorter value (e.g. "24h") in test environments to keep the CR list clean.

Webhook

The Janitor uses an admission webhook to validate CRs before they are persisted.

janitor:
  webhook:
    port: 9443
    certIssuer: "janitor-selfsigned-issuer"
    certProvider: cert-manager

certProvider

cert-manager is a required dependency. The webhook certificate is issued by the certIssuer ClusterIssuer and renewed automatically.

Metrics TLS

Serves the Prometheus metrics endpoint over TLS, with a cert-manager issued certificate.

janitor:
  metrics:
    tls:
      enabled: false
      certDir: "/tmp/k8s-metrics-server/metrics-certs"
      issuerName: ""
      issuerKind: "Issuer"
      issuerGroup: ""
      duration: "2160h"
      renewBefore: "720h"
      organization: "NVIDIA"

enabled

Turns on TLS for the metrics endpoint. Off by default. Your scrape configuration must then trust the issuing CA, or Prometheus stops collecting janitor metrics.

certDir

Directory where the metrics server reads its certificate and key.

issuerName

cert-manager issuer that signs the certificate. Falls back to webhook.certIssuer when empty.

issuerKind

Issuer for a namespaced issuer, or ClusterIssuer for a cluster-scoped one. Defaults to Issuer, so set it to ClusterIssuer when issuerName names one.

issuerGroup

API group of the issuer. Leave empty for cert-manager’s own issuers; set it only for an external issuer implementation.

duration

Certificate lifetime. Defaults to 2160h, which is 90 days.

renewBefore

How long before expiry cert-manager renews the certificate. Defaults to 720h, which is 30 days. It must stay shorter than duration.

organization

Organization name in the certificate subject.

Inactive Values

The janitor chart carries autoscaling, livenessProbe, readinessProbe, podSecurityContext and securityContext in values.yaml, but no template reads them. They are Helm scaffold left from the chart’s creation:

• autoscaling renders no HorizontalPodAutoscaler. The Deployment always uses replicaCount.
• livenessProbe and readinessProbe add no probes to the pod.
• podSecurityContext and securityContext are not applied. The only pod security context the chart sets is fsGroup: 65532, added automatically when global.auditLogging.enabled is true.

Setting any of them changes nothing. Do not rely on them.