> This page is for NVSentinel, version v1.26.0.
> For other versions, use one of these documentation indexes:
> - Latest · v1.28.0 (default): https://docs.nvidia.com/nvsentinel/v1.28.0/llms.txt
> - v1.27.0: https://docs.nvidia.com/nvsentinel/v1.27.0/llms.txt
> - v1.26.0: https://docs.nvidia.com/nvsentinel/v1.26.0/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nvsentinel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nvsentinel/_mcp/server.

# Janitor Configuration

## Overview

The Janitor module watches for Kubernetes Custom Resources (`RebootNode`, `TerminateNode`, `GPUReset`) created by fault-remediation and carries out the actual node operations by calling the Janitor Provider over gRPC. This document covers all Helm configuration options for system administrators.

## Configuration Reference

### Module Enable/Disable

Controls whether the janitor module is deployed in the cluster.

```yaml
global:
  janitor:
    enabled: true
```

### Replica Count

```yaml
janitor:
  replicaCount: 1
```

The Janitor processes one CR at a time per controller. A single replica is sufficient for most deployments.

### Resources

Defines CPU and memory resource requests and limits for the janitor pod. No defaults are set; configure for production use.

```yaml
janitor:
  resources: {}
```

### Logging

Sets the verbosity level for janitor logs. Inherited from `global.logLevel` when not overridden.

```yaml
janitor:
  logLevel: info  # Options: debug, info, warn, error
```

## Operation Timeout

Global default timeout applied to all node operations when no controller-specific timeout is set.

```yaml
janitor:
  config:
    timeout: "25m"
```

Choose a value that covers the slowest expected operation for your cloud provider. AWS, Azure, and OCI node reboots typically complete within 25 minutes. GCP can be shorter. For `kind` clusters used in development, a much shorter value (e.g. `"5m"`) is sufficient.

## CSP Provider Connection

The Janitor connects to the Janitor Provider over gRPC. Configure the endpoint and connection security here.

```yaml
janitor:
  config:
    cspProviderHost: "janitor-provider.nvsentinel.svc.cluster.local:50051"
```

`cspProviderHost` is the gRPC address of the Janitor Provider. Change this only when the Janitor Provider is deployed in a different namespace or under a custom service name.

### TLS

```yaml
janitor:
  config:
    cspProvider:
      tls:
        enabled: true
        caSecretName: "janitor-provider-grpc-cert"
        insecure: false
```

### enabled
When `true`, the Janitor uses TLS for the gRPC connection to the Janitor Provider. Disable only in isolated test environments.

### caSecretName
Name of the Kubernetes Secret containing `ca.crt` used to verify the Janitor Provider's TLS certificate. The self-signed certificate is managed by cert-manager, which is a required dependency.

### insecure
Set to `true` to skip TLS certificate verification. For development use only; never enable in production.

### Service Account Token Auth

```yaml
janitor:
  config:
    cspProvider:
      auth:
        enabled: true
        audience: "nvsentinel-csp-provider"
        expirationSeconds: 3600
```

### enabled
When `true`, the Janitor mounts a projected ServiceAccount token and sends it to the Janitor Provider for authentication.

### audience
The token audience must match the `auth.audiences` value configured on the Janitor Provider.

### expirationSeconds
Requested lifetime of the projected ServiceAccount token in seconds. Kubernetes automatically rotates the token before expiry.

## Manual Mode

```yaml
janitor:
  config:
    manualMode: false
```

When `true`, the Janitor creates the Custom Resource but does not call the Janitor Provider to execute any node operation. Use this to test the fault-remediation → janitor CR creation pipeline without triggering actual node reboots, terminations, or GPU resets.

## HTTP Port

```yaml
janitor:
  config:
    httpPort: 8082
```

Port for the Janitor's internal HTTP server (health and readiness endpoints).

## Node Exclusions

Stops new `RebootNode`, `TerminateNode`, and `GPUReset` CRs from being created for the matching nodes.

```yaml
janitor:
  config:
    nodes:
      exclusions: []
```

Each entry is a Kubernetes [label selector](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors), not a node name. Use `matchLabels`, `matchExpressions`, or both. A node that matches **any** entry is excluded.

```yaml
janitor:
  config:
    nodes:
      exclusions:
        # Every control-plane node
        - matchLabels:
            node-role.kubernetes.io/control-plane: ""

        # One specific node, selected through its hostname label
        - matchLabels:
            kubernetes.io/hostname: gpu-node-07

        # Any node in a critical or production tier
        - matchExpressions:
            - key: workload-tier
              operator: In
              values:
                - critical
                - production
```

To exclude one node, select it through the `kubernetes.io/hostname` label that kubelet sets on every node. Read the value first, because it does not always match the node's object name:

```bash
kubectl get node gpu-node-07 -o jsonpath='{.metadata.labels.kubernetes\.io/hostname}'
```

One list covers all three controllers: RebootNode, TerminateNode, and GPUReset.

### How exclusions are enforced

A validating admission webhook compares the node's labels against every exclusion when a Janitor CR is created, and rejects the request on the first match:

```text
node 'control-plane-1' is excluded from janitor operations due to a label on the node matching the label exclusion 'node-role.kubernetes.io/control-plane=' from config value global.nodes.exclusions
```

The check runs server-side on the API request, so it applies to every client, and `force: true` does not bypass it. Because it matches on labels read at admission time, relabelling a node changes what is excluded without a Janitor restart.

Admission is the only place exclusions are enforced. The controllers do not re-check them while reconciling, so a CR admitted before you added the exclusion, or before you labelled the node, still runs to completion. After you add an exclusion, check for CRs that are already in flight for those nodes:

```bash
kubectl get rebootnodes,terminatenodes,gpuresets -A
```

Use this for control-plane nodes, infrastructure nodes, or any node that NVSentinel should not reboot or terminate.

## Controllers

Each controller handles one CR type. All three are enabled by default.

### RebootNode Controller

```yaml
janitor:
  config:
    controllers:
      rebootNode:
        enabled: true
        timeout: "25m"
```

Handles `RebootNode` CRs. `timeout` overrides `config.timeout` for reboot operations.

### TerminateNode Controller

```yaml
janitor:
  config:
    controllers:
      terminateNode:
        enabled: true
        timeout: "25m"
```

Handles `TerminateNode` CRs. `timeout` overrides `config.timeout` for terminate operations.

### GPUReset Controller

```yaml
janitor:
  config:
    controllers:
      gpuReset:
        enabled: true
        timeout: "25m"
        serviceManager:
          name: "gpu-operator"
        resetJob:
          writeSysLogEvent: true
          runtimeClassName: "nvidia"
          hostDriverRootPath: "/run/nvidia/driver"
          driverRoot: "/run/nvidia/driver"
          image:
            repository: ghcr.io/nvidia/nvsentinel/gpu-reset
            tag: ""
          resources:
            requests:
              cpu: "50m"
              memory: "64Mi"
            limits:
              cpu: "100m"
              memory: "128Mi"
```

Handles `GPUReset` CRs. Before issuing the GPU reset, the controller pauses the deployment or DaemonSet named by `serviceManager.name` to prevent the GPU Operator from interfering with the reset sequence.

### serviceManager.name
Name of the Kubernetes Deployment or DaemonSet to pause during GPU reset. Set to the GPU Operator deployment name in your cluster.

### resetJob.writeSysLogEvent
When `true`, the reset job writes a kernel syslog message on reset completion. Useful for correlating reset events with node-level logs.

### resetJob.runtimeClassName
NVIDIA RuntimeClass name used by the GPU reset Job. Must match a RuntimeClass installed in the cluster.

### resetJob.hostDriverRootPath
Host path containing the NVIDIA driver filesystem. The reset Job mounts it at `resetJob.driverRoot` inside the reset container.

Keep the default `/run/nvidia/driver` for containerized driver installations. Set it to `/` when the driver is installed directly into the host filesystem and `nvidia-smi` is available at a path such as `/usr/bin/nvidia-smi`.

### resetJob.driverRoot
Path inside the reset container that the reset command chroots into. The reset command runs `chroot <driverRoot> nvidia-smi` for every `nvidia-smi` call, so this path must hold a complete filesystem: the `nvidia-smi` binary, its shared libraries, and a dynamic loader. The reset Job also mounts the host `/sys` at `<driverRoot>/sys`, so the chroot can read sysfs.

Keep the default `/run/nvidia/driver`. The GPU Operator driver container bind-mounts its full container root to that host path, which satisfies the chroot. This default also works with `hostDriverRootPath: "/"`, because the host root filesystem is complete as well.

Set `driverRoot` to `/` when the driver root holds only driver files and not a complete filesystem. A chroot into a partial driver root fails before `nvidia-smi` starts. With `/`, the chroot is a no-op and the reset command uses the `nvidia-smi` in the reset container image. The Job then does not mount `hostDriverRootPath`, and it mounts the host `/sys` at `/sys`.

The value must be an absolute, clean path. The janitor rejects any other value at startup.

### resetJob.image
Container image for the GPU reset Job. Leave `tag` empty to use the chart default.

### resetJob.resources
Resource requests and limits for the GPU reset Job container.

## TTL-Based CR Cleanup

Completed CRs are automatically deleted after the TTL expires.

```yaml
janitor:
  ttl:
    enabled: true
    defaultTTL: "336h"
```

### enabled
When `true`, the TTL controller deletes completed `RebootNode`, `TerminateNode`, and `GPUReset` CRs after `defaultTTL` has elapsed since completion.

### defaultTTL
Duration after which completed CRs are deleted. Default is `336h` (14 days). Use a shorter value (e.g. `"24h"`) in test environments to keep the CR list clean.

## Webhook

The Janitor uses an admission webhook to validate CRs before they are persisted.

```yaml
janitor:
  webhook:
    port: 9443
    certIssuer: "janitor-selfsigned-issuer"
    certProvider: cert-manager
```

### certProvider
cert-manager is a required dependency. The webhook certificate is issued by the `certIssuer` ClusterIssuer and renewed automatically.

## Metrics TLS

Serves the Prometheus metrics endpoint over TLS, with a cert-manager issued certificate.

```yaml
janitor:
  metrics:
    tls:
      enabled: false
      certDir: "/tmp/k8s-metrics-server/metrics-certs"
      issuerName: ""
      issuerKind: "Issuer"
      issuerGroup: ""
      duration: "2160h"
      renewBefore: "720h"
      organization: "NVIDIA"
```

### enabled
Turns on TLS for the metrics endpoint. Off by default. Your scrape configuration must then trust the issuing CA, or Prometheus stops collecting janitor metrics.

### certDir
Directory where the metrics server reads its certificate and key.

### issuerName
cert-manager issuer that signs the certificate. Falls back to `webhook.certIssuer` when empty.

### issuerKind
`Issuer` for a namespaced issuer, or `ClusterIssuer` for a cluster-scoped one. Defaults to `Issuer`, so set it to `ClusterIssuer` when `issuerName` names one.

### issuerGroup
API group of the issuer. Leave empty for cert-manager's own issuers; set it only for an external issuer implementation.

### duration
Certificate lifetime. Defaults to `2160h`, which is 90 days.

### renewBefore
How long before expiry cert-manager renews the certificate. Defaults to `720h`, which is 30 days. It must stay shorter than `duration`.

### organization
Organization name in the certificate subject.

## Inactive Values

The janitor chart carries `autoscaling`, `livenessProbe`, `readinessProbe`, `podSecurityContext` and `securityContext` in `values.yaml`, but no template reads them. They are Helm scaffold left from the chart's creation:

- `autoscaling` renders no HorizontalPodAutoscaler. The Deployment always uses `replicaCount`.
- `livenessProbe` and `readinessProbe` add no probes to the pod.
- `podSecurityContext` and `securityContext` are not applied. The only pod security context the chart sets is `fsGroup: 65532`, added automatically when `global.auditLogging.enabled` is true.

Setting any of them changes nothing. Do not rely on them.