Configuration

Janitor Configuration

View as Markdown

Overview

The Janitor module watches for Kubernetes Custom Resources (RebootNode, TerminateNode, GPUReset) created by fault-remediation and carries out the actual node operations by calling the Janitor Provider over gRPC. This document covers all Helm configuration options for system administrators.

Configuration Reference

Module Enable/Disable

Controls whether the janitor module is deployed in the cluster.

1global:
2  janitor:
3    enabled: true

Replica Count

1janitor:
2  replicaCount: 1

The Janitor processes one CR at a time per controller. A single replica is sufficient for most deployments.

Resources

Defines CPU and memory resource requests and limits for the janitor pod. No defaults are set; configure for production use.

1janitor:
2  resources: \{\}

Logging

Sets the verbosity level for janitor logs. Inherited from global.logLevel when not overridden.

1janitor:
2  logLevel: info  # Options: debug, info, warn, error

Operation Timeout

Global default timeout applied to all node operations when no controller-specific timeout is set.

1janitor:
2  config:
3    timeout: "25m"

Choose a value that covers the slowest expected operation for your cloud provider. AWS, Azure, and OCI node reboots typically complete within 25 minutes. GCP can be shorter. For kind clusters used in development, a much shorter value (e.g. "5m") is sufficient.

CSP Provider Connection

The Janitor connects to the Janitor Provider over gRPC. Configure the endpoint and connection security here.

1janitor:
2  config:
3    cspProviderHost: "janitor-provider.nvsentinel.svc.cluster.local:50051"

cspProviderHost is the gRPC address of the Janitor Provider. Change this only when the Janitor Provider is deployed in a different namespace or under a custom service name.

TLS

1janitor:
2  config:
3    cspProvider:
4      tls:
5        enabled: true
6        caSecretName: "janitor-provider-grpc-cert"
7        insecure: false

enabled

When true, the Janitor uses TLS for the gRPC connection to the Janitor Provider. Disable only in isolated test environments.

caSecretName

Name of the Kubernetes Secret containing ca.crt used to verify the Janitor Provider’s TLS certificate. The self-signed certificate is managed by cert-manager, which is a required dependency.

insecure

Set to true to skip TLS certificate verification. For development use only; never enable in production.

Service Account Token Auth

1janitor:
2  config:
3    cspProvider:
4      auth:
5        enabled: true
6        audience: "nvsentinel-csp-provider"
7        expirationSeconds: 3600

enabled

When true, the Janitor mounts a projected ServiceAccount token and sends it to the Janitor Provider for authentication.

audience

The token audience must match the auth.audiences value configured on the Janitor Provider.

expirationSeconds

Requested lifetime of the projected ServiceAccount token in seconds. Kubernetes automatically rotates the token before expiry.

Manual Mode

1janitor:
2  config:
3    manualMode: false

When true, the Janitor creates the Custom Resource but does not call the Janitor Provider to execute any node operation. Use this to test the fault-remediation → janitor CR creation pipeline without triggering actual node reboots, terminations, or GPU resets.

HTTP Port

1janitor:
2  config:
3    httpPort: 8082

Port for the Janitor’s internal HTTP server (health and readiness endpoints).

Node Exclusions

Prevents specific nodes from being targeted by any Janitor operation.

1janitor:
2  config:
3    nodes:
4      exclusions: []

Provide a list of node names to exclude:

1janitor:
2  config:
3    nodes:
4      exclusions:
5        - control-plane-node-1
6        - infra-node-2

Use this for control-plane nodes, infrastructure nodes, or any node that must never be rebooted or terminated by NVSentinel.

Controllers

Each controller handles one CR type. All three are enabled by default.

RebootNode Controller

1janitor:
2  config:
3    controllers:
4      rebootNode:
5        enabled: true
6        timeout: "25m"

Handles RebootNode CRs. timeout overrides config.timeout for reboot operations.

TerminateNode Controller

1janitor:
2  config:
3    controllers:
4      terminateNode:
5        enabled: true
6        timeout: "25m"

Handles TerminateNode CRs. timeout overrides config.timeout for terminate operations.

GPUReset Controller

1janitor:
2  config:
3    controllers:
4      gpuReset:
5        enabled: true
6        timeout: "25m"
7        serviceManager:
8          name: "gpu-operator"
9        resetJob:
10          writeSysLogEvent: true
11          runtimeClassName: "nvidia"
12          image:
13            repository: ghcr.io/nvidia/nvsentinel/gpu-reset
14            tag: ""
15          resources:
16            requests:
17              cpu: "50m"
18              memory: "64Mi"
19            limits:
20              cpu: "100m"
21              memory: "128Mi"

Handles GPUReset CRs. Before issuing the GPU reset, the controller pauses the deployment or DaemonSet named by serviceManager.name to prevent the GPU Operator from interfering with the reset sequence.

serviceManager.name

Name of the Kubernetes Deployment or DaemonSet to pause during GPU reset. Set to the GPU Operator deployment name in your cluster.

resetJob.writeSysLogEvent

When true, the reset job writes a kernel syslog message on reset completion. Useful for correlating reset events with node-level logs.

resetJob.runtimeClassName

NVIDIA RuntimeClass name used by the GPU reset Job. Must match a RuntimeClass installed in the cluster.

resetJob.image

Container image for the GPU reset Job. Leave tag empty to use the chart default.

resetJob.resources

Resource requests and limits for the GPU reset Job container.

TTL-Based CR Cleanup

Completed CRs are automatically deleted after the TTL expires.

1janitor:
2  ttl:
3    enabled: true
4    defaultTTL: "336h"

enabled

When true, the TTL controller deletes completed RebootNode, TerminateNode, and GPUReset CRs after defaultTTL has elapsed since completion.

defaultTTL

Duration after which completed CRs are deleted. Default is 336h (14 days). Use a shorter value (e.g. "24h") in test environments to keep the CR list clean.

Webhook

The Janitor uses an admission webhook to validate CRs before they are persisted.

1janitor:
2  webhook:
3    port: 9443
4    certIssuer: "janitor-selfsigned-issuer"
5    certProvider: cert-manager

certProvider

cert-manager is a required dependency. The webhook certificate is issued by the certIssuer ClusterIssuer and renewed automatically.