NVSentinel Configuration Documentation

This directory contains technical configuration guides for NVSentinel operators and system administrators.

Global Configuration

Global settings apply across all NVSentinel modules and are configured under the global: section in the Helm values.

Image Configuration

Image tag for all NVSentinel modules.

global:
  image:
    tag: "main"

Init Container Image

Shell image for the helper init containers that NVSentinel modules run before startup: fix-audit-log-permissions sets ownership on the audit log directory, and fix-cert-permissions copies PostgreSQL client certificates with the permissions the client needs. A module runs them only when audit logging is enabled or a client certificate is mounted.

global:
  initContainerImage:
    repository: docker.io/bitnamilegacy/os-shell
    tag: "12-debian-12-r30"
    pullPolicy: IfNotPresent

Change this when you mirror images into a private registry.

Dry Run Mode

Run all modules in dry-run mode where actions are logged but not executed.

global:
  dryRun: false

Metrics Port

Prometheus metrics port used by all modules.

global:
  metricsPort: 2112

Cluster Domain

DNS domain of the cluster. The chart uses it to build in-cluster service hostnames, such as the datastore URI and the deployment platform connector’s address, and the DNS names of the certificates it issues. The default is cluster.local; set it when the cluster uses another domain (kubeadm networking.dnsDomain, Cluster API clusterNetwork.serviceDomain).

global:
  clusterDomain: example.local

The vendored datastore charts do not read NVSentinel’s global values, and a few values hold a full hostname, so set the same domain there as well. The same goes for global.datastore.connection.host if you set it.

mongodb-store:
  mongodb:
    clusterDomain: example.local                # Bitnami MongoDB
  psmdb-db:
    clusterServiceDNSSuffix: svc.example.local  # Percona
postgresql:
  clusterDomain: example.local
janitor:
  config:
    cspProviderHost: janitor-provider.nvsentinel.svc.example.local:50051
fault-remediation:
  logCollector:
    uploadURL: http://nvsentinel-incluster-file-server.nvsentinel.svc.example.local/upload

Change Stream Resume Tokens

Watcher-based components persist change stream resume tokens so they can resume from the last processed event after a restart. To skip accumulated events and start from the current stream head, scale the component to zero, patch its key in the runtime resume-control ConfigMap from RESUME to CREATE, then restore its replicas. The component deletes only its own resume token, records a cold-start cutoff timestamp, skips startup cold-start recovery for that run, opens its watcher from the current stream head, and writes its key back to RESUME. Future restarts still run cold-start recovery, but only for records newer than the recorded cutoff.

Helm does not create the resume-control ConfigMap. Components create it at runtime if it is missing, so GitOps tools such as Argo CD do not revert operator patches to its data. When a component starts and its key is missing, it writes its key as RESUME; the ConfigMap therefore self-populates with explicit per-component state over time.

Example one-shot reset for node-drainer:

REPLICAS=$(kubectl -n nvsentinel get deployment node-drainer -o jsonpath='{.spec.replicas}')
kubectl -n nvsentinel scale deployment/node-drainer --replicas=0
kubectl -n nvsentinel rollout status deployment/node-drainer --timeout=180s
kubectl -n nvsentinel get configmap resume-control >/dev/null 2>&1 || \
  kubectl -n nvsentinel create configmap resume-control
kubectl -n nvsentinel patch configmap resume-control \
  --type merge \
  -p '{"data":{"node-drainer":"CREATE"}}'
kubectl -n nvsentinel scale deployment/node-drainer --replicas="${REPLICAS:-1}"
kubectl -n nvsentinel rollout status deployment/node-drainer --timeout=180s

This applies to fault-quarantine, node-drainer, fault-remediation, and health-events-analyzer.

Node Scheduling

Control where NVSentinel pods are scheduled.

global:
  # For GPU-bound pods (health monitors, metadata collector)
  nodeSelector: {}
  tolerations: []
  affinity: {}
  
  # For system pods (fault-quarantine, node-drainer etc)
  systemNodeSelector: {}
  systemNodeTolerations: []

Pod Priority

NVSentinel pods set no priorityClassName by default, so each takes the priority of the cluster’s globalDefault PriorityClass if one is configured, and 0 otherwise. A pod can preempt another only when its own priority is higher, so on a saturated cluster the scheduler leaves these pods Pending rather than preempting a lower-priority workload, and a node can end up with no health monitor while the workload still reports healthy.

Assign a priority class to avoid that. The split matches the node-scheduling values above: priorityClassName covers the node-level agents (the health monitor, metadata collector and NIC health monitor DaemonSets, the preflight image cache, plus platform-connectors), systemPriorityClassName covers the control-plane components (labeler, health-events-analyzer, fault-quarantine, node-drainer etc). Both apply only to the components NVSentinel’s own charts render — see Scope below for the datastore, which needs its own key.

global:
  priorityClassName: ""        # node-level agents (DaemonSets)
  systemPriorityClassName: ""  # control-plane components (Deployments)

Both default to empty, which leaves the field off the pod spec entirely and preserves the existing behaviour. Any component can also be set individually, and the global takes precedence when both are set, consistent with how tolerations behaves:

gpu-health-monitor:
  priorityClassName: my-gpu-agent-priority

The priority classes must already exist in the cluster. system-node-critical and system-cluster-critical are built in; anything else has to be created first.

A higher priority is necessary but not sufficient: preemption also needs an evictable lower-priority pod on a node that would then fit, and a class with preemptionPolicy: Never only improves queue order without evicting anything. Both built-in classes above preempt.

Scope: NVSentinel components only

Both globals cover the components NVSentinel’s own charts render. They do not reach the datastore. The MongoDB and PostgreSQL pods come from vendored upstream charts that never read NVSentinel’s global values, so setting the globals and rendering the release leaves those StatefulSets with no priorityClassName.

This matters because the datastore is the component whose eviction hurts most: every module reconciles from it, so a preempted database stops fault detection across the whole cluster while the health monitors it starved keep their own high priority.

Set the priority on the datastore through the upstream chart’s own key:

global:
  priorityClassName: system-node-critical
  systemPriorityClassName: system-cluster-critical
# The datastore needs its own key — the globals above do not apply to it.
mongodb-store:
  mongodb:
    priorityClassName: system-cluster-critical

Note that Percona spells it priorityClass, without Name, and that its key sits under the replica set. The Bitnami MongoDB chart takes separate keys for the arbiter and hidden members, mongodb-store.mongodb.arbiter.priorityClassName and mongodb-store.mongodb.hidden.priorityClassName, if you run them.

The create-mongodb-database initialization Job accepts no priority class from any key. It runs once at install or upgrade and then completes, so it is scheduled against whatever capacity is free at that moment. On a saturated cluster this Job can stay Pending and hold up the install.

Image Pull Secrets

Credentials for pulling images from private registries.

global:
  imagePullSecrets: []

Tracing

Enable OpenTelemetry distributed tracing to get end-to-end visibility into health event processing across all modules.

global:
  tracing:
    enabled: false       # Enable/disable tracing for all components
    endpoint: ""         # OTLP gRPC address of your OpenTelemetry Collector (e.g., "alloy.observability.svc.cluster.local:4317")
    insecure: true       # Set to false if the collector endpoint uses TLS

For full details, see Distributed Tracing.

Audit logging

Enable file-based audit logs of HTTP write operations (POST, PUT, PATCH, DELETE) to the Kubernetes and CSP APIs, with rotation and optional request-body capture.

global:
  auditLogging:
    enabled: true
    logRequestBody: false
    maxSizeMB: 100
    maxBackups: 7
    maxAgeDays: 30
    compress: true

For full details, see Audit Logging.

Shared Metadata Path

File path where the metadata collector writes GPU metadata and the syslog health monitor reads it. Both components must agree on this path.

global:
  metadataPath: /var/lib/nvsentinel/gpu_metadata.json

Kubelet Root Directory

The kubelet --root-dir. The chart mounts <kubeletRootDir>/pod-resources from the host at the fixed path the metadata collector reads. See Kubelet Root Directory.

global:
  kubeletRootDir: /var/lib/kubelet

Client Certificate Rotation

Lets modules pick up rotated datastore client certificates without a pod restart. A file watcher detects the change and supplies the new certificate to subsequent MongoDB connections. Existing connections keep the certificate they opened with.

global:
  certificateRotationEnabled: false

Kubernetes Datastore CRDs

Installs the HealthEvent custom resource definition used by the Kubernetes datastore backend. Enable it only when you run that backend instead of MongoDB or PostgreSQL.

global:
  k8sdatastoreCrds:
    enabled: false

Node Condition Cleanup

Removes node conditions that NVSentinel no longer writes. An upgrade that renames or retires a condition leaves the old one on every node, where it stays True forever because nothing updates it any more. The cleanup hook deletes the conditions you name.

nodeConditionCleanup:
  enabled: false
  deprecatedConditions: []
  # - "OldConditionType1"
  # - "OldConditionType2"

The hook runs as a Helm post-upgrade job, so it acts on upgrade rather than on a fresh install, and it takes no action while deprecatedConditions is empty. List only conditions you are certain are retired — deleting a live condition discards a node’s current health state until the monitor writes it again.

PodMonitor

Creates a Prometheus Operator PodMonitor for every NVSentinel component. Disable it if you scrape with annotation-based discovery instead.

podMonitor:
  enabled: true
  interval: 30s        # Scrape interval
  # scrapeTimeout: ""  # Defaults to the Prometheus default
  # metricsPath: "/metrics"
  # labels: {}         # Extra labels on the PodMonitor resource

See Metrics Reference for what each component exposes.

Platform Connectors Update Strategy

Rolling update strategy for the platform-connectors DaemonSet. type accepts RollingUpdate or OnDelete; rollingUpdate is rendered only for RollingUpdate.

updateStrategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 5%

maxUnavailable accepts a percentage or an absolute count — maxUnavailable: 34 updates 34 nodes at a time. A larger value drains the fleet’s health reporting faster during an upgrade; a smaller one keeps more nodes reporting while the rollout runs.

Common Module Settings

Several modules expose the same Kubernetes settings. They behave the same everywhere, so each module’s own guide documents only the options specific to it.

Module-Specific Configuration

Each module has additional configuration options documented in its dedicated guide:

• GPU Health Monitor
• Syslog Health Monitor
• CSP Health Monitor
• Kubernetes Object Monitor
• NVCRE Certification Monitor
• Platform Connectors
• Metadata Collector
• Labeler
• Fault Quarantine
• Lifecycle Manager
• Validation
• Node Drainer
• Fault Remediation
• Preflight
• Event Exporter