Configuration Overview | NVIDIA NVSentinel Documentation

This directory contains technical configuration guides for NVSentinel operators and system administrators.

Global Configuration

Global settings apply across all NVSentinel modules and are configured under the global: section in the Helm values.

Image Configuration

Image tag for all NVSentinel modules.

1 global:
2   image:
3     tag: "main"

Dry Run Mode

Run all modules in dry-run mode where actions are logged but not executed.

1 global:
2   dryRun: false

Metrics Port

Prometheus metrics port used by all modules.

1 global:
2   metricsPort: 2112

Change Stream Resume Tokens

Watcher-based components persist change stream resume tokens so they can resume from the last processed event after a restart. To skip accumulated events and start from the current stream head, scale the component to zero, patch its key in the runtime resume-control ConfigMap from RESUME to CREATE, then restore its replicas. The component deletes only its own resume token, records a cold-start cutoff timestamp, skips startup cold-start recovery for that run, opens its watcher from the current stream head, and writes its key back to RESUME. Future restarts still run cold-start recovery, but only for records newer than the recorded cutoff.

Helm does not create the resume-control ConfigMap. Components create it at runtime if it is missing, so GitOps tools such as Argo CD do not revert operator patches to its data. When a component starts and its key is missing, it writes its key as RESUME; the ConfigMap therefore self-populates with explicit per-component state over time.

Example one-shot reset for node-drainer:

$ REPLICAS=$(kubectl -n nvsentinel get deployment node-drainer -o jsonpath='\{.spec.replicas\}')
$ kubectl -n nvsentinel scale deployment/node-drainer --replicas=0
$ kubectl -n nvsentinel rollout status deployment/node-drainer --timeout=180s
$ kubectl -n nvsentinel get configmap resume-control >/dev/null 2>&1 || \
>   kubectl -n nvsentinel create configmap resume-control
$ kubectl -n nvsentinel patch configmap resume-control \
>   --type merge \
>   -p '\{"data":\{"node-drainer":"CREATE"\}\}'
$ kubectl -n nvsentinel scale deployment/node-drainer --replicas="$\{REPLICAS:-1\}"
$ kubectl -n nvsentinel rollout status deployment/node-drainer --timeout=180s

This applies to fault-quarantine, node-drainer, fault-remediation, and health-events-analyzer.

Node Scheduling

Control where NVSentinel pods are scheduled.

1 global:
2   # For GPU-bound pods (health monitors, metadata collector)
3   nodeSelector: \{\}
4   tolerations: []
5   affinity: \{\}
6   
7   # For system pods (fault-quarantine, node-drainer etc)
8   systemNodeSelector: \{\}
9   systemNodeTolerations: []

Image Pull Secrets

Credentials for pulling images from private registries.

1 global:
2   imagePullSecrets: []

Tracing

Enable OpenTelemetry distributed tracing to get end-to-end visibility into health event processing across all modules.

1 global:
2   tracing:
3     enabled: false       # Enable/disable tracing for all components
4     endpoint: ""         # OTLP gRPC address of your OpenTelemetry Collector (e.g., "alloy.observability.svc.cluster.local:4317")
5     insecure: true       # Set to false if the collector endpoint uses TLS

For full details, see Distributed Tracing.

Audit logging

Enable file-based audit logs of HTTP write operations (POST, PUT, PATCH, DELETE) to the Kubernetes and CSP APIs, with rotation and optional request-body capture.

1 global:
2   auditLogging:
3     enabled: true
4     logRequestBody: false
5     maxSizeMB: 100
6     maxBackups: 7
7     maxAgeDays: 30
8     compress: true

For full details, see Audit Logging.

Module-Specific Configuration

Each module has additional configuration options documented in its dedicated guide: