> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/nvsentinel/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nvsentinel/_mcp/server.

# NVSentinel Configuration Documentation

This directory contains technical configuration guides for NVSentinel operators and system administrators.

## Global Configuration

Global settings apply across all NVSentinel modules and are configured under the `global:` section in the Helm values.

### Image Configuration

Image tag for all NVSentinel modules.

```yaml
global:
  image:
    tag: "main"
```

### Dry Run Mode

Run all modules in dry-run mode where actions are logged but not executed.

```yaml
global:
  dryRun: false
```

### Metrics Port

Prometheus metrics port used by all modules.

```yaml
global:
  metricsPort: 2112
```

### Change Stream Resume Tokens

Watcher-based components persist change stream resume tokens so they can resume from the last processed event after a restart. To skip accumulated events and start from the current stream head, scale the component to zero, patch its key in the runtime resume-control ConfigMap from `RESUME` to `CREATE`, then restore its replicas. The component deletes only its own resume token, records a cold-start cutoff timestamp, skips startup cold-start recovery for that run, opens its watcher from the current stream head, and writes its key back to `RESUME`. Future restarts still run cold-start recovery, but only for records newer than the recorded cutoff.

Helm does not create the resume-control ConfigMap. Components create it at runtime if it is missing, so GitOps tools such as Argo CD do not revert operator patches to its data.
When a component starts and its key is missing, it writes its key as `RESUME`; the ConfigMap therefore self-populates with explicit per-component state over time.

Example one-shot reset for node-drainer:

```bash
REPLICAS=$(kubectl -n nvsentinel get deployment node-drainer -o jsonpath='\{.spec.replicas\}')
kubectl -n nvsentinel scale deployment/node-drainer --replicas=0
kubectl -n nvsentinel rollout status deployment/node-drainer --timeout=180s
kubectl -n nvsentinel get configmap resume-control >/dev/null 2>&1 || \
  kubectl -n nvsentinel create configmap resume-control
kubectl -n nvsentinel patch configmap resume-control \
  --type merge \
  -p '\{"data":\{"node-drainer":"CREATE"\}\}'
kubectl -n nvsentinel scale deployment/node-drainer --replicas="$\{REPLICAS:-1\}"
kubectl -n nvsentinel rollout status deployment/node-drainer --timeout=180s
```

This applies to `fault-quarantine`, `node-drainer`, `fault-remediation`, and `health-events-analyzer`.

### Node Scheduling

Control where NVSentinel pods are scheduled.

```yaml
global:
  # For GPU-bound pods (health monitors, metadata collector)
  nodeSelector: \{\}
  tolerations: []
  affinity: \{\}
  
  # For system pods (fault-quarantine, node-drainer etc)
  systemNodeSelector: \{\}
  systemNodeTolerations: []
```

### Image Pull Secrets

Credentials for pulling images from private registries.

```yaml
global:
  imagePullSecrets: []
```

### Tracing

Enable OpenTelemetry distributed tracing to get end-to-end visibility into health event processing across all modules.

```yaml
global:
  tracing:
    enabled: false       # Enable/disable tracing for all components
    endpoint: ""         # OTLP gRPC address of your OpenTelemetry Collector (e.g., "alloy.observability.svc.cluster.local:4317")
    insecure: true       # Set to false if the collector endpoint uses TLS
```

For full details, see [Distributed Tracing](/nvsentinel/observability/distributed-tracing).

### Audit logging

Enable file-based audit logs of HTTP write operations (POST, PUT, PATCH, DELETE) to the Kubernetes and CSP APIs, with rotation and optional request-body capture.

```yaml
global:
  auditLogging:
    enabled: true
    logRequestBody: false
    maxSizeMB: 100
    maxBackups: 7
    maxAgeDays: 30
    compress: true
```

For full details, see [Audit Logging](/nvsentinel/observability/audit-logging).

## Module-Specific Configuration

Each module has additional configuration options documented in its dedicated guide:

- [GPU Health Monitor](/nvsentinel/configuration/gpu-health-monitor)
- [Syslog Health Monitor](/nvsentinel/configuration/syslog-health-monitor)
- [CSP Health Monitor](/nvsentinel/configuration/csp-health-monitor)
- [Kubernetes Object Monitor](/nvsentinel/configuration/kubernetes-object-monitor)
- [Platform Connectors](/nvsentinel/configuration/platform-connectors)
- [Metadata Collector](/nvsentinel/configuration/metadata-collector)
- [Labeler](/nvsentinel/configuration/labeler)
- [Fault Quarantine](/nvsentinel/configuration/fault-quarantine)
- [Node Drainer](/nvsentinel/configuration/node-drainer)
- [Fault Remediation](/nvsentinel/configuration/fault-remediation)
- [Preflight](/nvsentinel/configuration/preflight)
- [Event Exporter](/nvsentinel/configuration/event-exporter)