Fault Quarantine Configuration
Overview
The Fault Quarantine module isolates nodes with detected hardware or software failures by cordoning and/or tainting them. This document covers all Helm configuration options available for system administrators.
Configuration Reference
Module Enable/Disable
Controls whether the fault-quarantine module is deployed in the cluster.
Note: This module depends on the datastore being enabled. Therefore, ensure the datastore is also enabled.
Resources
Defines CPU and memory resource requests and limits for the fault-quarantine pod.
Logging
Sets the verbosity level for fault-quarantine logs.
Kubernetes API Rate Limits
Fault Quarantine inherits the Kubernetes client limits from global.qps and global.burst (defaults: 5 and 10). Set component values only when Fault Quarantine needs different limits:
Positive qps values enable client-side throttling, 0 uses the client-go default, and a negative value disables client-side throttling. burst must be non-negative; 0 uses the client-go default.
Change Stream Resume Token
To make fault-quarantine skip accumulated events and start from the current stream head, scale it to zero, set its key in the shared resume-control ConfigMap to CREATE, then restore its replicas. Fault-quarantine deletes only its own resume token and resets the key back to RESUME during startup.
On a normal RESUME startup, fault-quarantine queries the datastore for processable health events that do not have a quarantine status and replays them before consuming live events. An event is suppressed only when later events collectively cover every check, entity, and error-code scope it could change; a check-wide event therefore requires a later check-wide event. This conservative rule prevents old failures or recoveries from temporarily overwriting newer node state. Recovery uses one current API-server node snapshot per replayed event instead of the informer cache. The live watcher opens first, so events inserted during recovery remain queued for normal processing.
Skipped, invalid, and superseded documents receive a terminal fault-quarantine recovery marker in their status; the detailed result remains available through the recovery metric. Successful quarantine updates already carry a terminal quarantine status. Both forms are excluded from later scans. Normal recovery does not advance a wall-clock checkpoint, because event creation time and datastore insertion order can differ. If an installation has no persisted cutoff, as on the first upgrade to this behavior, the lower bound is initialized and persisted at the current startup boundary rather than scanning the cluster’s entire history.
A consumed CREATE request skips cold start and records its creation time. Later restarts only consider unresolved events created after that cutoff, so events that an operator deliberately abandoned are not revived.
Cold-start activity is exposed through fault_quarantine_cold_start_events_total{result=...} and fault_quarantine_cold_start_duration_seconds.
GPU Node Label
Label selector, as key=value, that identifies GPU nodes. The node informer watches only matching nodes, and the circuit breaker counts only those nodes when it measures the cordoned percentage.
The default matches the label the NVIDIA GPU Operator applies. Override it when your cluster labels GPU nodes differently.
A selector that matches no node gives the circuit breaker a node count of zero. It then logs Total nodes is still 0 after all retry attempts - cluster may have no GPU nodes and returns an error instead of a decision, so check this value first when the breaker behaves unexpectedly on a cluster with non-standard labels.
Label Prefix
Defines the prefix for all node labels created by the module to track cordon/uncordon lifecycle.
Generated Labels
Labels use the configured {labelPrefix} (default k8saas.nvidia.com/):
{labelPrefix}cordon-by— Service that cordoned the node{labelPrefix}cordon-reason— Reason for cordoning{labelPrefix}cordon-timestamp— Cordon timestamp (format: 2006-01-02T15-04-05Z){labelPrefix}uncordon-by— Service that uncordoned the node{labelPrefix}uncordon-reason— Reason for uncordoning{labelPrefix}uncordon-timestamp— Uncordon timestamp (format: 2006-01-02T15-04-05Z)
Circuit Breaker
Prevents too many nodes from being quarantined simultaneously, protecting against cluster-wide cascading failures.
Configuration
Parameters
enabled
Enables or disables circuit breaker protection. When disabled, unlimited nodes can be quarantined.
percentage
Maximum percentage of GPU nodes that can be quarantined within the time window. When reached, the circuit breaker trips and blocks all new quarantine actions. Rounds up, so 1 on a 288-node cluster is 3 nodes, not 2. Set to 0 to bound by maxNodes alone.
maxNodes
Maximum absolute number of nodes that can be quarantined within the time window, independent of cluster size. Defaults to 0, which disables this bound and preserves the percentage-only behaviour.
When both bounds are set, the lower one binds. This is the point of the setting: a percentage silently tracks fleet growth, so a limit chosen for a 100 node cluster becomes three times as permissive at 300 nodes, while an absolute bound does not move. Fleet-wide operators can therefore express “no more than X% and never more than N nodes” in one config that stays correct as clusters grow.
maxCordonedNodes
Standing cap on how many nodes NVSentinel holds quarantined at once, counting only its own cordons. Defaults to 0, which disables it.
This is the only bound that limits a total. percentage and maxNodes both count cordon events inside duration, so they limit the rate of cordoning and not the number of nodes that end up cordoned: a rate that stays just under the threshold still reaches the whole fleet given enough windows. On a 300 node cluster, maxNodes: 5 with a 5m window permits 4 cordons every window indefinitely, which is the entire fleet in about six hours without ever tripping.
Lengthening duration is not a substitute. The window is bucketed at one second, so 24h allocates 86,400 buckets and the sliding-window advance is O(steps x buckets); it also trips on a day’s accumulated churn rather than on how many nodes are actually out of service.
Nodes cordoned by anything other than NVSentinel are not counted, so a GPU operator upgrade or an operator’s own kubectl cordon does not consume this budget.
The two kinds of bound compose rather than compete: keep percentage or maxNodes to bound a burst, and add maxCordonedNodes to bound the standing total. It is evaluated only when the window bounds have not already tripped, so it can add a reason to trip but never remove one.
At least one of percentage, maxNodes and maxCordonedNodes must be positive, and a negative value for any of them is rejected rather than ignored. The breaker refuses to start otherwise, so it cannot be silently reduced to a no-op while enabled: true.
For the same reason, a window threshold above the fleet size is clamped to the fleet size rather than being left unreachable, and the bound label described below reports fleetSize when that happens. This clamp applies to percentage and maxNodes only. maxCordonedNodes is compared directly against the current count of nodes NVSentinel holds quarantined and is not clamped, so it never reports fleetSize.
To turn the breaker off, use enabled: false rather than an unreachable threshold.
duration
Time window for tracking cordon events. The circuit breaker counts unique node cordons within this sliding window.
Observability
fault_quarantine_breaker_threshold_nodes{bound} reports the effective threshold in nodes, with bound naming what produced it: percentage, maxNodes, maxCordonedNodes when the standing bound is what tripped, or fleetSize when a configured bound exceeded the cluster size and was clamped. fault_quarantine_breaker_utilization keeps its existing meaning, the recent cordon count as a fraction of GPU nodes, so existing dashboards and alerts are unaffected.
Configuration Examples
Aggressive:
Conservative:
Percentage with an absolute ceiling, for a fleet whose clusters vary in size:
Absolute bound only, for “never cordon more than 2 nodes regardless of cluster size”:
Burst bound plus a standing bound, for “no more than 5 cordons in any 5 minutes, and never more than 20 nodes out of service at once”:
Disabled:
Rule Sets
Rule sets define conditions for quarantining nodes using CEL expressions. Each rule set specifies match conditions (when to trigger) and actions (what to do).
Rule Set Structure
Parameters
enabled
When false, the ruleset is skipped entirely. Use this to turn off a built-in ruleset without removing its definition.
version
Rule set format version for future compatibility.
name
Unique identifier used in logs, metrics, and as part of the cordon-reason label.
priority
Optional integer for resolving conflicts when multiple rule sets apply the same taint key-value pair or label key. Higher values take precedence. Label priority is preserved across all matching health events in the same quarantine session; a later lower-priority event cannot downgrade the current label. For labels with equal priority, the later rule set in configuration order wins.
match
Defines conditions that must be satisfied for the rule set to trigger. Supports all (AND) and any (OR) logic.
kind
Specifies the object type to evaluate in the CEL expression. Valid values: HealthEvent (evaluates against health event data) or Node (evaluates against Kubernetes node object).
expression
CEL (Common Expression Language) expression that evaluates to true or false. For HealthEvent kind, access fields via the event variable. For Node kind, access node.metadata and node.spec; node.status is not cached or available.
cordon
Specifies whether to mark the node as unschedulable when the rule matches.
taint
Optional Kubernetes taint to apply. Taints can prevent pod scheduling or evict existing pods based on the effect.
label
Optional Kubernetes label to apply for the lifetime of the quarantine session. If the key already exists, fault-quarantine overwrites its value. The label, priority, and configuration order are recorded in the quarantineHealthEventAppliedLabels node annotation so later events use the same conflict policy. The label is removed after every health event tracked in quarantineHealthEvent has recovered. Manual uncordon, manual untaint, and stale-state cleanup remove the tracking annotation but preserve the label itself, mirroring the existing applied-taint behavior. In dry-run mode, the intended label is recorded in annotations for observability, but Kubernetes Node labels are not added, overwritten, or removed.
Example Rule Sets
Example 1: Fatal GPU Errors from GPU Health Monitor AND node not labeled with k8saas.nvidia.com/ManagedByNVSentinel=false
Example 2: Syslog Fatal Errors Excluding XID 45 AND node not labeled with k8saas.nvidia.com/ManagedByNVSentinel=false
Adding and Modifying Rule Sets
fault-quarantine.ruleSets is a list. When Helm merges multiple values files (-f), lists are replaced entirely — not merged by name. A file that sets only one ruleset replaces the whole list; built-in rulesets from values.yaml are dropped unless you include them again. The chart has no keyed-merge for rulesets.
Copy the default ruleSets from values.yaml into your values file, edit the full list, then upgrade:
- Disable a ruleset: set
enabled: falseon that entry in your complete list. - Add a ruleset: append an entry (see examples above).
- Modify a ruleset: change
match,cordon, ortainton that entry in your complete list.
The chart renders the list into the fault-quarantine ConfigMap (config.toml); a config change triggers a pod restart.
Use global.dryRun: true to test without cordoning nodes (see Dry Run Mode). Confirm the rollout with kubectl -n nvsentinel rollout status deployment/fault-quarantine and check kubectl -n nvsentinel logs deployment/fault-quarantine.