CSP Health Monitor Configuration | NVIDIA NVSentinel Documentation

Overview

The CSP Health Monitor detects cloud provider maintenance events and triggers automated node quarantine workflows. This document covers all Helm configuration options.

Module Enable/Disable

Controls whether the csp-health-monitor module is deployed in the cluster.

global:
  cspHealthMonitor:
    enabled: true

Cloud Provider Selection

The cspName field determines which cloud provider to monitor. Only one provider can be active at a time.

csp-health-monitor:
  cspName: "gcp"  # Options: "gcp", "aws", "lambda", or "akamai"

Global Settings

Settings that apply regardless of cloud provider.

csp-health-monitor:
  logLevel: info  # Options: debug, info, warn, error
  
  configToml:
    # Cluster identifier used in health events
    clusterName: "my-cluster"
    
    # How often the sidecar polls MongoDB for maintenance events (seconds)
    maintenanceEventPollIntervalSeconds: 60
    
    # Minutes before maintenance start time to trigger quarantine
    triggerQuarantineWorkflowTimeLimitMinutes: 30
    
    # Minutes after maintenance ends to send healthy event
    postMaintenanceHealthyDelayMinutes: 15
    
    # Timeout for node to become ready after maintenance (minutes)
    nodeReadinessTimeoutMinutes: 60

GCP Configuration

Required Fields

csp-health-monitor:
  cspName: "gcp"
  
  configToml:
    clusterName: "my-gke-cluster"
    
    gcp:
      # GCP project ID where the cluster runs
      targetProjectId: "my-gcp-project-id"
      
      # GCP Service Account name (without @project.iam.gserviceaccount.com)
      # Must match the GCP SA created in IAM setup
      gcpServiceAccountName: "csp-health-monitor"
      
      # How often to poll Cloud Logging API (seconds)
      apiPollingIntervalSeconds: 60
      
      # Cloud Logging filter for maintenance events
      logFilter: 'logName="projects/my-gcp-project-id/logs/cloudaudit.googleapis.com%2Fsystem_event" AND protoPayload.methodName="compute.instances.upcomingMaintenance"'

GCP Parameters

targetProjectId

GCP project ID where the GKE cluster is running. The monitor queries Cloud Logging in this project.

gcpServiceAccountName

Name of the GCP Service Account (without the @project.iam.gserviceaccount.com suffix). Used to generate the Workload Identity annotation on the Kubernetes ServiceAccount.

apiPollingIntervalSeconds

How frequently the monitor polls the Cloud Logging API for new maintenance events. Lower values provide faster detection but increase API usage.

logFilter

Cloud Logging filter expression to select maintenance events. Common filters:

# Standard GCE instance maintenance
'logName="projects/{PROJECT_ID}/logs/cloudaudit.googleapis.com%2Fsystem_event" AND protoPayload.methodName="compute.instances.upcomingMaintenance"'
# Include termination events
'logName="projects/{PROJECT_ID}/logs/cloudaudit.googleapis.com%2Fsystem_event" AND (protoPayload.methodName="compute.instances.upcomingMaintenance" OR protoPayload.methodName="compute.instances.terminateOnHostMaintenance")'

Complete GCP Example

global:
  cspHealthMonitor:
    enabled: true
csp-health-monitor:
  cspName: "gcp"
  logLevel: info
  
  configToml:
    clusterName: "production-gke-cluster"
    maintenanceEventPollIntervalSeconds: 60
    triggerQuarantineWorkflowTimeLimitMinutes: 30
    postMaintenanceHealthyDelayMinutes: 15
    nodeReadinessTimeoutMinutes: 60
    
    gcp:
      targetProjectId: "my-production-project"
      gcpServiceAccountName: "csp-health-monitor"
      apiPollingIntervalSeconds: 60
      logFilter: 'logName="projects/my-production-project/logs/cloudaudit.googleapis.com%2Fsystem_event" AND protoPayload.methodName="compute.instances.upcomingMaintenance"'

AWS Configuration

Required Fields

csp-health-monitor:
  cspName: "aws"
  
  configToml:
    clusterName: "my-eks-cluster"
    
    aws:
      # AWS Account ID (12-digit number)
      accountId: "123456789012"
      
      # AWS region where the EKS cluster runs
      region: "us-east-1"
      
      # How often to poll AWS Health API (seconds)
      pollingIntervalSeconds: 60
      
      # (Optional) Custom IAM role name for IRSA
      iamRoleName: ""

AWS Parameters

accountId

AWS account ID (12-digit number) where the EKS cluster is running. Used to construct the IAM role ARN annotation.

region

AWS region where the EKS cluster is deployed. The monitor queries the AWS Health API in this region.

pollingIntervalSeconds

How frequently the monitor polls the AWS Health API for maintenance events. Lower values provide faster detection but increase API usage.

iamRoleName

Custom IAM role name for IRSA (IAM Roles for Service Accounts). When set, the ServiceAccount annotation uses this role name directly instead of constructing one from clusterName.

If left empty (default), the role name is generated as {CLUSTER_NAME}-nvsentinel-health-monitor-assume-role-policy.

Important (EKS): AWS IAM role names have a maximum of 64 characters. The default suffix -nvsentinel-health-monitor-assume-role-policy is 45 characters, leaving only 19 characters for the cluster name. If your EKS cluster name exceeds 19 characters, you must set iamRoleName to a custom value.

Complete AWS Example

global:
  cspHealthMonitor:
    enabled: true
csp-health-monitor:
  cspName: "aws"
  logLevel: info
  
  configToml:
    clusterName: "production-eks-cluster"
    maintenanceEventPollIntervalSeconds: 60
    triggerQuarantineWorkflowTimeLimitMinutes: 30
    postMaintenanceHealthyDelayMinutes: 15
    nodeReadinessTimeoutMinutes: 60
    
    aws:
      accountId: "123456789012"
      region: "us-east-1"
      pollingIntervalSeconds: 60

AWS Example with Custom IAM Role Name

For clusters with long names (>19 characters), set iamRoleName explicitly:

csp-health-monitor:
  cspName: "aws"
  
  configToml:
    clusterName: "my-very-long-production-eks-cluster-name"
    
    aws:
      accountId: "123456789012"
      region: "us-east-1"
      pollingIntervalSeconds: 60
      iamRoleName: "my-custom-nvsentinel-role"

Lambda Configuration

Polls the Lambda maintenance-events API and raises a health event for each event affecting a node in this cluster. Nodes are matched by the instance UUID in the event’s entity_lrns, so they must carry spec.providerID in the form lambda://<instanceID>.

Static API Key Example

csp-health-monitor:
  cspName: "lambda"
  lambdaApiKeySecret:
    name: "lambda-api-key"
    key: "LAMBDA_API_KEY"
  configToml:
    clusterName: "my-cluster"
    lambda:
      apiEndpoint: "https://cloud.lambda.ai"
      workspaceId: "c4d291f47f9d436fa39f58493ce3b50d"
      pollingIntervalSeconds: 30

apiEndpoint

Base URL of the Lambda API. Defaults to the production endpoint, so the happy path needs no override.

workspaceId

Optional. Scopes the maintenance-events query to one workspace, in dashed or undashed UUID form. Leave it empty to use the default workspace for the credential.

Set it whenever that default is not the workspace running this cluster. Getting it wrong is quiet: the API answers 200 with an empty list rather than an error, so the monitor looks healthy while never seeing a maintenance event for these nodes. A value that is not a UUID fails at startup rather than returning 400 on every poll.

This matters more with workload identity, where the credential belongs to a service identity whose role is assigned at workspace scope — its default workspace is unlikely to be the cluster’s.

pollingIntervalSeconds

How often the maintenance-events API is polled. Minimum 30.

lambdaApiKeySecret

Secret holding the Lambda API key, injected as LAMBDA_API_KEY. name is required when cspName is lambda, unless lambdaWorkloadIdentity.identityLRN is set; the install fails without one of the two. The same Secret can be shared with the Janitor Provider, which reads the key the same way. The key needs permission to read maintenance events in the target workspace.

lambdaWorkloadIdentity.identityLRN

The service identity to assume, in the form lrn:iam:identity:<id>. Preferred over a static key: no Lambda credential is stored in the cluster, and the token is short-lived and rotated automatically.

Setting it annotates the csp-health-monitor ServiceAccount with lambda.ai/identity-lrn. The pod is then given a short-lived Kubernetes token, which the monitor exchanges for a Lambda API key and refreshes before it expires — so no Lambda credential is stored in the cluster.

csp-health-monitor:
  cspName: "lambda"
  lambdaWorkloadIdentity:
    identityLRN: "lrn:iam:identity:3cd2d107c6a347eeb0ef9498820d637d"
  configToml:
    clusterName: "my-cluster"
    lambda:
      workspaceId: "c4d291f47f9d436fa39f58493ce3b50d"

When set, lambdaApiKeySecret is not required and no LAMBDA_API_KEY is placed in the pod. If both are configured, workload identity wins and the static key is ignored; the monitor logs which one it selected at startup (authMode).

On the Lambda side, the identity must exist, hold permission to read maintenance events in the workspace named by workspaceId, and trust this cluster’s ServiceAccount.

(
  set -euo pipefail
  NAMESPACE=$(
    kubectl get deployments --all-namespaces \
      -l app.kubernetes.io/name=csp-health-monitor -o json |
      jq -er '
        .items
        | if length == 1 then .[0] else error("expected exactly one csp-health-monitor Deployment") end
        | .metadata.namespace
      '
  )
  kubectl -n "$NAMESPACE" get sa -l app.kubernetes.io/name=csp-health-monitor
)

Setting up the identity

Run once per cluster, with an admin API key. $WS is the workspace the cluster’s instances belong to — the same one you set as workspaceId above.

# Stop at the first failed call: a half-applied setup prints an identityLRN
# that looks usable but has no role or no trust behind it.
set -euo pipefail
KEY=<admin-api-key>
export WS=<workspace-id>
lambda_curl() {
  printf 'Authorization: Bearer %s\n' "$KEY" | curl -sf -H @- "$@"
}
# 1. Create a service identity for the CSP Health Monitor.
SID=$(lambda_curl "https://cloud.lambda.ai/api/v1/identities" \
  -H 'Content-Type: application/json' \
  -d '{"display_name":"nvsentinel-csp-health-monitor"}' | jq -r .data.id)
[ -n "$SID" ] && [ "$SID" != "null" ] || { echo "identity was not created" >&2; exit 1; }
# 2. Add it to the workspace. A workspace-scoped role assignment needs membership.
lambda_curl "https://cloud.lambda.ai/api/v1/workspaces/$WS/memberships" \
  -H 'Content-Type: application/json' \
  -d "{\"member_type\":\"identity\",\"member_id\":\"$SID\"}"
# 3. Assign the built-in roles, scoped to that workspace.
for role in maintenance-api-reader; do
  ROLE=$(lambda_curl "https://cloud.lambda.ai/api/v1/roles" \
    | jq -r --arg n "$role" '.data[] | select(.name==$n) | .id')
  [ -n "$ROLE" ] && [ "$ROLE" != "null" ] || { echo "role $role not found" >&2; exit 1; }
  lambda_curl "https://cloud.lambda.ai/api/v1/identities/$SID/role-assignments" \
    -H 'Content-Type: application/json' \
    -d "{\"role_id\":\"$ROLE\",\"scope\":{\"type\":\"workspace\",\"workspace_id\":\"$WS\"}}"
done
# 4. Trust this cluster's ServiceAccount to assume the identity. Registering the
#    issuer is an idempotent upsert keyed on the issuer URL, so re-running is safe.
SUBJECT=$(
  kubectl get deployments --all-namespaces \
    -l app.kubernetes.io/name=csp-health-monitor -o json |
    jq -er '
      .items
      | if length == 1 then .[0] else error("expected exactly one csp-health-monitor Deployment") end
      | "system:serviceaccount:\(.metadata.namespace):\(.spec.template.spec.serviceAccountName)"
    '
)
ISS=$(kubectl get --raw /.well-known/openid-configuration |
  jq -er '.issuer | select(type == "string" and length > 0)') ||
  { echo "OIDC discovery returned no valid issuer" >&2; exit 1; }
kubectl get --raw /openid/v1/jwks > /tmp/jwks.json
lambda_curl "https://cloud.lambda.ai/api/v1/oidc-providers" \
  -H 'Content-Type: application/json' \
  -d "$(jq -n --arg iss "$ISS" --arg lrn "lrn:iam:identity:$SID" \
        --arg subject "$SUBJECT" --slurpfile jwks /tmp/jwks.json \
        '{issuer_url:$iss, jwks:$jwks[0],
          trusts:[{identity_lrn:$lrn, subject:$subject}]}')"
echo "identityLRN: lrn:iam:identity:$SID"

The printed LRN is what goes in the chart value above.

Built-in role	Grants	Needed for
`maintenance-api-reader`	`support:maintenance-event:read`	Polling the maintenance-events API

The role is workspace-scoped, so the identity only sees events for $WS — which is why workspaceId should name that same workspace.

Verifying

The identity is attached when the pod is created, so it only appears on pods created after the annotation lands. Restart the deployment if you added it to a running install.

(
  set -euo pipefail
  NAMESPACE=$(
    kubectl get deployments --all-namespaces \
      -l app.kubernetes.io/name=csp-health-monitor -o json |
      jq -er '
        .items
        | if length == 1 then .[0] else error("expected exactly one csp-health-monitor Deployment") end
        | .metadata.namespace
      '
  )
  # The identity the pod received.
  kubectl -n "$NAMESPACE" get pod -l app.kubernetes.io/name=csp-health-monitor \
    -o jsonpath='{.items[0].spec.containers[0].env[?(@.name=="LAMBDA_IDENTITY_LRN")]}'
  # Which credential the monitor selected, and the workspace it polls.
  kubectl -n "$NAMESPACE" logs -l app.kubernetes.io/name=csp-health-monitor | grep authMode
)

Expect authMode=workload-identity. If it says api-key, the pod never received an identity — check that the annotation is on the ServiceAccount and that the pod was created after it landed.

Failures in the exchange itself surface on the first poll, not at startup, because the token is minted lazily. When the exchange endpoint rejects the token it returns 401 with no detail by design, so an unauthenticated caller cannot probe for which identities exist; that means a missing trust, a wrong identity LRN and a disabled account all look identical from the client. Check the trust and the identity LRN first.

Akamai Configuration

Polls the Akamai Cloud account maintenance API (GET /account/maintenance) and raises a health event for each event affecting a node in this cluster. The account lists maintenance for every instance in it, so events are matched to nodes by the Linode instance ID in the event’s entity, and events for instances that are not nodes of this cluster are ignored. Nodes must carry spec.providerID in the form linode://<instanceID>, which the Linode cloud controller manager sets on LKE nodes.

global:
  cspHealthMonitor:
    enabled: true
csp-health-monitor:
  cspName: "akamai"
  akamaiApiTokenSecret:
    name: "akamai-api-token"
    key: "LINODE_TOKEN"
  configToml:
    clusterName: "my-cluster"
    akamai:
      pollingIntervalSeconds: 60

Which maintenance events quarantine a node

Every maintenance type the API reports (reboot, cold_migration, live_migration, migrate and power_off_on) quarantines the node. A GPU workload cannot live-migrate, so live_migration is not treated as harmless. The type and the maintenance policy are kept in the event metadata for reference.

Events whose description is emergency are quarantined on the next poll, without waiting for the trigger window. All others are quarantined ahead of the event’s when time, within triggerQuarantineWorkflowTimeLimitMinutes. Notice varies by policy, from a few hours for a live migration to seven days for a power off and on, so size that window to the notice you need to drain a node.

The API does not report an event’s last-updated time, so the monitor compares a fingerprint of the status, type and schedule instead, and writes an event again only when one of them changes. A cancellation or a reschedule therefore still takes effect.

Cancelled maintenance

The monitor does not rely on the API to return a canceled status. It treats a pending or scheduled maintenance of this cluster that is no longer listed in two consecutive polls as cancelled, and releases the node through a CANCELLED event. The monitor uses the event it stored earlier as the record of why the node was quarantined. A poll that fails never counts as an absence. A list that has fewer entries than the total the API reports is an error, so it never counts as an absence either. A maintenance that is started, in-progress or completed is never released this way. The event carries the check name CSPMaintenance, so only the maintenance fault is cleared and any other fault on the node keeps it quarantined. Two rules keep the monitor from releasing a node too early. First, if the whole list is empty, the monitor waits for 10 consecutive polls, because a lasting empty list is more likely a fault on the way to the API than a cancellation of every maintenance. Second, the monitor does not release a node while another stored pending, scheduled, started or in-progress event of this cluster is on the same node, because the healthy event clears the maintenance check of the node and not one event. An event for an instance that is still listed but is no longer a node of this cluster is not cancelled. It stays stored until the API stops listing it. Without a datastore the monitor cannot detect a cancellation and logs one warning.

apiEndpoint

Optional. Base URL of the Akamai Cloud API, version included. Defaults to https://api.linode.com/v4, so the happy path needs no override. Must be https, and must not carry userinfo, a query string, or a fragment. The host must be api.linode.com and the path must be /v4, so any other host or path fails at startup rather than on the first poll.

pollingIntervalSeconds

How often the maintenance API is polled. Minimum 30.

akamaiApiTokenSecret

Secret holding the Akamai Cloud API token, injected as LINODE_TOKEN, the variable the Linode CLI and SDKs read. name is required when cspName is akamai; the install fails without it. Create the Secret in the release namespace before the install, because the pod cannot start without it. key defaults to LINODE_TOKEN. The monitor only lists maintenance events, so give it a read-only token with the account:read_only scope. A restricted identity that Akamai limits by permission needs the list_maintenances permission. The legacy read_only grant also authorizes the endpoint, so a read-only token needs no extra role. Do not share the Janitor Provider’s token, which can reboot and delete instances. The pod reads the token from its environment when it starts, so after you rotate the Secret, restart the monitor pods, for example with kubectl rollout restart deployment/csp-health-monitor -n nvsentinel.

CSP-Specific IAM Requirements

Each cloud provider handles IAM identity for the CSP Health Monitor differently:

Provider	IAM Identity Configuration	Naming Flexibility
GCP	`gcp.gcpServiceAccountName` — User provides any GCP Service Account name. The ServiceAccount annotation is built as `{SA_NAME}@{PROJECT}.iam.gserviceaccount.com`.	Fully flexible. No naming convention enforced.
AWS (EKS)	`aws.iamRoleName` (optional) — User provides a custom IAM role name. If omitted, the role name defaults to `{CLUSTER_NAME}-nvsentinel-health-monitor-assume-role-policy`.	Flexible when `iamRoleName` is set. The default convention imposes a 19-character cluster name limit (AWS IAM role names max 64 chars, default suffix is 45 chars).
Lambda	Either a static API key in `lambdaApiKeySecret`, or `lambdaWorkloadIdentity.identityLRN` — the ServiceAccount is annotated with `lambda.ai/identity-lrn` and the pod receives a short-lived token instead of a stored key.	Fully flexible. The identity is named by LRN, so no naming convention is enforced.
Akamai	A static API token in `akamaiApiTokenSecret`, injected as `LINODE_TOKEN`. There is no workload identity, so the token is stored in the cluster.	Not applicable. Use a read-only token.

Recommendation for EKS users: If your cluster name is longer than 19 characters, always set aws.iamRoleName explicitly and create the corresponding IAM role with that name. See IAM Setup for detailed instructions.

Advanced Configuration

Out-of-Cluster Monitoring

For monitoring a tenant cluster from a separate management cluster:

csp-health-monitor:
  configToml:
    # Path to kubeconfig for tenant cluster
    kubeconfigPath: "/etc/kubeconfig/tenant-cluster.yaml"

When kubeconfigPath is set, the monitor uses the specified kubeconfig to connect to the tenant cluster’s Kubernetes API for node mapping. If empty, uses in-cluster config.

Resources

Configure resource requests and limits for the main container and sidecar.

csp-health-monitor:
  # Main container resources
  resources:
    limits:
      cpu: "1"
      memory: "1Gi"
    requests:
      cpu: "200m"
      memory: "256Mi"
  
  # Sidecar (Quarantine Trigger Engine) resources
  quarantineTriggerEngine:
    resources:
      limits:
        cpu: "500m"
        memory: "512Mi"
      requests:
        cpu: "100m"
        memory: "128Mi"

Scheduling

Configure pod placement using node selectors, tolerations, and affinity rules.

csp-health-monitor:
  nodeSelector:
    node-role.kubernetes.io/control-plane: ""
  
  tolerations:
    - key: "node-role.kubernetes.io/control-plane"
      operator: "Exists"
      effect: "NoSchedule"
  
  affinity: {}