> This page is for NVSentinel, version Latest · v1.28.0 (default). > For other versions, use one of these documentation indexes: > - Latest · v1.28.0 (default): https://docs.nvidia.com/nvsentinel/v1.28.0/llms.txt > - v1.27.0: https://docs.nvidia.com/nvsentinel/v1.27.0/llms.txt > - v1.26.0: https://docs.nvidia.com/nvsentinel/v1.26.0/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/nvsentinel/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/nvsentinel/_mcp/server. # Lifecycle Manager Configuration ## Overview The Lifecycle Manager hosts the controllers that manage a node's transitions in and out of service. It runs two independent controllers, each with its own feature gate: - **ValidationRequest controller** — runs validation tests against a node and keeps it gated until the tests pass. It is enabled by default. - **MaintenanceRequest controller** — prepares a node for upcoming maintenance, then releases it when the request is deleted. It is disabled by default. The two are unrelated at runtime. Enabling one does not require the other. ## Module Enable/Disable ```yaml global: lifecycleManager: enabled: true ``` --- ## MaintenanceRequest Controller A `MaintenanceRequest` is a cluster-scoped resource carrying a health event. Creating one publishes that event into the NVSentinel pipeline so the node is prepared; deleting one publishes the matching clearing event so the node is released. **Presence means active, absence means cleared** — deleting the resource is the instruction, not merely cleanup. ### Enabling ```yaml lifecycle-manager: controllers: maintenanceRequest: enabled: true ``` The controller is disabled by default. Turning this on changes more than the reconciler. It also: - registers the mutating and validating webhooks for `MaintenanceRequest` - grants RBAC for the resource, its status and its finalizers - mounts the platform-connector socket into the pod - projects a ServiceAccount token into the pod, when authentication is enabled - adds this ServiceAccount to platform-connector's cross-node allowlist While the gate is off, the validating webhook rejects any `MaintenanceRequest` at admission rather than letting it be created and silently ignored. > **Operational note:** the allowlist lives in platform-connector's ConfigMap, and that > ConfigMap's checksum is stamped on the DaemonSet pod template. Enabling this controller > therefore rolls platform-connector across every node. This is true of enabling any > cross-node publisher, not specific to this one, but it is worth scheduling deliberately. By default the controller publishes through the socket. With `lifecycle-manager.publishTo: deployment`, it publishes to the [deployment platform connector](/nvsentinel/configuration/platform-connectors#deployment-platform-connector) instead. The chart then sets the `HEALTH_PUBLISH_*` environment variables on the pod. The controller presents the same projected token as on the socket path. The socket mount stays in the pod, but the controller does not use it. ### Authentication This is the part most easily missed. A `MaintenanceRequest` may name **any** node, but lifecycle-manager is a Deployment running on one node. platform-connector pins a publisher to its own node unless that publisher is on the cross-node allowlist and presents a valid token, so both halves are required: - the pod is given a projected ServiceAccount token, minted for `global.platformConnectorAuth.audience`, and passed to the process with `--platform-connector-token-path` - the umbrella chart adds `system:serviceaccount::lifecycle-manager` to `AuthCrossNodeServiceAccounts` Both are derived from the feature gate above, so there is nothing extra to configure. Do **not** add lifecycle-manager to `global.platformConnectorAuth.crossNodeServiceAccounts` by hand; that list is for publishers this chart does not ship, and a hand-written entry with the wrong namespace renders cleanly and then has every event rejected at runtime. With `global.platformConnectorAuth.enabled: false`, no token is projected and no allowlist entry is added, which matches the unauthenticated behaviour of every other publisher. See [Authentication](./authentication.md) for the full node-binding model, and the `platform_connector_auth_violations_total` metric to watch if events are being rejected. ### Quarantine Rule Publishing the event is only half of the outcome. Whether the node is actually cordoned depends on fault-quarantine having a ruleset that matches the event. The bundled `MaintenanceRequest ruleset` in the fault-quarantine chart matches on the `maintenanceRequestName` metadata key, which the mutating webhook stamps on every event this controller publishes. Removing or narrowing that ruleset leaves MaintenanceRequests accepted but inert: the resource is created, its condition goes green, and the node is never cordoned. ### Example ```yaml apiVersion: nvsentinel.dgxc.nvidia.com/v1 kind: MaintenanceRequest metadata: name: csp-maintenance-ip-10-0-31-7 spec: healthEvent: version: 1 agent: csp-maintenance-integration componentClass: Node checkName: ScheduledHostMaintenance nodeName: ip-10-0-31-7 isHealthy: false isFatal: false recommendedAction: RESTART_VM message: "Host maintenance scheduled by the cloud provider" ``` The webhook changes `agent` to `lifecycle-manager` before it stores the object. It preserves `csp-maintenance-integration` in `healthEvent.metadata.maintenanceRequestRequesterAgent`. ### Validation Rules The validating webhook rejects a request that: - omits `spec.healthEvent` or `spec.healthEvent.nodeName` - sets `isHealthy: true`, since an MR describes work starting, not a recovery - leaves `version` at zero - names a node that does not exist - sets `startTime` in the past - names a node that already has an open MR - changes any stored field of `healthEvent` after creation The mutating webhook fills in `id` and `generatedTimestamp` if absent. It sets the publishing agent to `lifecycle-manager`, preserves the supplied agent in `maintenanceRequestRequesterAgent`, and always sets the `maintenanceRequestName` metadata key. ### Status and Deletion The controller sets a `HealthEventEmitted` condition once the opening event has been accepted by platform-connector, and takes a finalizer on the resource. On deletion it publishes the clearing event, releases its leases, and only then drops the finalizer — so a failed clearing publish leaves the object in `Terminating` and retries, rather than losing the clear and leaving the node cordoned indefinitely. The controller uses two Lease objects in the lifecycle-manager namespace: - **Claim** (`mr-claim.`): the MR takes it before the opening event and holds it until deletion, so a node has at most one open MR. If a second MR for the node passes the webhook, the controller sets `HealthEventEmitted=False` with reason `Rejected` and does not retry it. If the node name is too long for a Lease name, the end of the name is replaced with a short hash. - **Janitor node lock** (``): the lock that janitor operations such as RebootNode share. The MR holds it only while it publishes the opening event, so the event does not start maintenance while a janitor operation runs. If the lock is held, the controller sets reason `Blocked` and retries every 30 seconds. The MR releases the lock right after it publishes, so the janitor operation that its event starts can take the lock. If the clearing event can never be delivered, the resource stays in `Terminating`. Removing the finalizer by hand is the escape hatch, at the cost of the node staying cordoned until an operator clears it.