Interrupt Flow and Ordering

This document explains how NodeWright handles packages that require interrupts and the specific ordering of operations to ensure safe and reliable execution.

Overview

When a package requires an interrupt (such as a reboot or service restart), NodeWright follows a specific sequence to ensure that workloads are safely evacuated from the node before any potentially disruptive operations occur.

Interrupt Flow Sequence

For packages WITH interrupts:

1. Cordon - Node is marked as unschedulable to prevent new workloads from being scheduled
2. Wait - System waits for any conflicting workloads to naturally complete or be rescheduled
3. Drain - Remaining workloads are gracefully evicted from the node
4. Apply / Upgrade (if upgrading) - Package installation/upgrade operations are executed
5. Config - Configuration and setup operations are performed
6. Interrupt - The actual interrupt operation (reboot, service restart, etc.) is executed. A node restart is complete only after a later agent invocation observes that the host boot ID changed.
7. Post-Interrupt - Any cleanup or verification operations after the interrupt

Uninstalling a package with an interrupt goes through the same cordon, wait and drain steps first. Its uninstall runs only once the node has drained, and is followed by its uninstall-interrupt stage.

Before requesting a node restart, the agent writes a pending marker containing the current host boot ID. If reboot exits successfully before shutdown reaches the agent, the agent waits for up to 10 seconds for shutdown to terminate the process. Remaining alive for the full window is reported as a failure because the reboot was enqueued but did not take effect. After the node returns, the next invocation compares the pending boot ID with the current value. A changed boot ID promotes the pending marker to complete and allows post-interrupt work; an unchanged boot ID removes the stale marker and retries the restart.

When the shutdown terminates the reboot command first, the agent keeps the pending marker and exits successfully, and the container is not restarted. When the shutdown kills the agent first, the container dies mid-run, whatever the agent exits with. Like any failed attempt, that shows on the package’s state but does not mark the node erroring: the interrupt Job restarts the container in place, and once the node returns that invocation completes the stage or retries the restart. The node is marked erroring only when the Job fails: at the stage timeout, or, for every interrupt except a reboot, when it fails again after JOB_BACKOFF_LIMIT restarts. A reboot’s restarts are never counted, so a failing reboot is bounded only by the stage timeout. See A failing interrupt for how long that takes, how to spot one, and how to end it sooner.

For packages WITHOUT interrupts:

1. Apply / Upgrade (if upgrading) - Package installation/upgrade operations are executed
2. Config - Configuration and setup operations are performed

For several packages with interrupts on one node:

When more than one package with an interrupt is runnable on a node at the same time, that is, every package each of them depends on is already complete, each goes through its own apply/upgrade and config, but the node is interrupted once for all of them:

1. The operator merges their interrupts into one, as described in Interrupts coalesce, and gives it to the first package, in name order, whose interrupt has the highest rank.
2. Each other package that finishes its config is marked skipped at its interrupt stage and waits.
3. The merged interrupt runs once every other package with an interrupt has been marked skipped at its interrupt stage and no package Job is running on the node. Until then the package it belongs to waits at its interrupt stage. On a serial node the operator moves past it to the packages it waits for.
4. When it completes, the skipped packages are marked complete at interrupt, and every package goes on to its post-interrupt.

A package that is erroring before its interrupt does not hold the merged interrupt. Neither does a package being uninstalled: its uninstall-interrupt runs without waiting for the others. A package that becomes runnable only after the merged interrupt has started, because a package it depends on completed then, is not part of it. To interrupt the node once per package instead, order them with dependsOn: a package does not start until the packages it depends on are complete, interrupt included.

Why This Order Matters

The cordon → wait → drain → apply/upgrade → config → interrupt sequence is critical for several reasons:

Safety First

• Workloads are safely removed before any potentially disruptive operations
• Prevents data loss or service interruption for running applications
• Ensures the node is in a clean state before package operations begin

Use Cases

This ordering is particularly important for scenarios such as:

• Kernel module changes: Unloading kernel modules while workloads are present could cause system instability
• GPU mode switching: Changing GPU from graphics to compute mode requires exclusive access
• Driver updates: Hardware driver changes need exclusive access to the hardware
• System reboots: Obviously require all workloads to be evacuated first

Example Scenario

Consider a package that needs to unload a kernel module, perform some operations, and then reboot:

apiVersion: nodewright.nvidia.com/v1alpha1
kind: NodeWright
metadata:
  name: gpu-mode-switch
spec:
  packages:
    gpu-driver:
      version: "1.0.0"
      image: "example/gpu-driver"
      interrupt:
        type: "reboot"

Flow:

1. Cordon: Node becomes unschedulable
2. Wait: Any non-interrupt workloads are given time to complete
3. Drain: Remaining workloads are evicted
4. Apply: GPU driver package operations run (unload old module, install new)
5. Config: Configuration files are updated
6. Interrupt: System reboots to complete the driver change
7. Post-Interrupt: Verification that the new driver is loaded correctly

Technical Implementation

The interrupt flow is managed by the ProcessInterrupt and EnsureNodeIsReadyForInterrupt functions in the NodeWright controller, which:

• Check for conflicting workloads using label selectors
• Coordinate the cordon and drain operations
• Ensure the node is ready before proceeding with package operations
• Handle the timing and sequencing of all stages

Cordon Is Durable Before Drain Starts

The cordon and the drain are deliberately split across two reconcile passes. The operator applies the cordon in memory and writes it to the API server at the end of the pass, together with every other node it selected. Draining does not begin until a later pass observes spec.unschedulable already set on the node.

This matters because eviction is what triggers a replacement pod. If the operator evicted in the same pass that first cordoned the node, spec.unschedulable would still be local to the controller, and the scheduler could place the replacement back onto a node that is about to be interrupted.

The split costs one reconcile per drain cycle, not one per node: a single pass still cordons every node it selected, so a batch of nodes is cordoned together and then drained together on the following pass.

The exception is a cordon some other party already placed: it is already in the API, so the drain starts in the same pass. See Cordons NodeWright Did Not Create.

Shared Cordon Ownership

Each NodeWright that cordons a node records ownership with a nodewright.nvidia.com/cordon_<nodewright-name> annotation. When that NodeWright completes, it removes only its own cordon annotation. The node is marked schedulable only after no nodewright.nvidia.com/cordon_* annotations remain, so one NodeWright cannot uncordon a node that another NodeWright is still preparing for interrupt work. Additionally, if a persistent cordon was previously applied from a NodeWright which set runtimeRequiredCordonAfter to true, the cordon will persist even after all nodewright.nvidia.com/cordon_* annotations are removed.

Other NodeWright annotations, such as status_*, nodeState_*, and version_*, do not keep a node cordoned. Only the cordon_* annotation family and runtimeRequiredCordon participate in shared cordon ownership.

Cordons NodeWright Did Not Create

A NodeWright only takes ownership of a cordon it created, or one another NodeWright holds. If a node is already unschedulable when a NodeWright reaches its interrupt, and no cordon_* or runtimeRequiredCordon annotation is present, the cordon belongs to something else: a health-check controller, an autoscaler, or a kubectl cordon. The NodeWright drains and interrupts under that cordon without writing its own cordon_<nodewright-name> annotation, and leaves the node unschedulable when it completes. Releasing the cordon is up to whoever placed it. Because the cordon is already in the API, the drain starts in the same pass rather than waiting one.

If the other party uncordons the node while the NodeWright is still working, the NodeWright cordons it again the next time it passes its cordon-and-drain check, and owns that cordon from then on, releasing it on completion as usual. It passes that check on every reconcile while one of its interrupt packages has a stage running. An uncordon just before the interrupt stage starts is not caught: the interrupt Job is created without another cordon check, so the interrupt runs on a schedulable node. The same is true when someone removes NodeWright’s own cordon at that moment.

This only covers an external cordon placed before the NodeWright cordons. If a NodeWright cordons first and another controller cordons the same node afterwards, spec.unschedulable was already set and there is nothing to tell the two apart, so the NodeWright still uncordons the node on completion.

With REAPPLY_ON_REBOOT=true, a reboot resets the node’s NodeWright state so the packages run again, but keeps the cordon_<nodewright-name> annotation. A NodeWright’s own interrupt: reboot is a reboot too, and the node is still cordoned when the reset runs; keeping the annotation is what lets the reapply release that cordon when it completes.

A reboot reset also clears the previous drain start time and recorded blockers, even if the first package has not started yet. The next drain gets a fresh timeout. If the reset lands but recording the new boot ID fails, retries preserve the reset state instead of repeatedly clearing the version stamp.

If deleting a previous-boot Job fails, the operator retries cleanup and leaves that NodeWright’s Jobs on the affected node out of validation and resumption for the pass. Enabled NodeWrights keep the node out of new-stage selection until cleanup succeeds; other nodes can continue within their existing rollout budgets. A disabled NodeWright’s pending cleanup does not block enabled NodeWrights on the same node. The boot ID stays pending until cleanup succeeds.

Orphaned Cordon Recovery

If a NodeWright is force-deleted in a way that bypasses finalizer cleanup, or cleanup fails after the node was cordoned, its cordon_<nodewright-name> annotation can be left behind. That stale annotation will keep the node unschedulable from the operator’s point of view until it is removed.

Use kubectl nodewright reset <nodewright-name> --confirm to clear NodeWright metadata for affected nodes that still have nodeState_<nodewright-name> annotations, or kubectl nodewright node reset <node-name> --nodewright <nodewright-name> --confirm for a specific node. These reset commands never clear spec.unschedulable. On nodes the NodeWright still selects they also keep its cordon_<nodewright-name> annotation, so the re-run releases the cordon when it completes; without the annotation NodeWright would treat the cordon as one it did not create and leave it. Where no re-run will come, because the NodeWright is gone or its nodeSelector no longer matches the node, the reset commands remove the annotation, and the node stays cordoned until you uncordon it. If the CLI cannot read the NodeWright it keeps the annotation and warns; re-run the reset once the lookup works. After the stale cordon annotation is removed, if no other nodewright.nvidia.com/cordon_* annotations remain, check for the runtimeRequiredCordon annotation before running kubectl uncordon <node-name>. If the annotation is absent, uncordoning is a safe recovery step. If the annotation is present, it means a NodeWright intentionally applied a persistent cordon, and kubectl uncordon is not a recovery step unless removing the persistent cordon is intentional.

Against an operator after v0.19.1, a kubectl nodewright CLI at v0.4.0 or earlier removes the annotation even where the NodeWright will re-run. After a reset with one of those, the re-run cannot release the cordon, so uncordon the node once it completes, subject to the same runtimeRequiredCordon check. Operators at v0.19.1 and earlier adopt the cordon again on the re-run, so the older CLI needs no extra step there.

Drain Configuration

Interrupt-enabled NodeWrights can tune drain behavior with spec.drainConfig. Unset fields preserve the operator’s existing behavior:

apiVersion: nodewright.nvidia.com/v1alpha1
kind: NodeWright
metadata:
  name: gpu-mode-switch
spec:
  drainConfig:
    disableEviction: false
    deleteEmptyDirData: true
    force: true
    ignoreDaemonSets: true
    timeout: 10m
    gracePeriod: 30s

The fields map to Kubernetes drain behavior:

• disableEviction: when true, pods are deleted directly instead of evicted. This bypasses PodDisruptionBudgets. The default is false, so the eviction API is used.
• deleteEmptyDirData: when false, pods with emptyDir volumes block drain. The default is true.
• force: when false, pods without a managing controller block drain. The default is true.
• ignoreDaemonSets: when true, DaemonSet-managed pods are skipped during drain. The default is true.
• timeout: bounds how long a node may spend draining, measured from drain start — the first pass that finds the node not yet drained, which is before any evict or delete is issued — until the node is drained. Unset or zero means no timeout, so a pod that never finishes terminating holds the node in in_progress indefinitely. When the timeout expires, the node is marked erroring and package stages do not proceed on that node.
• gracePeriod: overrides the grace period used for eviction or direct deletion. Unset uses each pod’s own terminationGracePeriodSeconds.

The operator also skips pods that tolerate the node.kubernetes.io/unschedulable taint, mirror/static pods, and its own package pods. Package pods are identified by the nodewright.nvidia.com/name and nodewright.nvidia.com/package labels stamped on every one of them, so they remain exempt even if an admission controller rewrites or strips their tolerations. The label exemption only applies to pods in the operator’s own namespace, so workloads elsewhere cannot opt out of drain by copying the labels. These exclusions are not user-configurable.

No namespace is exempt. Like kubectl drain, the operator drains pods in kube-system, such as CoreDNS or metrics-server, under the same rules as any other pod. They go through the eviction API, so drain never takes more replicas than a PodDisruptionBudget covering them allows. A workload with no PDB, or with one that allows every replica to go, gets no such protection, and disableEviction: true bypasses PDBs here as everywhere else. force: false or deleteEmptyDirData: false makes an unmanaged or emptyDir pod there block drain, as it would elsewhere. A blocker that never clears, such as a kube-system PDB whose minAvailable equals the replica count, holds the drain until timeout, or indefinitely if no timeout is set. Operator releases up to v0.20.0 skipped every kube-system pod instead.

Compared to earlier releases, the default drain filter now follows Kubernetes matching more closely: the unschedulable toleration check uses Kubernetes ToleratesTaint semantics, DaemonSet pods are identified from the controller owner reference, and mirror/static pods are ignored.

podNonInterruptLabels remains a pre-drain barrier. Matching pods must finish or move away before the operator starts the configurable drain step. The node is cordoned before entering this barrier and remains cordoned throughout the wait so no replacement workloads can schedule on it. This wait is unbounded and drain timeout does not apply. While this barrier holds, administrators will see a Blocked condition with reason NonInterruptPodsRunning reporting which nodes are held, along with Warning events on both the NodeWright and affected Node objects detailing the hold.

What a held node costs the rollout

A node held at this barrier has been picked for its batch but has not run its package, and these follow from that:

• The node does not reliably read waiting. The barrier gates only the package that has an interrupt, so a held node reads waiting only until any of its packages has been dispatched, and that can happen during the hold on a first apply too. A package without an interrupt, or one the interrupt package depends on that has not yet run, is dispatched on the cordoned node, and creating its Job sets the node in_progress; nothing moves it back. With a driver package (interrupt: reboot) and a tuning package (no interrupt) on a first apply, the held node reads in_progress. A serial NodeWright stops at the held package instead, so nothing else is dispatched during the hold. On a version bump, HandleVersionChange sets every changed node to in_progress before the barrier runs, so a held node reads in_progress there as well. The NodeWright’s Ready condition follows the roll-up of all its nodes, which reports in_progress (reason Progressing) whenever any node is in progress. So Ready reads Waiting (for example 0/3 nodes complete, 3 waiting (...)) only while every node is still waiting, which is the case only for a first apply where nothing has been dispatched on any node. A hold is therefore not identifiable from node status or Ready. Check the Warning event on the Node, which is emitted while the barrier holds the node and does not depend on any condition, and the Blocked condition with reason NonInterruptPodsRunning. That condition has two limits:
  • A NodeWright has one Blocked condition, and updateDrainBlockedCondition returns early while another reason, such as DependencyUninstalled, holds it. A node can stay held without NonInterruptPodsRunning being published.
  • It lists every selected node that has an interrupt package waiting on a drain and a running protected pod, whether or not the node has been picked for a batch. With an interruptionBudget of count: 1 and protected pods on ten nodes it reads 10 nodes blocked while one node is cordoned. The held nodes are the cordoned ones that carry the Warning event, and the others are not yet held.
• The node keeps its batch slot. It stays in status.nodePriority until it completes, so its compartment does not start its next batch while the node is held; see batch membership. With one node per batch (an interruptionBudget of count: 1, say) no other node in that compartment progresses for as long as the workload runs. With the default single compartment that is the whole rollout. Other compartments are not affected.
• The node stays cordoned for the whole wait, as described above, so capacity on a held node is unavailable for as long as the protected workload runs.
• No per-node hold start is recorded. Two timestamps approximate it: the node’s entry in status.nodePriority (when it was picked for its batch) and the Blocked condition’s lastTransitionTime (when that condition last turned true, which may predate this node’s hold).

The wait ends only when no pod matching podNonInterruptLabels is Running or Pending on the node. The operator then attempts the configured drain and runs the package only after the drain succeeds; other drain blockers can delay or prevent package execution. The remaining nodes of the compartment follow as their batches start.

When Drain Is Complete

Drain is complete once the pods it selected have actually left the node, not once their evictions were accepted. A pod that has an eviction or delete in flight keeps a deletionTimestamp while the kubelet works through its terminationGracePeriodSeconds, and the operator treats it as still present: the node stays in in_progress, no further evict or delete is issued for it, and the interrupt does not run until the pod object is gone. This matches kubectl drain, which waits for the pods it selected to disappear.

Only pods the operator selected for eviction or deletion are waited on. The exclusions above still apply while a pod is terminating, so a DaemonSet pod mid-rollout, a mirror pod, or one of the operator’s own package pods never holds up a drain.

Because drain now waits out termination, a workload’s terminationGracePeriodSeconds is on the critical path for every interrupt, and a pod that cannot finish terminating — a stuck finalizer, an unresponsive kubelet — blocks the node’s interrupt until it is cleared. Set spec.drainConfig.timeout to bound that wait; there is no default.

DrainBlocked Condition

A DrainBlocked condition is set on the NodeWright whenever one or more selected nodes have a drain that cannot currently make progress. It names the blocked nodes and, where the blocker is a PodDisruptionBudget, includes the apiserver’s own message verbatim:

- type: DrainBlocked
  status: "True"
  reason: PodDisruptionBudget
  message: "2/5 nodes blocked draining (node-a, node-c); default/web-0 on node-a:
            The disruption budget web-pdb needs 3 healthy pods and has 3 currently"

reason is one of PodDisruptionBudget, UnmanagedPod, EmptyDirData, or MultipleCauses when more than one kind of blocker is present across the affected nodes. The condition clears automatically once every previously blocked node has drained, or no longer has a package with an interrupt waiting on a drain — no action is required beyond removing the underlying blocker. Uninstalls count: a package with an interrupt drains the node before its uninstall runs, just as it does before an apply, and a drain blocked there is reported the same way. A node whose nodeState_<nodewright-name> annotation cannot be parsed keeps the blockers last recorded for it until the annotation is repaired; NodeStateMalformed names that node.

DrainBlocked is a separate condition type from Blocked, whose reasons are DependencyUninstalled and the NonInterruptPodsRunning hold described above, so a NodeWright can report both at the same time.

A PodDisruptionBudget rejection is treated as a self-resolving wait state, not a reconcile error: it does not abort the reconcile pass for the remaining nodes. While a PDB keeps refusing, the operator waits 30 seconds after each refusal before evicting on that node again, and in between the node keeps reporting the blockers from its last refused attempt whose pods are still blocking. A throttled pass with none of those left for its NodeWright on the node attempts the eviction instead: for example when another NodeWright’s attempt was the one refused, or when the refused pods have since left the node or begun terminating. Only the attempt is throttled: the reconcile still runs every 2 seconds, so other nodes and other NodeWrights are unaffected.

A blocking pod that goes away is noticed within one 2-second pass, because every pass checks whether the node has drained before it considers evicting. A PDB that regains headroom can only be discovered by evicting again, so it is picked up by the next attempt, up to 30 seconds later. Likewise, a spec.drainConfig change made mid-drain (disableEviction, force, deleteEmptyDirData) takes effect at the node’s next attempt, up to 30 seconds later. spec.drainConfig.timeout is checked before the throttle on every pass, so it fires on time. The time of the last refused attempt lives only in the operator’s memory, so an operator restart or leader change retries the eviction at once.

Unmanaged pods (force: false) and emptyDir pods (deleteEmptyDirData: false) are also reported in DrainBlocked, though — unlike a PDB rejection — these were already wait states before this condition existed; DrainBlocked only makes them visible without reading operator logs.

Recovering From a Drain Timeout

When spec.drainConfig.timeout expires, the operator records a DrainTimeout warning event, marks the node and NodeWright erroring, and leaves the node cordoned. The operator stops issuing further evict/delete actions while the blocking condition remains, so package stages do not proceed on that node.

DrainTimeout and DrainBlocked answer different questions: DrainBlocked names what is currently preventing progress and clears itself once the blocker is gone, with no configured limit on how long that may take. DrainTimeout only fires once spec.drainConfig.timeout is set and elapses — which, for a PDB blocker, is now the only path that turns a stuck drain into an erroring node; the PDB rejection itself no longer does so directly.

To recover, remove the underlying blocker first, such as a PDB with zero allowed disruptions, an unmanaged pod when force: false, or an emptyDir pod when deleteEmptyDirData: false. Then reset the failed rollout metadata:

kubectl nodewright reset <nodewright-name> --confirm

For a single node, use:

kubectl nodewright node reset <node-name> --nodewright <nodewright-name> --confirm

If the blocker clears after the timeout without a reset, a later reconcile can observe the node as drained and continue from current cluster state. Reset is still the recommended recovery workflow in production because it explicitly clears the erroring status, drain-start metadata, and batch state before retrying. It keeps the NodeWright’s cordon annotation, so the node stays cordoned through the retry and is released when the retry completes. If the blocker is still present after reset, the drain will time out again.

Best Practices

• Always test interrupt-enabled packages in non-production environments first
• Use appropriate podNonInterruptLabels selectors to identify important workloads that should block interrupts
• Consider the impact of node cordoning on cluster capacity
• Monitor package logs during interrupt operations for troubleshooting
• Use Grafana dashboards to monitor interrupt operations and track package state transitions across your cluster (see docs/observability/metrics.md for dashboard setup and configuration)