NodeWright CLI

kubectl plugin for managing NodeWright deployments, packages, and nodes.

Overview

The NodeWright CLI (kubectl nodewright) provides SRE tooling for managing NodeWright deployments and their packages across Kubernetes cluster nodes. It supports inspecting node/package state, forcing re-runs, managing node lifecycle, and retrieving logs.

Compatibility

Required API Group

This CLI targets the nodewright.nvidia.com API group and the NodeWright kind, and it reads and writes nodewright.nvidia.com/* node annotations. It requires an operator that serves nodewright.nvidia.com; older Skyhook-only operators that serve just skyhook.nvidia.com and the Skyhook kind are unsupported. Upgrade the operator to a NodeWright-capable version before using this CLI.

Every cluster-backed command runs a preflight check on the served API groups. If the cluster serves only the legacy skyhook.nvidia.com group and not nodewright.nvidia.com, the command fails fast with a clear, actionable error naming both groups and telling you to upgrade to a NodeWright-capable operator, rather than a confusing NotFound.

Namespace resolution

The install namespace for new deployments moved from skyhook to nodewright with the rename. Namespaces cannot be renamed in place, and Helm cannot move a release between namespaces, so an install that predates the rename legitimately stays in skyhook indefinitely. The CLI therefore does not simply default to nodewright.

When --namespace is not passed, the CLI resolves the operator’s namespace in this order and caches the answer for the invocation:

1. nodewright — if a NodeWright operator Deployment is there, use it.
2. skyhook — if the operator is there instead, use it and print a one-line note to stderr saying the legacy namespace was used and that new installs default to nodewright. Pass --namespace skyhook to silence the note.
3. A cluster-wide Deployment sweep, for installs in some other namespace. This needs cluster-scoped list permission; if you don’t have it, the step is skipped silently rather than failing.
4. If nothing is found, nodewright is used so the command’s own lookup produces the specific error (for example, “no operator deployment found in namespace “nodewright"").

When --namespace is passed, it is used verbatim with no discovery and no note. Nothing else about the CLI is version-gated on this: the namespace is only used to locate the operator Deployment and the package pods.

The NodeWright and DeploymentPolicy CRDs are cluster-scoped, so --namespace never affects which CRs a command sees.

Minimum Operator Version

The CLI requires operator version v0.8.0 or later for full functionality of all commands.

Command Compatibility Matrix

Note on pause stop strength: the command works on every v0.8.0+ operator, but how hard it stops is version-dependent, so the column above is about availability only. On operators that run package stages as Jobs, pause suspends the stage that is currently executing. On v0.8.0+ operators that predate that change — and, briefly, for a stage still running as a pre-upgrade pod during the upgrade itself — pause blocks all new stage scheduling but lets an in-flight stage finish. (On v0.7.x and earlier the command does not exist at all; use spec.pause.) disable never stops in-flight work on any version. See Emergency Stop for the full semantics and for what resume re-runs.

Note on update-state and reset --package: These commands edit the nodewright.nvidia.com/nodeState_<nodewright> annotation in-place. The annotation’s map[string]PackageStatus shape has been stable since operator v0.7.5, and the CLI refuses to run against anything older. What has evolved across operator releases is the set of recognised stage and state values — for example uninstall and uninstall-interrupt were added in v0.16.0. Picking a stage/state your operator doesn’t recognise will leave the package in a state the operator can’t progress from. Confirm the stage/state values are valid for your operator before running these commands.

Breaking Change: Pause/Disable Mechanism

In operator versions v0.7.x and earlier, pausing and disabling a Skyhook was done via spec fields:

spec:
  pause: true  # Old method - no longer used by operator

Starting with v0.8.0, the operator uses annotations instead:

metadata:
  annotations:
    nodewright.nvidia.com/pause: "true"
    nodewright.nvidia.com/disable: "true"

The CLI’s pause, resume, disable, and enable commands set these annotations. If you’re running an older operator (v0.7.x or earlier), these commands will appear to succeed but the operator won’t recognize the annotations - you’ll need to edit the Skyhook spec directly using kubectl edit.

Installation

# Build from source
cd operator
make build-cli
# Install as kubectl plugin
cp bin/nodewright /usr/local/bin/kubectl-nodewright # or another directory in $PATH with write access
# Verify installation
kubectl nodewright version

Usage Structure

Basic Command Structure

kubectl nodewright [global-flags] <command> [subcommand] [flags] [arguments]

Global Flags

• -h, --help - Show help for any command
• --version - Show version information
• -n, --namespace - Kubernetes namespace the operator is installed in. When omitted, the CLI discovers it (see Namespace resolution); the fallback default is nodewright.
• -o, --output - Output format: table|json|yaml|wide
• -v, --verbose - Enable verbose output
• --dry-run - Preview changes without applying them
• --kubeconfig - Path to kubeconfig file

Commands

Version Command

Show plugin and operator versions.

# Show both plugin and operator versions
kubectl nodewright version
# Show only plugin version (no cluster query)
kubectl nodewright version --client-only
# With custom timeout
kubectl nodewright version --timeout 10s

Pause/Resume Commands

Control NodeWright processing state.

Note: Requires operator v0.8.0+. See Compatibility for details.

# Pause a Skyhook (stops processing new nodes)
kubectl nodewright pause my-skyhook
kubectl nodewright pause my-skyhook --confirm  # Skip confirmation
# Resume a paused Skyhook
kubectl nodewright resume my-skyhook

Disable/Enable Commands

Completely disable or re-enable a NodeWright.

Note: Requires operator v0.8.0+. See Compatibility for details.

# Disable a Skyhook completely
kubectl nodewright disable my-skyhook
kubectl nodewright disable my-skyhook --confirm
# Re-enable a disabled Skyhook
kubectl nodewright enable my-skyhook

Reset Command

Reset all package state for a NodeWright, causing re-execution from the beginning.

# Reset all nodes for a Skyhook (also resets batch state by default)
kubectl nodewright reset gpu-init --confirm
# Preview changes without applying (dry-run)
kubectl nodewright reset gpu-init --dry-run
# Reset nodes only, preserve deployment policy batch state
kubectl nodewright reset gpu-init --skip-batch-reset --confirm
# Reset only a single package across all tracked nodes
kubectl nodewright reset gpu-init --package pkg1:1.0 --confirm --skip-batch-reset

Note: By default, reset also resets the deployment policy batch state so the next rollout starts from batch 1, and clears node ordering state (NodeOrderOffset and NodePriority) so SKYHOOK_NODE_ORDER restarts from 0. Use --skip-batch-reset to preserve the existing batch and ordering state.

--package <name>[:<version>]

When --package is set, reset removes only the named package’s entry from each node’s nodeState annotation instead of removing the whole annotation:

• If <version> is supplied (e.g. pkg1:1.0), only entries whose recorded version matches are removed; nodes with the package at a different version are left untouched and reported in the command output.
• If the package was the last entry in the annotation, the entire annotation is removed (matching the behavior of a full reset).
• Batch state is deliberately not reset on this path regardless of --skip-batch-reset: restarting the full rollout from batch 1 to recover a single package would be disproportionate. Pair with kubectl nodewright deployment-policy reset explicitly if you also want to restart batches.

Update-State Command

Edit the recorded state of a single package on NodeWright-managed nodes. update-state performs a surgical edit to the per-node nodewright.nvidia.com/nodeState_<nodewright> annotation — replacing (or, with --add, inserting) one entry in the map[string]PackageStatus value. It is an administrator escape hatch for recovering from stuck rollouts and deliberately does not validate that the requested (stage, state) combination is one the operator could legally produce, nor does it gate destructive stages (uninstall, uninstall-interrupt) behind extra prompts.

kubectl nodewright update-state <skyhook-name> <package> <version> <stage> <state>

⚠️ Pause the NodeWright before running update-state.

update-state performs a read-modify-write on the node-state annotation and uses a merge patch with no resource-version check. The patch rewrites the entire annotation value, not just the targeted package’s entry — so a concurrent operator (or CLI) edit to any other package’s entry in the same annotation will also be clobbered. If the operator reconciles the node between the CLI’s read and write, the operator can immediately overwrite the manual edit — at best wasting the operation, at worst racing the operator into an inconsistent state. Always pause the NodeWright (kubectl nodewright pause <name> --confirm) before running this command, and resume only when finished.

update-state also requires the package + version to still be present in skyhook.Spec.Packages. It cannot edit an orphaned node-state entry left behind after the package was removed from the spec — use reset --package <name>[:<version>] for that.

# Mark pkg1@1.0 as complete on every node that already tracks this Skyhook
kubectl nodewright update-state gpu-init pkg1 1.0 config complete --confirm
# Same, but only on one node
kubectl nodewright update-state gpu-init pkg1 1.0 config complete --node worker-1 --confirm
# Narrow to a label-selected set of nodes
kubectl nodewright update-state gpu-init pkg1 1.0 interrupt in_progress -l role=gpu --confirm
# Preview the changes without writing them
kubectl nodewright update-state gpu-init pkg1 1.0 config erroring --dry-run

The global --dry-run flag is honored: the command prints the set of nodes it would patch and exits without writing.

By default update-state targets only nodes that already have a nodeState entry for the named NodeWright. If --node names a node that does not exist or has no state for the NodeWright, the command warns and skips that node rather than failing.

--add

--add creates a fresh nodeState entry on nodes that do not yet have one for the given <package>@<version> — useful for bootstrapping state on a node the operator has not visited yet, or for re-creating an entry that was manually deleted.

--add requires either --node or --selector so the scope of the creation is explicit. Without one of these flags, --add would apply to every node in the cluster that matches the NodeWright’s selector, which is far too broad for a creation operation — the CLI rejects this combination with an error.

If a targeted node already has an entry for <package>@<version>, --add warns and skips that node (use update-state without --add if you intend to overwrite the existing entry).

Migrating manifests to NodeWright

There is no dedicated migrate command: the conversion is a mechanical group and kind swap with no change to the spec body. Both Skyhook and DeploymentPolicy change API group to nodewright.nvidia.com/v1alpha1; Skyhook additionally changes its kind to NodeWright, while DeploymentPolicy keeps its kind.

For a source manifest, that is:

sed -e '/^ *apiVersion:/ s|skyhook\.nvidia\.com/|nodewright.nvidia.com/|' \
    -e 's|^\( *kind: *\)Skyhook[[:space:]]*$|\1NodeWright|' skyhook.yaml > nodewright.yaml

Rewrite apiVersion and kind only. A blanket s|skyhook\.nvidia\.com/|nodewright.nvidia.com/|g is tempting but wrong: it also rewrites key names inside the spec body that are yours, not the operator’s. nodeSelectors.matchLabels and podNonInterruptLabels refer to labels on your own nodes and pods; the rename does not touch those labels, so rewriting the selector points the CR at a key nothing carries and it silently matches nothing. It also makes the re-apply a real spec change rather than the no-op adoption of the mirrored object that the migration flow expects.

The one thing to rename by hand, if your manifest sets it, is the operator’s own lifecycle annotations on the CR: skyhook.nvidia.com/pause and skyhook.nvidia.com/disable become nodewright.nvidia.com/*.

Leave everything else alone. In particular, do not rewrite the runtime-required taint key here: it is the bare string skyhook.nvidia.com (no trailing slash), so a blanket group swap would corrupt it, and the taint on a node is not something a CR manifest edit can change. The default key did move to nodewright.nvidia.com, but the operator tolerates and removes both keys for the deprecation window, so an additionalTolerations entry naming the legacy key stays valid. Migrating it is a change to your provisioning config (autoscaler, node pool, --register-with-taints), not to your CRs; see docs/user-guide/runtime-required.md.

The operator’s mirror controller already converts the live objects in the cluster automatically; the edit above is only for the manifests in your source of truth (git / GitOps). See docs/getting-started/migration.md for the full flow.

Deployment Policy Commands

Manage deployment policy batch state.

Note: Requires operator v0.8.0+.

# Reset batch state for a Skyhook (starts rollout from batch 1)
kubectl nodewright deployment-policy reset gpu-init --confirm
# Preview what would be reset (dry-run)
kubectl nodewright deployment-policy reset gpu-init --dry-run
# Using the short alias
kubectl nodewright dp reset gpu-init --confirm

The deployment-policy reset command resets the batch processing state for all compartments in the specified NodeWright, including:

• Current batch number (reset to 1)
• Consecutive failure count
• Completed and failed node counts
• Stop flag
• Node ordering state (NodeOrderOffset and NodePriority) — SKYHOOK_NODE_ORDER restarts from 0

When to use:

• After a rollout completes and you want to start a new rollout fresh
• When batch processing is stuck and needs to be reset
• Before re-running a rollout with the same deployment policy

See Deployment Policy documentation for details on auto-reset configuration.

Node Commands

Manage NodeWright nodes across the cluster.

# List all nodes targeted by a Skyhook
kubectl nodewright node list --nodewright my-nodewright
kubectl nodewright node list --nodewright my-nodewright -o json
# Show all Skyhook activity on specific node(s)
kubectl nodewright node status worker-1
kubectl nodewright node status worker-1 worker-2
kubectl nodewright node status "worker-.*"  # Regex pattern
kubectl nodewright node status worker-1 --nodewright my-nodewright  # Filter by NodeWright
# Reset all package state on node(s)
kubectl nodewright node reset worker-1 --nodewright my-nodewright --confirm
kubectl nodewright node reset "node-.*" --nodewright my-nodewright --dry-run
# Ignore/unignore nodes from processing
kubectl nodewright node ignore worker-1
kubectl nodewright node ignore "test-node-.*"
kubectl nodewright node unignore worker-1

Node Status Columns

node status reports two cordon columns alongside the rollout state. They appear in the default table, in -o wide, and as fields in -o json / -o yaml:

Read them together to tell a transient cordon from one that is waiting on you:

• CORDONED=true, RUNTIME-REQUIRED-CORDON=false — the operator uncordons this node itself once every NodeWright holding a nodewright.nvidia.com/cordon_* annotation on it has finished, unless the cordon is one you applied yourself.
• CORDONED=true, RUNTIME-REQUIRED-CORDON=true — the cordon will not clear on its own. It survives later NodeWright interrupts and kubectl nodewright reset, and only an external actor can release it. See runtime-required.md for the patch that clears both the annotation and the cordon.

Against an operator that predates spec.runtimeRequiredCordonAfter the annotation is never set, so RUNTIME-REQUIRED-CORDON reads false on every node. That is accurate, not degraded — no such cordon can exist — so the columns need no operator version gate.

Node Flags

Package Commands

Manage NodeWright packages.

# Query package status across nodes
kubectl nodewright package status my-package --nodewright my-nodewright
kubectl nodewright package status my-package --nodewright my-nodewright --node worker-1
kubectl nodewright package status my-package --nodewright my-nodewright -o wide
# Force package re-run on specific nodes
kubectl nodewright package rerun my-package --nodewright my-nodewright --node worker-1
kubectl nodewright package rerun my-package --nodewright my-nodewright --node "worker-.*" --confirm
kubectl nodewright package rerun my-package --nodewright my-nodewright --node worker-1 --stage config
# Get package logs
kubectl nodewright package logs my-package --nodewright my-nodewright
kubectl nodewright package logs my-package --nodewright my-nodewright --node worker-1
kubectl nodewright package logs my-package --nodewright my-nodewright --stage apply
kubectl nodewright package logs my-package --nodewright my-nodewright -f  # Follow
kubectl nodewright package logs my-package --nodewright my-nodewright --tail 100

Package Flags

Help System

# General help
kubectl nodewright --help
# Command group help
kubectl nodewright node --help
kubectl nodewright package --help
kubectl nodewright deployment-policy --help
# Specific command help
kubectl nodewright node reset --help
kubectl nodewright package rerun --help
kubectl nodewright deployment-policy reset --help

Common Usage Patterns

Debugging a Failed Package

# 1. Check package status
kubectl nodewright package status my-package --nodewright my-nodewright -o wide
# 2. View logs for the failed package
kubectl nodewright package logs my-package --nodewright my-nodewright --node worker-1
# 3. Fix the issue, then force re-run
kubectl nodewright package rerun my-package --nodewright my-nodewright --node worker-1 --confirm

Node Maintenance

# 1. Ignore node before maintenance
kubectl nodewright node ignore worker-1
# 2. Perform maintenance...
# 3. Unignore and reset to re-run all packages
kubectl nodewright node unignore worker-1
kubectl nodewright node reset worker-1 --nodewright my-nodewright --confirm

Cluster-Wide Status Check

# List all nodes for a Skyhook
kubectl nodewright node list --nodewright my-nodewright
# Check status of all nodes
kubectl nodewright node status
# Check specific Skyhook across all nodes
kubectl nodewright node status --nodewright my-nodewright -o json

Resetting a Rollout

# 1. Full reset: nodes + batch state (starts everything fresh)
kubectl nodewright reset my-skyhook --confirm
# 2. Or reset only batch state (keep node state, restart batch progression)
kubectl nodewright deployment-policy reset my-skyhook --confirm
# 3. Or reset only nodes (keep batch progression)
kubectl nodewright reset my-skyhook --skip-batch-reset --confirm

Surgical Recovery

When a single package on a single node is wedged and a full reset is too disruptive, pause the NodeWright, edit the recorded state directly, then resume:

# 1. Pause so the operator doesn't clobber the manual edit
kubectl nodewright pause my-skyhook --confirm
# 2. Mark the wedged package as complete (or set whatever stage/state
#    you need to unblock the rollout)
kubectl nodewright update-state my-skyhook my-package 1.0 config complete \
    --node worker-1 --confirm
# 3. Resume processing
kubectl nodewright resume my-skyhook --confirm

For a single-package reset across all nodes (without disturbing the deployment policy batch state), prefer reset --package:

kubectl nodewright reset my-skyhook --package my-package:1.0 --confirm

Emergency Stop

Note: Requires operator v0.8.0+. For older operators, use kubectl edit skyhook my-skyhook and set spec.pause: true.

pause and disable stop different things. pause is the one that can halt work already running: on operators that run package stages as Jobs it suspends the executing stage, and on older ones it blocks new scheduling only — see the version note below. disable takes the NodeWright out of processing so no new work is scheduled, but it does not stop a stage already under way; that stage runs to completion.

They compose safely in either order: disabling a paused NodeWright leaves its suspended stages suspended rather than resuming them. Those stages resume only once both annotations are cleared — re-enabling on its own does nothing while the NodeWright is still paused. If you want everything stopped now, pause is the command.

# Pause all processing
kubectl nodewright pause my-skyhook --confirm
# Or prevent new work being scheduled
kubectl nodewright disable my-skyhook --confirm

How hard pause stops depends on the operator version. On operators that run package stages as Jobs, pause suspends the stage that is currently executing — its pod is signaled to stop and nothing new starts until you resume, at which point the stage re-runs from the start of its current phase (the agent skips already-completed steps). On earlier operators — and, briefly, for a stage still running as a pre-upgrade pod during an operator upgrade — a stage already in flight finishes its current run before pause takes hold; pause still blocks all new stage scheduling. If you need a running stage to stop immediately, confirm the operator executes packages as Jobs.

Output Formats

All status commands support multiple output formats:

# Table (default) - human-readable
kubectl nodewright node list --nodewright my-nodewright
# Wide - table with additional columns
kubectl nodewright node list --nodewright my-nodewright -o wide
# JSON - machine-readable
kubectl nodewright node list --nodewright my-nodewright -o json
# YAML - machine-readable
kubectl nodewright node list --nodewright my-nodewright -o yaml

Architecture

Package Structure

operator/cmd/cli/app/           # CLI commands
├── cli.go                      # Root command (NewSkyhookCommand)
├── version.go                  # Version command
├── reset.go                    # Reset command (nodes + batch state)
├── lifecycle.go                # Pause, resume, disable, enable commands
├── deploymentpolicy/           # Deployment policy subcommands
│   ├── deploymentpolicy.go     # Parent command
│   └── deploymentpolicy_reset.go  # Batch state reset
├── node/                       # Node subcommands
│   ├── node.go                 # Parent command
│   ├── node_list.go
│   ├── node_status.go
│   ├── node_reset.go
│   └── node_ignore.go          # Ignore and unignore commands
└── package/                    # Package subcommands
    ├── package.go              # Parent command
    ├── package_status.go
    ├── package_rerun.go
    └── package_logs.go
operator/internal/cli/          # Shared CLI utilities
├── client/                     # Kubernetes client wrapper
├── context/                    # CLI context and global flags
└── utils/                      # Shared utilities

Command Creation Flow

main()
  └── cli.Execute()
      └── NewSkyhookCommand(ctx)
          ├── NewVersionCmd(ctx)
          ├── NewResetCmd(ctx)
          ├── NewPauseCmd(ctx)
          ├── NewResumeCmd(ctx)
          ├── NewDisableCmd(ctx)
          ├── NewEnableCmd(ctx)
          ├── deploymentpolicy.NewDeploymentPolicyCmd(ctx)
          │   └── NewResetCmd(ctx)
          ├── node.NewNodeCmd(ctx)
          │   ├── NewListCmd(ctx)
          │   ├── NewStatusCmd(ctx)
          │   ├── NewResetCmd(ctx)
          │   ├── NewIgnoreCmd(ctx)
          │   └── NewUnignoreCmd(ctx)
          └── pkg.NewPackageCmd(ctx)
              ├── NewStatusCmd(ctx)
              ├── NewRerunCmd(ctx)
              └── NewLogsCmd(ctx)

Testing

# Run CLI tests
make test-cli
# Run all tests
make test