Nodewright and nodewright-customizations are two halves of the integration. Nodewright is a Kubernetes Operator that applies nodewright packages with consistent, repeatable, and tested lifecycles within a cluster. Nodewright-customizations are instances of the Skyhook Custom Resource that define one or more nodewright packages to deploy. These packages were selected to provide two main functions:

1. Optimize a node for inference or training workloads via grub, sysctl and systemd service settings.
2. Be able to install all of the necessary software to bring a vanilla kubernetes node to the AICR spec.

References

1. Nodewright documentation

Optimizer

Uses tuned to apply a sequence of profiles to optimize primarily grub and sysctl settings. Your mileage may vary depending on the particulars of the virtualization if not running baremetal.

Package: nvidia-tuned

Configuration documentation:

A full configuration supplies: intent, accelerator, service. A minimal configuration is just accelerator

configMap:
    intent: inference
    accelerator: h100
    service: eks

Supported accelerators: h100, gb200

Integration notes:

• If you provide a service it MUST exist in the profiles service directory
• If you are integrating a new service beware that even tested paths may not fully work due to limitations in that service. For example you will notice that eks has overrides to remove setting kernel.sched_latency_ns and kernel.sched_min_granularity_ns as these are not available on AWS kernels. They cannot fail silently as the package will test to make sure the changes asked for actually happens and error if it does not.

Secondary optimizer

A second, more stripped down, optimizer is available for operating systems that are mostly read only such as GKE’s ContainerOptimizedOS. In this case the nvidia-tuning-gke is available to directly perform sysctl writes. Also note the change in Nodewright configuration to write to a different directory tree in order to have a writable FS and to re-apply changes every boot: recipes/overlays/gke-cos.yaml

    - name: nodewright-operator
      type: Helm
      overrides:
        controllerManager:
          manager:
            env:
              # GKE COS has a read-only rootfs, so we need to use a different directory
              # /etc is stateless so better represents the flag and history on reboot
              copyDirRoot: /etc/nodewright
              # Because what nodewright does is generally on /etc we need to reapply on reboot
              reapplyOnReboot: "true"

Versioning and extension notes

Both of these packages (nvidia-tuned and nvidia-tuning-gke) extend other nodewright packages (tuned and tuning) and as such could directly use those and provide the configuration via configmaps. The choice was made to go with specific versioned packages in order to provide a more clear path for upgrades and understanding differences. However, the base packages are still useful to quickly iterate on configurations without requiring new versions of the extended packages used in AICR.

Setup

Uses a set of bash scripts to do the necessary actions to bring an ubuntu worker to the desired AICR spec.

Package: nvidia-setup

See the Tuning status table below for the current service + accelerator coverage. Each service must be added explicitly; the documentation for how to make this update is in the nvidia-setup README.

The version overview has all of the information about what each version for a service + accelerator pair will install or configure.

Manifests

Tuning and Setup

Tuning are typically alterations to sysctl, kernel boot parameters and service drop ins to make the system better optimized for AI workloads.

Setup generally is any configuration needed to make AI workloads work in that service that is not already provided by that service. In EKS for example this is kernel and EFA installation. For BCM it is symlinks to support gpu operator.

The table below is generated from the recipes by make tuning-docs — do not edit it by hand. Setup and Tuning are the pinned nodewright package versions applied for each service + accelerator. Profile is the tuning profile accelerator when it differs from the selected accelerator (for example, h200 and a100 nodes use the h100 profile), and - when identical. In the Setup and Tuning columns, - means no such package is applied for that service + accelerator. * means the recipe pins no value for that dimension.

Note: the generated table lists the packages pinned in the manifests and cannot see per-recipe value gates. The aks rows show nvidia-tuned 0.3.2, but AKS recipes disable it by default via nodewright-customizations tuningEnabled: false under the Azure-managed driver profile; see AKS GPU setup for the rationale and re-enable path.

The tuningEnabled gate (default true; only an explicit false disables) applies uniformly across the tuning manifests: on the shared tuning.yaml it omits the nvidia-tuned package while nvidia-setup keeps running, and on the single-package manifests (tuning-gke.yaml, tuning-generic.yaml) it suppresses the whole tuning Skyhook CR, since the tuning package is that CR’s only content. No recipe sets it outside AKS today, so default renderings are unchanged elsewhere.

The tuning Skyhook CR is a normal release-managed resource (no Helm hooks), so flipping tuningEnabled from true to false retracts it on all deployers — under Helm/Argo (which already stripped the hooks at bundle time) and under Flux (where a hook resource would otherwise have been orphaned on the flip, since before-hook-creation never fires when the CR leaves the render). Ordering after the Skyhook CRD install is handled by component dependency ordering (nodewright-customizations depends on nodewright-operator), not an intra-release hook.

One-time Flux migration: a cluster that was already running a pre-hook-removal bundle created the tuning Skyhook as a Helm hook (outside the release lifecycle). The first upgrade to a bundle carrying this change must keep tuningEnabled=true so Helm adopts the now-managed CR into the release; only then does a later upgrade with tuningEnabled=false prune it. If the first upgrade both removes the hooks and sets tuningEnabled=false, nothing renders for Helm to adopt and the legacy hook-created Skyhook is left behind — delete it explicitly in that case (kubectl delete skyhook tuning -n skyhook). Helm/Argo clusters are unaffected (they never created it as a hook).

Known limitation: the nodewright-customizations deployment health check asserts the tuning Skyhook CR reaches status.status: complete and cannot see value gates. When the whole CR is suppressed — tuningEnabled: false on a tuning-gke.yaml/tuning-generic.yaml recipe, or enabled: false anywhere — aicr validate --phase deployment fails that check on the deliberately untuned cluster; skip it in that configuration. (AKS is unaffected by tuningEnabled: false: its tuning CR still renders with the nvidia-setup packages.) A value-aware health check that tolerates the intentionally-absent CR is tracked in #1844.

See recipes/components/nodewright-customizations/manifests for the specifics on packages and their configuration.

Tuning-gke

A GKE + Container Optimized OS (COS) specific tuning that only sets some of the sysctl settings and does NOT require any interrupts due to being able to configure seamlessly while workloads are running.

See recipes/components/nodewright-customizations/manifests/tuning-gke.yaml

No-op

A no-op package may be used as a place holder until a full package suite can be tested. See recipes/components/nodewright-customizations/manifests/no-op.yaml