> This page is for AI Cluster Runtime, version v0.21.1.
> For other versions, use one of these documentation indexes:
> - Latest · v1.0.0 (default): https://docs.nvidia.com/aicr/latest/llms.txt
> - v1.0.0: https://docs.nvidia.com/aicr/v1.0.0/llms.txt
> - v0.22.0: https://docs.nvidia.com/aicr/v0.22.0/llms.txt
> - v0.21.1: https://docs.nvidia.com/aicr/v0.21.1/llms.txt

> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.nvidia.com/aicr/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/aicr/_mcp/server.

# What is it

Nodewright and nodewright-customizations are two halves of the integration. [Nodewright](https://github.com/NVIDIA/nodewright) is a Kubernetes Operator that applies [nodewright packages](https://github.com/NVIDIA/nodewright-packages) with consistent, repeatable, and tested lifecycles within a cluster. Nodewright-customizations are instances of the [Skyhook Custom Resource](https://github.com/NVIDIA/nodewright/blob/main/chart/templates/skyhook-crd.yaml) that define one or more nodewright packages to deploy. These packages were selected to provide two main functions:
1. Optimize a node for inference or training workloads via grub, sysctl and systemd service settings.
2. Be able to install all of the necessary software to bring a vanilla Kubernetes node to the AICR spec.

## References

1. [Nodewright documentation](https://github.com/NVIDIA/nodewright/blob/main/docs)

# Optimizer

Uses tuned to apply a sequence of profiles to optimize primarily grub and sysctl settings. Your mileage may vary depending on the particulars of the virtualization if not running baremetal.

Package: [nvidia-tuned](https://github.com/NVIDIA/nodewright-packages/tree/main/nvidia-tuned)

Configuration [documentation](https://github.com/NVIDIA/nodewright-packages/tree/main/nvidia-tuned#usage):

A full configuration supplies: `intent`, `accelerator`, `service`. A minimal configuration is just `accelerator`
```
configMap:
    intent: inference
    accelerator: h100
    service: eks
```

Supported accelerators: `h100`, `gb200`

Integration notes:
  * If you provide a service it MUST exist in the [profiles service directory](https://github.com/NVIDIA/nodewright-packages/tree/main/nvidia-tuned/profiles/service)
  * If you are integrating a new service beware that even tested paths may not fully work due to limitations in that service. For example you will notice that `eks` has overrides to remove setting `kernel.sched_latency_ns` and `kernel.sched_min_granularity_ns` as these are not available on AWS kernels. They cannot fail silently as the package will test to make sure the changes asked for actually happens and error if it does not.

## Secondary optimizer

A second, more stripped down, optimizer is available for operating systems that are mostly read only such as GKE's ContainerOptimizedOS. In this case the [nvidia-tuning-gke](https://github.com/NVIDIA/nodewright-packages/tree/main/nvidia-tuning-gke) is available to directly perform sysctl writes. Also note the change in Nodewright configuration to write to a different directory tree in order to have a writable FS and to re-apply changes every boot: [recipes/overlays/gke-cos.yaml](https://github.com/NVIDIA/aicr/blob/main/recipes/overlays/gke-cos.yaml#L69)
```
    - name: nodewright-operator
      type: Helm
      overrides:
        controllerManager:
          manager:
            env:
              # GKE COS has a read-only rootfs, so we need to use a different directory
              # /etc is stateless so better represents the flag and history on reboot
              copyDirRoot: /etc/nodewright
              # Because what nodewright does is generally on /etc we need to reapply on reboot
              reapplyOnReboot: "true"
```

## Versioning and extension notes

Both of these packages (nvidia-tuned and nvidia-tuning-gke) extend other nodewright packages (tuned and tuning) and as such could directly use those and provide the configuration via configmaps. The choice was made to go with specific versioned packages in order to provide a more clear path for upgrades and understanding differences. However, the base packages are still useful to quickly iterate on configurations without requiring new versions of the extended packages used in AICR.

# Setup

Uses a set of bash scripts to do the necessary actions to bring an ubuntu worker to the desired AICR spec.

Package: [nvidia-setup](https://github.com/NVIDIA/nodewright-packages/blob/main/nvidia-setup)

See the [Tuning status](#tuning-and-setup) table below for the current service + accelerator coverage. Each service must be added explicitly; the documentation for how to make this update is in the [nvidia-setup README](https://github.com/NVIDIA/nodewright-packages/tree/main/nvidia-setup).

The [version overview](https://github.com/NVIDIA/nodewright-packages/blob/main/nvidia-setup/VERSION_OVERVIEW.md) has all of the information about what each version for a service + accelerator pair will install or configure.

# Manifests

## Tuning and Setup

Tuning are typically alterations to sysctl, kernel boot parameters and service drop ins to make the system better optimized for AI workloads.

Setup generally is any configuration needed to make AI workloads work in that service that is not already provided by that service. In EKS for example this is kernel and EFA installation. For BCM it is symlinks to support gpu operator.

The table below is generated from the recipes by `make tuning-docs` — **do not edit it by hand**. **Setup** and **Tuning** are the pinned nodewright package versions applied for each service + accelerator. **Profile** is the tuning profile accelerator when it differs from the selected accelerator (for example, `h200` and `a100` nodes use the `h100` profile), and `-` when identical. In the **Setup** and **Tuning** columns, `-` means no such package is applied for that service + accelerator. `*` means the recipe pins no value for that dimension.

\{/* BEGIN AICR-TUNING */\}

| Service | Accelerator  | Profile | Setup              | Tuning                  |
|---------|--------------|---------|--------------------|-------------------------|
| aks     | a100         | h100    | nvidia-setup 0.5.0 | nvidia-tuned 0.3.2      |
| aks     | h100         | -       | nvidia-setup 0.5.0 | nvidia-tuned 0.3.2      |
| bcm     | *            | h100    | nvidia-setup 0.3.0 | -                       |
| bcm     | h100         | -       | nvidia-setup 0.3.0 | -                       |
| eks     | a100         | h100    | nvidia-setup 0.5.0 | nvidia-tuned 0.3.2      |
| eks     | gb200        | -       | nvidia-setup 0.5.0 | nvidia-tuned 0.3.2      |
| eks     | gb300        | -       | -                  | -                       |
| eks     | h100         | -       | nvidia-setup 0.5.0 | nvidia-tuned 0.3.2      |
| eks     | h200         | h100    | nvidia-setup 0.5.0 | nvidia-tuned 0.3.2      |
| eks     | rtx-pro-6000 | generic | -                  | nvidia-tuned 0.3.2      |
| generic | gb300        | -       | -                  | -                       |
| gke     | a100         | h100    | -                  | nvidia-tuning-gke 0.1.2 |
| gke     | b200         | -       | -                  | nvidia-tuning-gke 0.1.2 |
| gke     | h100         | -       | -                  | nvidia-tuning-gke 0.1.2 |
| rke2    | vr200        | -       | -                  | nvidia-tuned 0.9.0      |

\{/* END AICR-TUNING */\}

Note: the generated table lists the packages *pinned in the manifests* and
cannot see per-recipe value gates. The `aks` rows show `nvidia-tuned 0.3.2`,
but AKS recipes disable it by default via `nodewright-customizations`
`tuningEnabled: false` under the Azure-managed driver profile; see
[AKS GPU setup](/aicr/v0.21.1/integrator-guide/aks-gpu-setup#infiniband-rdma-host-setup-nodewright)
for the rationale and re-enable path.

The `tuningEnabled` gate (default `true`; only an explicit `false` disables)
applies uniformly across the tuning manifests: on the shared `tuning.yaml` it
omits the `nvidia-tuned` package while `nvidia-setup` keeps running, and on the
single-package manifests (`tuning-gke.yaml`, `tuning-generic.yaml`) it
suppresses the whole tuning Skyhook CR, since the tuning package is that CR's
only content. No recipe sets it outside AKS today, so default renderings are
unchanged elsewhere.

The tuning Skyhook CR is a normal release-managed resource (no Helm hooks), so
flipping `tuningEnabled` from `true` to `false` retracts it on all deployers —
under Helm/Argo (which already stripped the hooks at bundle time) and under
Flux (where a hook resource would otherwise have been orphaned on the flip,
since `before-hook-creation` never fires when the CR leaves the render).
Ordering after the Skyhook CRD install is handled by component dependency
ordering (`nodewright-customizations` depends on `nodewright-operator`), not an
intra-release hook.

One-time Flux migration: a cluster that was **already** running a pre-hook-removal
bundle created the `tuning` Skyhook as a Helm hook (outside the release
lifecycle). The first upgrade to a bundle carrying this change must keep
`tuningEnabled=true` so Helm adopts the now-managed CR into the release; only
then does a later upgrade with `tuningEnabled=false` prune it. If the first
upgrade *both* removes the hooks *and* sets `tuningEnabled=false`, nothing
renders for Helm to adopt and the legacy hook-created Skyhook is left behind —
delete it explicitly in that case (`kubectl delete skyhook tuning -n skyhook`).
Helm/Argo clusters are unaffected (they never created it as a hook).

Known limitation: the `nodewright-customizations` deployment health check
asserts the `tuning` Skyhook CR reaches `status.status: complete` and cannot
see value gates. When the whole CR is suppressed — `tuningEnabled: false` on a
`tuning-gke.yaml`/`tuning-generic.yaml` recipe, or `enabled: false` anywhere —
`aicr validate --phase deployment` fails that check on the deliberately
untuned cluster; skip it in that configuration. (AKS is unaffected by
`tuningEnabled: false`: its `tuning` CR still renders with the `nvidia-setup`
packages.) A value-aware health check that tolerates the intentionally-absent
CR is tracked in [#1844](https://github.com/NVIDIA/aicr/issues/1844).

See [recipes/components/nodewright-customizations/manifests](https://github.com/NVIDIA/aicr/blob/main/recipes/components/nodewright-customizations/manifests) for the specifics on packages and their configuration.

## Tuning-gke

A GKE + Container Optimized OS (COS) specific tuning that only sets some of the sysctl settings and does NOT require any interrupts due to being able to configure seamlessly while workloads are running.

See [recipes/components/nodewright-customizations/manifests/tuning-gke.yaml](https://github.com/NVIDIA/aicr/blob/main/recipes/components/nodewright-customizations/manifests/tuning-gke.yaml)

## No-op

A no-op package may be used as a place holder until a full package suite can be tested. See [recipes/components/nodewright-customizations/manifests/no-op.yaml](https://github.com/NVIDIA/aicr/blob/main/recipes/components/nodewright-customizations/manifests/no-op.yaml)