Kubernetes

Kubernetes is an open-source platform for automating deployment, scaling, and operations of application containers across clusters of hosts. With Kubernetes, it is possible to:

• scale applications on the fly

• seamlessly update running services

• optimize hardware usage by using only the resources that are needed

BCM provides the administrator with the required packages, allows Kubernetes to be set up on a cluster, and manages and monitors Kubernetes. More information about the design of Kubernetes, its command line interfaces, and other Kubernetes-specific details, can be found at the official online documentation at https://kubernetes.io/docs/.

To deploy most of Kubernetes, NVIDIA Base Command Manager from version 10.0 onwards uses kubeadm (https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/).

BCM runs CoreDNS, the Kubelet component (https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/), and the entire control plane inside Kubernetes.

The distributed key-value store used for Kubernetes, etcd, is typically run inside of Kubernetes on control planes. However, in BCM etcd is run outside of Kubernetes.

Kubernetes integration with BCM is available for Kubernetes v1.34, v1.35, and v1.36 at the time of writing of this paragraph (July 2026). Kubernetes runs on the x86_64 and aarch64 architectures for all the BCM-supported Linux distributions. For a more up-to-date status, the feature matrix at https://service.bcm.nvidia.com/feature-matrix/ can be checked.

The version that is running on a cluster can be found with:

Example

root@basecm11:~# kubeadm version -o short
v1.34.9

The versions that are available with BCM integration can be found with the cm-kubernetes-setup utility (section 4.2):

Example

root@basecm11:~# cm-kubernetes-setup --list-versions
1.36
1.35 (NVIDIA AI Enterprise certified)
1.34 (NVIDIA AI Enterprise certified)

4.1 Reference Architecture

A reference architecture for Kubernetes in BCM comprises:

• etcd nodes: An etcd cluster—the Kubernetes distributed key-value storage—runs on an odd number (1, 3, 5 …) of nodes.

• control plane nodes: typically run on head nodes or on dedicated nodes. 2 or 3 are recommended.

• worker nodes: typically run on regular nodes that are designated to run user workloads.

To avoid a single point of failure and to achieve high availability, a minimum of three nodes is recommended for etcd, and a minimum of two control plane nodes is recommended.

Kubernetes API server proxy: An NGINX server is configured on the head node(s) and on all other nodes involved in the Kubernetes cluster (control plane and worker nodes). This NGINX server takes care of exposing the Kubernetes API server on a specific port, and load balances requests to the control plane nodes. Should one of those nodes go down it can detect this and stop sending requests until the node comes back up.

Since BCM version 8.2, multiple clusters of Kubernetes can be deployed. In such a configuration, the same nodes cannot be shared across different Kubernetes clusters. Because of the NGINX server, a port is reserved on the head node(s) for every Kubernetes cluster. This is required for Kubernetes HA , and it also allows kubectl and other tools such as Helm to be used from the head node, to access each Kubernetes cluster.

Gateway Proxy: The same NGINX server is also used as the entry point on the head node(s) to forward traffic to the Kubernetes Gateways. This functionality is referred to as the “Gateway proxy” and is described in detail in section 4.17.

4.1.1 Hardware Requirements

To run properly, Kubernetes deployment requires a per-node minimum of

• 16 GB RAM

• two CPU cores

Without these minimum requirements, deployment may succeed, but further configuration, such as starting up Jupyter kernels, may fail with problems that are hard to diagnose. For example, error reporting may not show a lack of resources because some components are automatically deleted.

4.2 Kubernetes Setup

The usual, and recommended way, to install Kubernetes with BCM is to install it interactively from a TUI session (section 4.2.8).

An alternative way, typically used for automated non-interactive setups, is to use a plain command line specification (section 4.2.7), which usually results in a long command line.

BCM deploys Kubernetes with the cm-kubernetes-setup utility, which is part of the cm-setup package. Several recent versions of Kubernetes are offered (figure 4.1):

Figure 4.1: Kubernetes setup TUI session (section 4.2.8): version selection screen

The TUI version selection offers the Kubernetes minor versions that the installed packages provide. The patch (http://semver.org) level within the chosen minor version is then asked for on the following screen (figure 4.5). At the time of writing of this section (August 2026), minor versions 1.34, 1.35, and 1.36 are the recommended and tested versions offered, providing patch levels 1.34.10, 1.35.7, and 1.36.3 respectively. The selected patch version should normally be retained. In special cases, the patch version can be changed in the saved configuration file:

• The patch version can be removed to use the latest released patch version. However, because the latest version can change between runs, this can make the installation non-idempotent.

• The patch version can be set to a version other than the recommended and tested version.

• For an air-gapped deployment (Appendix B), each exact Kubernetes version expects a specific combination of system packages and container images, all of which must be present in the air-gapped deployment bundle. From BCM 11.34 onwards, the wizard validates this before installation begins, and fails early with a specific error message if the selected version does not match what was downloaded, instead of failing later during installation. Two kinds of error types can be shown, depending on which mismatch is detected:

  1. A package version mismatch names each affected package and its found and expected versions, for both the head node and the default software image:

     Example

     Required package versions are not available in the air-gapped
     package set (kube version or BCM operator package version
     mismatch):
       /:
         - kubeadm==1.36.3 (found: 1.34.10-1.1)
         - kubectl==1.36.3 (found: 1.34.10-1.1)
         - kubelet==1.36.3 (found: 1.34.10-1.1)
       /cm/images/default-image:
         - kubeadm==1.36.3 (found: 1.34.10-1.1)
         - kubectl==1.36.3 (found: 1.34.10-1.1)
         - kubelet==1.36.3 (found: 1.34.10-1.1)

  2. A Helm chart version mismatch names the chart and its found and expected versions:

     Example

     Deployment is impossible due to missing or mismatched HELM
     charts:
     HELM charts version mismatch: network-operator: ['26.1.0']
     not in ['26.1.1']

• BCM provides or uses the following Kubernetes-related packages:

  • conntrack (conntrack-tools on RHEL-based distributions) and nginx: These distribution packages are always installed on the head node(s) and on the control-plane and worker node(s).

  • cm-etcd: This BCM package is installed on the nodes selected for etcd. In a similar way to the case of Harbor or Docker Registry (section 3.2), the nodes that run etcd are protected by BCM with the datanode option (here). For etcd nodes, the option is added during the cm-kubernetes-setup installation. As in the case for the registries, the datanode option is set in order to help prevent the administrator from wiping out the existing state of etcd nodes. Wiping out the state of etcd nodes means that the Kubernetes cluster becomes incoherent and that it cannot be restored to where it was just before the etcd nodes were wiped.

  • cm-containerd: This BCM package has the containerd runtime.

  • cm-nvidia-container-toolkit: This BCM package has the NVIDIA container toolkit (includes NVIDIA container runtime).

  • cm-kube-diagnose: This BCM package has Helper tools to diagnose malfunctioning Kubernetes clusters.

  Kubernetes .rpm and .deb packages themselves are installed from the Kubernetes community-owned software repositories (https://kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/). These repositories host Kubernetes versions starting from v1.24.0 at the time of writing of this paragraph (March 2024).

4.2.1 Kubernetes Networking

Early on during the wizard session (figure 4.2), a name for the cluster is requested. The wizard pre-fills it with default, but this should not be confused with the Kubernetes default namespace. Here, the name is used instead, inside BCM, to identify the cluster, configuration files, and other resources such as module files.

Figure 4.2: Kubernetes setup TUI session: networking selection screen

This screen also allows the following important choices:

• Kubernetes external FQDN: This is the FQDN that is placed as one of the entries in the public-facing certificates generated for this Kubernetes cluster.

  Configuring the public-facing certificate of the bcm-gateway Gateway (served by Kgateway) is discussed further in section 4.17.7.

• Service network base address and Service network netmask bits: These define the CIDR for the service network. The wizard pre-fills the fields. It also tries to avoid pre-filling them with overlapping network ranges, by taking any existing network known to BCM into account.

• Pod network base address and Pod network netmask bits: These define the CIDR for the pod network. The wizard pre-fills these. It also tries to avoid pre-filling them with overlapping network ranges, by taking any existing network known to BCM into account. By default, entire /24 network ranges are assigned to individual Kubernetes nodes from the pod CIDR.

The packages are installed automatically from the repository when the administrator runs cm-kubernetes-setup from the command line.

The log file produced by the setup can be found in /var/log/cm-kubernetes-setup.log.

4.2.2 Kubernetes Core Add-ons

During setup, some critical add-on components such as the Networking Component for Kubernetes are automatically deployed. Most components are in the kube-system namespace, but others have their own namespaces. In BCM some add-ons are treated as Kubernetes applications (Chapter 5), and belong to the default app group, system. In the future Helm is expected to manage most, if not all, components instead of BCM Kubernetes applications.

The user is prompted for a Networking Component, Kubernetes operators (managed using Helm), Kubernetes add-ons (managed using Kubernetes applications), and more. They all result in either Helm charts being deployed, or in Kubernetes applications being managed from BCM.

A cmsh treeview illustrating the hierarchy to access these applications is:

[cmsh]
  |-- ...
  |
  |-- kubernetes[default]
  |              `-- appgroups[system]
  |                            `-- applications
  |-- ...

Helm charts can be found using the helm command. For example using helm list -A on a cluster (after loading the correct module file).

In the past the DNS component was also provided as an add-on. However since BCM version 10.0, kubeadm is used to bundle CoreDNS as part of the default control plane.

Kubernetes apps are discussed further in Chapter 5. Kubernetes operators are discussed further in Chapter 6.

4.2.2.1 Kubeadm Components

Components deployed as part of the default Kubernetes control plane are:

• Kubernetes API server

• Kubernetes Scheduler

• Kubernetes Controller Manager

• Kubernetes Proxy

• Core DNS

These are deployed through the “static pod” mechanism (https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/). Kubernetes control plane components are described at: https://kubernetes.io/docs/concepts/overview/components/.

CoreDNS: CoreDNS is the DNS server add-on for internal service discovery. It reads the IP addresses of services and pods from etcd, and resolves domain names for them. If a domain name is not found because the domain is external to the Kubernetes cluster, then CoreDNS forwards the request to the main DNS server. BCM uses CoreDNS version 1.13.1 with Kubernetes version 1.35.

4.2.2.2 Networking Component

An important component managed through BCM’s Kubernetes Apps is the Networking component. This adheres to the Container Networking Interface, or CNI. At the time of writing the available CNIs in the setup wizard are either Calico or Flannel. However BCM does support other CNIs, for example Cilium, as described in the knowledge base article at https://enterprise-support.nvidia.com/s/article/installing-cilium-networking-for-kubernetes.

Flannel: Flannel is a simple and easy way to configure a layer 3 network fabric designed for Kubernetes.

Calico: Calico is a popular open source networking and network security solution for Kubernetes. It provides network connectivity between workloads and security policies features. It can be configured to use eBPF.

Further details on Calico can be found at https://docs.projectcalico.org/.

If the Kubernetes cluster is composed of more than 50 nodes, then the Calico component Typha is also automatically deployed for better scalability. The number of Typha replicas is calculated by allocating one Typha replica per 150 nodes, with a minimum of 3 (above 50 nodes) and a maximum of 20.

If an initial deployment of the Kubernetes cluster has fewer than 50 nodes, but nodes are then added to the Kubernetes cluster so that the 50 node threshold is exceeded, then Typha is not automatically enabled. In this case, Typha can be enabled manually using cmsh as follows:

Example

    [basecm11->kubernetes[default]->appgroups[system]->applications[calico]]% environment
    [basecm11->kubernetes[default]->appgroups[system]->applications[calico]->environment]% list
    Name (key)             Value            Nodes environment
    ---------------------- ---------------- ------------------
    calico_typha_replicas  0                no
    calico_typha_service   none             no
    cidr                   10.141.0.0/16    no
    [basecm11->kubernetes...[calico]->environment]% set calico_typha_service value calico-typha
    [basecm11->kubernetes*...[calico*]->environment*]% set calico_typha_replicas value 3
    [basecm11->kubernetes*[default*]->appgroups*[system*]->applications*[calico*]->environment*]% commit
    [basecm11->kubernetes[default]->appgroups[system]->applications[calico]->environment]%

Cilium: Cilium (https://cilium.io/) is a networking, observability, and security solution with an eBPF-based dataplane.

4.2.3 Kubernetes Optional Components

The following components are installed by default unless otherwise noted. However, the user can choose to skip some or all of them during the setup.

4.2.3.1 Kgateway

Kgateway is the implementation that BCM uses to fulfill the Kubernetes Gateway API specification. It is discussed further in section 4.17.

4.2.3.2 MetalLB

MetalLB is a bare-metal load balancer that allocates external IP addresses to any LoadBalancer-type Service on the cluster. BCM deploys it alongside Kgateway to back the Kubernetes Gateways (section 4.17.4.1), but it can also be used outside the Gateway scope.

4.2.3.3 Kubernetes Metrics Server

The Kubernetes Metrics Server is an add-on that is a replacement for Heapster. It aggregates metrics, and provides container monitoring and performance analysis. It exposes metrics using an API.

4.2.3.4 Kubernetes State Metrics

kube-state-metrics is an add-on agent to generate and expose cluster-level Kubernetes metrics (section G.1.14 of the Administrator Manual) for objects. The project is not focused on the health of the individual Kubernetes components, but rather on the health of the various objects inside, such as deployments, nodes and pods.

4.2.4 Kubernetes Deprecated And Retired Components

The following components were part of BCM’s Kubernetes installation in earlier releases. They are no longer installed by default, and are documented here for reference and to point users to their replacements.

4.2.4.1 Kubernetes Dashboard (retired)

The Kubernetes Dashboard provided a web UI for cluster administration and metrics visualization in BCM versions before 11.32.0. It was retired upstream and has been removed in BCM version 11.32.0.

4.2.4.2 Ingress NGINX Controller (deprecated)

The Ingress NGINX Controller (ingress-nginx) provided ingress routing and web load balancing in BCM versions before 11.33.0 using standard Kubernetes Ingress resources, which were consumed together with ingress-nginx-specific annotations, such as nginx.ingress.kubernetes.io/*.

The Ingress NGINX Controller was deployed as a ingress-nginx Helm release. In BCM, it was exposed externally using the legacy Ingress proxy on the non-standard ports 30080 (HTTP) and 30443 (HTTPS), and the TLS certificate for externally-exposed services was served on 30443. It was deprecated upstream and in BCM starting with version 11.32.0, and is disabled by default from BCM 11.33.0, where it is replaced by Kgateway (section 4.17). For BCM clusters that were originally deployed with the Ingress NGINX Controller, the procedure to migrate to the Gateway-based stack is described in section 4.17.8.

The wizard marks the Ingress NGINX Controller as deprecated on the operator selection screen (section 4.2.8). It should not be selected for a new deployment without a good reason to do so.

The Ingress NGINX Controller should not be confused with the Ingress proxy. The controller runs inside Kubernetes. The Ingress proxy is an NGINX instance that runs outside Kubernetes, on the head node, and forwards external traffic into the cluster (section 4.9.3). The two are configured independently, so the Ingress Proxy Enable parameter being set says nothing about whether the Ingress NGINX Controller is in use.

4.2.5 Helm Kubernetes Package Manager

Helm is an add-on that manages charts, which are packages of pre-configured Kubernetes resources. The Helm component is installed and properly configured with BCM’s Kubernetes installation by default. It is initialized and ready for use by every Kubernetes user when the Kubernetes module is loaded. BCM uses Helm version 4.2.3.

When the Helm binary is installed during the Kubernetes setup, an offer is made to deploy charts during the setup process. Most operators are documented in Chapter 6.

Other parts of the wizard also determine which Helm charts are deployed as part of the setup for the following:

• Kyverno and Kyverno Policies (section 4.10.1)

• BCM Permissions Manager (section 4.15)

• BCM Local Path Provisioner (section 4.20)

Example

root@basecm11:~# helm list -A
NAME                        NAMESPACE               CHART                                       APP VERSION
agentgateway                agentgateway-system ... agentgateway-v1.3.1                         v1.3.1
agentgateway-crds           agentgateway-system ... agentgateway-crds-v1.3.1                    v1.3.1
calico                      tigera-operator     ... tigera-operator-v3.32.1                     v3.32.1
calico-crds                 tigera-operator     ... crd.projectcalico.org.v1-v3.32.1            v3.32.1
cm-jupyter-kernel-operator  cm                  ... cm-jupyter-kernel-operator-0.3.12           0.3.12
cm-kubernetes-mpi-operator  cm                  ... mpi-operator-0.8.0                          0.8.0
gpu-operator                gpu-operator        ... gpu-operator-v26.3.3                        v26.3.3
grafana-operator            grafana             ... grafana-operator-5.24.0                     v5.24.0
k8s-nim-operator            nim-operator        ... k8s-nim-operator-3.1.1                      3.1.1
kgateway                    kgateway-system     ... kgateway-2.3.6                              2.3.6
kgateway-crds               kgateway-system     ... kgateway-crds-2.3.6                         2.3.6
knative-operator            knative-operator    ... knative-operator-1.22.3
kube-prometheus-stack       prometheus          ... kube-prometheus-stack-87.18.0               v0.92.1
kube-state-metrics          kube-system         ... kube-state-metrics-8.0.0                    2.19.1
kyverno                     kyverno             ... kyverno-3.8.2                               v1.18.2
kyverno-policies            kyverno             ... kyverno-policies-3.8.2                      v1.18.2
local-path-provisioner      cm                  ... local-path-provisioner-0.0.36               v0.0.36
lws                         lws-system          ... lws-v0.9.0                                  v0.9.0
metallb                     metallb-system      ... metallb-0.16.1                              v0.16.1
metrics-server              kube-system         ... metrics-server-3.13.1                       0.8.1
network-operator            network-operator    ... network-operator-26.1.1                     v26.1.1
permissions-manager         cm                  ... cm-kubernetes-permissions-manager-0.7.0     0.7.0
postgres-operator           postgres-operator   ... postgres-operator-1.15.1                    1.15.1
prometheus-adapter          prometheus          ... prometheus-adapter-5.3.0                    v0.12.0
spark-operator              spark-operator      ... spark-operator-2.5.1                        2.5.1

4.2.6 Note On Run:ai Operator Dependencies

4.2.6.1 Selecting The Run:ai Operator Dependencies

When the cm-kubernetes-setup wizard is run, with the intent of running Run:ai as well, the operator selection screen must include the dependencies that Run:ai requires. For Run:ai 2.25, these are:

• NVIDIA GPU Operator

• Grafana Operator

• Kgateway

• Knative Operator

• Kubeflow Training Operator

• Kubernetes Metrics Server

• Kubernetes MPI Operator

• Kubernetes State Metrics

• LeaderWorkerSet Operator

• MetalLB

• Network Operator (technically optional, but recommended)

• NIM Operator (optional)

• Prometheus Adapter

• Prometheus Operator Stack

The Run:ai operator and its prerequisites are described in more detail in section 6.11.

4.2.7 Kubernetes Setup From The Command Line

The cm-kubernetes-setup command line utility has the following usage synopsis:

root@basecm11:~# cm-kubernetes-setup --help
usage: Kubernetes Setup cm-kubernetes-setup [-c <config_file>]
                                            [--list-versions]
                                            [--list-operators-versions]
                                            [--cluster CLUSTER_NAME]
                                            [--skip-image-update]
                                            [--skip-kube-version-check]
                                            [--skip-dns-configuration-check]
                                            [--skip-ping-check]
                                            [--skip-mtu-check]
                                            [--skip-package-manager-update-check]
                                            [--skip-install-repos]
                                            [--adv-configuration-bcm-nvidia-packages]
                                            [--add-user USERNAME_ADD]
                                            [--list-users]
                                            [--get-user GET_USER]
                                            [--modify-user USERNAME_MODIFY]
                                            [--remove-user USERNAME_REMOVE]
                                            [--namespace NAMESPACE]
                                            [--add-to-namespace]
                                            [--remove-from-namespace]
                                            [--role {edit,admin,view,cluster-admin}]
                                            [--runas-uid RUNAS_UID]
                                            [--runas-gids RUNAS_GIDS]
                                            [--user-paths USER_PATHS]
                                            [--allow-all-uids]
                                            [--operators OPERATORS]
                                            [--backup-permissions FILE]
                                            [--restore-permissions FILE]
                                            [--list-operators]
                                            [--add-operators | --remove-operators]
                                            [--patch-kube-prometheus-stack]
                                            [--configure-gateway-proxy]
                                            [--remove]
                                            [--yes-i-really-mean-it] [--pull]
                                            [--images IMAGES] [--nodes NODES]
                                            [--node-selector NODE_SELECTOR]
                                            [--pull-registry-server PULL_REGISTRY_SERVER]
                                            [--pull-registry-username PULL_REGISTRY_USERNAME]
                                            [--pull-registry-email PULL_REGISTRY_EMAIL]
                                            [--pull-registry-password PULL_REGISTRY_PASSWORD]
                                            [--docker-io-username DOCKER_IO_USERNAME]
                                            [--docker-io-password DOCKER_IO_PASSWORD]
                                            [--kubeadm-image-repository KUBEADM_IMAGE_REPOSITORY]
                                            [--github-token GITHUB_TOKEN]
                                            [--containerd-sandbox-image CONTAINERD_SANDBOX_IMAGE]
                                            [--enable-ovn] [--no-cni]
                                            [--latest-versions]
                                            [--skip-ovn-prerequisites-checks]
                                            [--skip-netq-prerequisites-checks]
                                            [--provision-node PROVISION_NODE]
                                            [--provision-software-image PROVISION_SOFTWARE_IMAGES]
                                            [--airgap-registry AIRGAP_REGISTRY]
                                            [--airgap-registry-as-helm | --airgap-helm-repo AIRGAP_HELM_REPO]
                                            [--airgap-install-system-packages | --airgap-push-
                                                container-images | --airgap-push-helm-charts]
                                            [--ca-file CA_FILE]
                                            [--airgap-registry-username AIRGAP_REGISTRY_USERNAME]
                                            [--airgap-registry-password AIRGAP_REGISTRY_PASSWORD]
                                            [-v] [--store-name-aliases]
                                            [--no-distro-checks] [--json]
                                            [--output-remote-execution-runner]
                                            [--on-error-action {debug,remotedebug,undo,abort}]
                                            [--skip-reboot] [--skip-packages]
                                            [--min-reboot-timeout <reboot_timeout_seconds>]
                                            [--allow-running-from-secondary]
                                            [--dev] [-h]
options:
  --cluster CLUSTER_NAME
                        Name of the referred Kubernetes cluster
  -h, --help            Print this screen
common:
  Common arguments
  -c <config_file>      Load runtime configuration for plugins from a YAML config file
Supported versions information:
  --list-versions       Show supported Kubernetes versions
  --list-operators-versions
                        List available for selection Kubernetes operators versions
installing Kubernetes clusters:
  Flags for installing or managing Kubernetes clusters
  --skip-image-update   Skip the image update steps.
  --skip-kube-version-check
                        Skip Kubernetes version check before install (setup will not continue
                        if it finds unsupported version).
  --skip-dns-configuration-check
                        Skip DNS configuration check before install (setup will not continue if
                        it finds DNS malfunctioning).
  --skip-ping-check     Skip Ping check before install (setup will not continue if there is no
                        inter-node connectivity across the Kubernetes cluster).
  --skip-mtu-check      Skip MTU check before install (inconsistent MTU sizes on Kubernetes
                        nodes can result in packet loss).
  --skip-package-manager-update-check
                        Skip Package Manager update/refresh check before install (setup will
                        not continue if 'apt update' fails on Ubuntu for example).
  --skip-install-repos  Skip installation of kubernetes and helm package repos. (assumes they
                        have been manually configured, e.g., in the case of air-gapped).
  --adv-configuration-bcm-nvidia-packages
                        Allow the option to override the selection of BCM NVIDIA packages to be
                        installed when setting up the NVIDIA GPU operator
user management:
  Flags for adding a new user in a Kubernetes cluster
  --add-user USERNAME_ADD
                        Create a new user in a Kubernetes cluster
  --list-users          Get information about configured Kubernetes users
  --get-user GET_USER   Get information about configured Kubernetes users
  --modify-user USERNAME_MODIFY
                        Modify user in a Kubernetes cluster
  --remove-user USERNAME_REMOVE
                        Remove existing user from a Kubernetes cluster
  --namespace NAMESPACE
                        Specify namespace for user (--get-user, --modify-user) role binding
  --add-to-namespace    Indicate if permissions to manage namespace needs to be granted for a
                        given user (--modify-user)
  --remove-from-namespace
                        Indicate if permissions to manage namespace needs to be revoked for a
                        given user (--modify-user)
  --role {edit,admin,view,cluster-admin}
                        Specify role for the new (--add-user) and existing (--modify-user) role
                        binding (Default: edit)
                        For 'cluster-admin' namespace flag is ignored
  --runas-uid RUNAS_UID
                        UID is allowed to be used in unprivileged pods (--add-user, --modify-user)
  --runas-gids RUNAS_GIDS
                        Comma-separated list of GIDs allowed to be used in unprivileged pods
                        (--add-user, --modify-user)
  --user-paths USER_PATHS
                        Comma-separated list of paths user is able to mount in pods (--add-
                        user, --modify-user)
  --allow-all-uids      Allow user to run processes in pods as any user (--add-user, --modify-user)
                        hostPath volumes will be disabled for such pods
  --operators OPERATORS
                        Comma-separated list of operators user has access to (--add-user, --modify-user)
backup or restore Permission Manager user configurations:
  Flag for managing permission manager user configuration
  --backup-permissions FILE
                        Save permissions to file
  --restore-permissions FILE
                        Restore permissions from file. Workload which is already run by users
                        in their namespaces will be affected
list available operators:
  Flag to list available Kubernetes operators
  --list-operators      List available Kubernetes operators
modify operators:
  Flags for config-driven Day 2 operator management on an existing cluster
  --add-operators       Add operators to an existing Kubernetes cluster. Requires a config file
                        (-c); only operators it explicitly marks install_helm: true are added
  --remove-operators    Remove operators from an existing Kubernetes cluster. Requires a config
                        file (-c); only operators it explicitly marks install_helm: false are
                        removed
update kubernetes addons:
  Flags for updating Kubernetes addons
  --patch-kube-prometheus-stack
                        Patch to improve kube-prometheus-stack helm chart (scrape config)
gateway proxy:
  Flags for configuring the Gateway proxy
  --configure-gateway-proxy
                        Configure the Gateway proxy to forward traffic from the head node(s) to
                        the Kubernetes Gateways
removing Kubernetes clusters:
  Flags for removing a Kubernetes cluster
  --remove              Remove a Kubernetes cluster
  --yes-i-really-mean-it
                        Required for additional safety
pulling images to the nodes:
  Flags for pulling images to the nodes
  --pull                Pull images to the nodes
  --images IMAGES       Comma-separated list of images to pull (--pull)
  --nodes NODES         Comma-separated list of nodes to pull images to (--pull)
  --node-selector NODE_SELECTOR
                        Selector (label query) to filter on, supports '=', '==', and '!='.(e.g.
                        key1=value1,key2=value2) (--pull)
  --pull-registry-server PULL_REGISTRY_SERVER
                        Registry server to authenticate
  --pull-registry-username PULL_REGISTRY_USERNAME
                        Registry username
  --pull-registry-email PULL_REGISTRY_EMAIL
                        Email to authenticate
  --pull-registry-password PULL_REGISTRY_PASSWORD
                        Registry password
docker.io:
  Flags for authenticating to docker hub
  --docker-io-username DOCKER_IO_USERNAME
                        Username
  --docker-io-password DOCKER_IO_PASSWORD
                        Password
kubeadm:
  Flags for configuring kubeadm
  --kubeadm-image-repository KUBEADM_IMAGE_REPOSITORY
                        Custom imageRepository
GitHub:
  Flags for configuring GitHub API access
  --github-token GITHUB_TOKEN
                        GitHub token for authenticated release metadata requests. Stored
                        base64-encoded in Kubernetes config.
containerd:
  Flags for configuring containerd
  --containerd-sandbox-image CONTAINERD_SANDBOX_IMAGE
                        Custom sandbox image (e.g. my-registry:5000/pause:3.8)
ovn:
  Flags for configuring OVN CNI
  --enable-ovn          Allow selection of OVN CNI during Kubernetes setup (currently not shown by default).
  --no-cni              Instead of deploying a CNI, the setup will exit early after having
                        setup Kubernetes without a CNI, the user is expected to configure a
                        CNI, and other components.
  --latest-versions     Dynamically determine and use latest versions of OVN Kubernetes CNI and
                        Dashboard charts.
  --skip-ovn-prerequisites-checks
                        Provide this flag to skip precondition checks such as checking for
                        Network Layouts, Linux distro requirements for OVN CNI.
netq:
  Flags for configuring NetQ
  --skip-netq-prerequisites-checks
                        Provide this flag to skip precondition checks such as checking for disk
                        size, Linux distro, etc.
provisioning:
  Flags for provisioning Kubernetes packages and configuration
  --provision-node PROVISION_NODE
                        Provision a node with Kube packages and prepare other configuration
  --provision-software-image PROVISION_SOFTWARE_IMAGES
                        Provision a software image with Kube packages (excludes NVIDIA GPU
                        (driver, toolkit, ...) packages) and prepares other configuration.
                        Please ensure that software images are prepared before kubelet role
                        assignment. If manual provisioning of nodes is required, an imageupdate
                        is insufficient, either use --provision-node, or give the node an
                        additional reboot to make it join the Kubernetes cluster.
Air-gapped deployment:
  --airgap-registry AIRGAP_REGISTRY
                        AirGap container registry address
  --airgap-registry-as-helm
                        Use AirGap container registry as helm repository
  --airgap-helm-repo AIRGAP_HELM_REPO
                        Helm repository address
Air-gapped deployment preparation:
  --airgap-install-system-packages
                        Install pre-downloaded system packages
  --airgap-push-container-images
                        Push pre-downloaded container images to the AirGap registry
  --airgap-push-helm-charts
                        Push pre-downloaded helm charts to the AirGap registry
  --ca-file CA_FILE     path to CA file for AirGap registry TLS verification
  --airgap-registry-username AIRGAP_REGISTRY_USERNAME
                        AirGap registry username
  --airgap-registry-password AIRGAP_REGISTRY_PASSWORD
                        AirGap registry password
advanced:
  Various *advanced* configuration options flags.
  -v, --verbose         Verbose output
  --store-name-aliases  Store hostname aliases for head nodes (active and passive) and default category
  --no-distro-checks    Disable distribution checks based on ds.json
  --json                Use json formatting for log lines printed to stdout
  --output-remote-execution-runner
                        Format output for CMDaemon
  --on-error-action {debug,remotedebug,undo,abort}
                        Upon encountering a critical error, instead of asking the user for
                        choice, setup will do selected action.
  --skip-reboot         Skip the any stages which reboot compute nodes.
  --skip-packages       Skip the any stages which install packages. Requires packages to be already installed.
  --min-reboot-timeout <reboot_timeout_seconds>
                        How long to wait for nodes to finish reboot (default and minimum
                        allowed: 300 seconds).
  --allow-running-from-secondary
                        Allow to run the wizard from the secondary when it is the active head node.
  --dev                 Enables additional command line arguments

The preceding output does not list every argument that the utility accepts. Some valid arguments are not shown by --help. For example, --skip-headnode-check is accepted by the utility but is not listed. It is a development argument, and it is not part of any documented BCM procedure. An argument that is absent from the --help output may be valid, but is typically not intended for use by the end user.

The cm-kubernetes-setup utility should be executed on the console.

4.2.7.1 Dealing With A Pre-existing Docker Installation

Docker (Chapter 2) is no longer a requirement for Kubernetes configured by BCM. This is because Kubernetes can directly interface with containerd through its Container Runtime Interface (CRI). Docker can co-exist with Kubernetes, and can be set up as discussed in section 2.1.

4.2.8 Kubernetes Setup From A TUI Session

When the Kubernetes installation is carried out using cm-kubernetes-setup without any options, a TUI wizard starts up. The administrator can answer several questions in the wizard. Questions that are asked include questions about which of the node categories or which of the individual nodes should be configured to run the Kubernetes services. There are also questions about the service and pod networks parameters, the port numbers that will be configured for the daemons, which operators to install, and so on. After the wizard has been completed, a configuration file can be saved that can be used to set up Kubernetes, by default cm-kubernetes-setup.conf.

The configuration file can be deployed immediately from the wizard, or it can be deployed later by specifying it as an option to cm-kubernetes-setup, in the form -c <file>.

The layout of the configuration file can change between BCM releases. For example, the Flannel settings are stored under modules.kubernetes.flannel in BCM 11.33, and under modules.kubernetes.operators.flannel in BCM 11.34. A configuration file that was saved by one release should therefore not be assumed to work with another.

If no deployment has been carried out earlier, then the main operations screen of the wizard shows just two options, Deploy and Exit.

If deployment has already been carried out, then the further options that are available are also displayed (figure 4.3):

Example

[root@basecm11 ~]# cm-kubernetes-setup
           TUI session starts up:

Figure 4.3: Kubernetes setup TUI session: main operations screen after a deployment

One of the further options shown after a deployment is a Modify operators item, for adding operators to, or removing operators from, an existing cluster without re-running the full setup. This is described in section 6.15.

The deployment using CLI or using TUI assigns the appropriate roles, and adds the new Kubernetes cluster. The deployment adds health checks to the monitoring configuration, and it generates certificates for the Kubernetes daemons.

Operating the wizard:

The wizard is a terminal UI, where

• the arrow keys (up, down, left, right) move between fields

• the <TAB> key moves between widgets such as the field area and the buttons along the bottom

• the <SPACE> bar selects an option on screens that present a list of choices

• the <ENTER> key, if pressed on a screen with a list of choices, advances to the next screen without changing the selection, which can take some getting used to

• the < Help > button on the screen, if present, describes the setting in more detail than the screen itself.

The screens that follow are the ones presented when the wizard is run with the default operator selection. That is the path through the wizard, not a recommendation to accept every pre-filled value: some screens, such as the DockerHub registry mirror, need an answer that suits the site. Screens that appear only when a particular operator has been selected are not described here. Those are covered with the operator that they belong to, in chapter 6.

Kubernetes operations:

If no deployment has been carried out earlier, then this screen offers only Deploy and Exit as options (figure 4.4). The fuller set of options shown in figure 4.3 appears only after a cluster exists.

Figure 4.4: Kubernetes setup TUI session: main operations screen before any deployment

Choosing a Kubernetes version:

The wizard lists the Kubernetes minor versions available in the installed packages (figure 4.1). Versions certified for NVIDIA AI Enterprise are marked with an asterisk, and the legend on the screen explains the marking.

Choosing a Kubernetes patch level:

The version chosen on the preceding screen is a minor version. This screen selects the patch level within it (figure 4.5). Two entries are offered: latest, and the specific patch release that the installed packages provide. The specific release is pre-selected.

Choosing latest means that the patch level is resolved when the deployment runs, rather than being fixed now. For an air-gapped cluster the specific release should be chosen, because only the patch level that was mirrored into the tarball is available (section B.2).

Figure 4.5: Kubernetes setup TUI session: choosing the Kubernetes patch level

DockerHub registry mirror:

A registry mirror can be specified so that images are pulled from it rather than from DockerHub (figure 4.6). The field is pre-filled with an NVIDIA-internal host that is not reachable from customer sites. The value should therefore be replaced with the address of a mirror that the cluster can reach, or it should be cleared if no mirror is used.

For an air-gapped cluster, the mirror is the local container registry that is set up as part of the air-gapped procedure (section B.2).

Figure 4.6: Kubernetes setup TUI session: DockerHub registry mirror

Basic cluster values:

The cluster name, domain, external FQDN, and the service and pod network CIDRs are set here (figure 4.2). The wizard pre-fills the network fields and tries to avoid ranges that overlap networks already known to BCM. These fields are described in detail in section 4.2.1.

Exposing the Kubernetes API server:

The Kubernetes API server can be exposed to the external network (figure 4.7). The wizard offers yes as the pre-selected answer, so accepting the default exposes the API server.

Leaving it unexposed means that the port is not opened to the external network, which is the safer choice. The API server is still reached through the NGINX proxy on the head node, so kubectl and Helm continue to work there. It should be exposed only if clients outside the cluster need to reach the API server directly.

Figure 4.7: Kubernetes setup TUI session: exposing the Kubernetes API server

Selecting the internal network:

This is the network that Kubernetes nodes use to reach each other (figure 4.8). One network must be selected, using <SPACE>. On a cluster configured for high availability a third option, failovernet, is offered alongside externalnet and internalnet.

If the nodes do not share an IP address on the selected network, then their internal networks are configured as fallback networks.

Figure 4.8: Kubernetes setup TUI session: selecting the internal network

Node assignment:

The control-plane, worker, and etcd nodes are assigned on four consecutive screens (figures 4.9–4.12). The head node is pre-selected as a control-plane node and as an etcd node.

Worker nodes are assigned on two of those screens. The first selects whole node categories, and the second selects individual nodes. Selecting a category is sufficient on its own. If the intended worker nodes are all members of a category that has been selected, then the screen for individual nodes can be left with nothing selected, and it lists only the nodes that the category selection did not already cover.

An odd number of etcd nodes should be selected, as the screen itself states. An even-numbered cluster still establishes a quorum, but it tolerates no more failures than the odd number below it: a three-node and a four-node cluster both keep working with one node down, so the fourth node adds cost without adding fault tolerance.

Figure 4.9: Kubernetes setup TUI session: assigning control-plane nodes

Figure 4.10: Kubernetes setup TUI session: assigning worker nodes by category

Figure 4.11: Kubernetes setup TUI session: assigning individual worker nodes

Figure 4.12: Kubernetes setup TUI session: etcd node assignment

Main Kubernetes component values:

The port numbers configured for the daemons have default assignments, but can be re-assigned as needed, and the etcd spool directory can be set (figure 4.13). The defaults are port 10443 for the API server proxy, and /var/lib/etcd for the etcd spool directory.

Figure 4.13: Kubernetes setup TUI session: main Kubernetes component values

Kubernetes network plug-in:

The CNI network plug-in is chosen here (figure 4.14). Calico, installed by way of the Tigera Operator, is pre-selected and is the recommended choice. Flannel is the alternative. These are the only two that the wizard offers. Both are described in section 4.2.1.

Figure 4.14: Kubernetes setup TUI session: selecting the CNI network plug-in

Kyverno policy engine:

The Kyverno policy engine is installed by default (figure 4.27). Kyverno is what BCM uses to enforce Pod Security Standards. PodSecurityPolicy provided that enforcement in older releases, and was removed in Kubernetes v1.25 (section 4.10.1).

If the policy engine is not installed, then Kubernetes Permissions Manager (section 4.15) still creates the namespace and the role bindings for each user, so users are separated from each other by RBAC. What is not created are the policies that restrict what a user may do inside their own namespace. On Kubernetes v1.25 and later there is no fallback for this, because the PodSecurityPolicy that used to provide it no longer exists.

A user in that situation can run privileged pods, and can mount arbitrary host paths. Those pods are created in the user’s own namespace, but neither capability is confined to it: a privileged pod and a host path mount reach the node filesystem and the container runtime. The exposure is therefore at node level, and through the node it extends to other workloads and to cluster data. The RBAC separation described above does not prevent this.

On a cluster with one administrator and no other users this can be acceptable. On a cluster that is shared between users it usually is not.

Kyverno policies:

The Kyverno policies are installed by default (figure 4.29). The policies are what implement the Pod Security Standards (section 4.14.2). The engine on its own enforces nothing.

On a cluster configured for high availability, a further screen asks whether Kyverno itself is to be set up for high availability (figure 4.28). That screen is not presented on a cluster with a single head node.

Namespaces excluded from the Kyverno policies:

Namespaces can be excluded from each of the cluster-wide policies (figure 4.15). Pods running in an excluded namespace are not subject to that policy. Each of the five policies takes its own comma-separated list, and each is pre-filled with the default namespace.

Further exclusions are added automatically during installation, based on the features that have been configured and the operators that have been selected. A namespace named on this screen is not created by the wizard, as the screen itself states.

Figure 4.15: Kubernetes setup TUI session: namespaces excluded from the Kyverno policies

Selecting the operators:

Operator packages are application managers, and are described further in chapter 6. The operators that can be selected are listed below, in the order in which the wizard presents them:

• NVIDIA GPU Operator

• AgentGateway

• AgentGateway CRDs

• Argo CD

• Grafana Alloy

• Grafana Loki

• Grafana Operator

• Ingress NGINX Controller (deprecated, section 4.2.4.2)

• Jupyter Kernel Operator (cm-jupyter-kernel-operator)

• Kgateway

• Knative Operator

• KubeFlow Training Operator

• Kubernetes Metrics Server

• Kubernetes MPI Operator (cm-kubernetes-mpi-operator)

• Kubernetes State Metrics

• LeaderWorkerSet operator

• MetalLB

• Network Operator

• NIM Operator

• PostgreSQL Operator (postgresql-operator)

• Prometheus Adapter

• Prometheus Operator Stack

• Spark Operator (spark-operator)

Four of these are selected by default: Kgateway, Kubernetes Metrics Server, Kubernetes State Metrics, and MetalLB (figure 4.16). The rest are installed only if they are selected here.

The NVIDIA GPU Operator is not selected by default. It manages the Kubernetes device plugin that exposes NVIDIA GPUs as schedulable resources (section 6.7), and the wizard provides no other way to install that plugin. A cluster with NVIDIA GPUs therefore cannot run GPU workloads until the operator is installed. The wizard does not check whether the cluster has GPUs, and does not warn if the operator is left unselected. It does not have to be installed here: it can also be added to a running cluster (section 6.7.2).

The CNI network plug-in is not part of this list. It is chosen on the earlier screen shown in figure 4.14.

Each operator that is selected adds its own configuration screens later in the wizard. Those screens are described with the operator concerned, in chapter 6.

Figure 4.16: Kubernetes setup TUI session: selecting the operators to install

Prior to BCM 11.33, Run:ai was installed as part of cm-kubernetes-setup (figure 4.17). From BCM 11.33 onwards, Run:ai is deployed separately using the cm-runai-setup wizard after Kubernetes setup is complete (section 6.11.2.2).

Figure 4.17: Kubernetes setup TUI session: Run:ai operator selection (prior to BCM 11.33)

MetalLB mode:

MetalLB advertises the Gateway addresses either by ARP, in layer 2 mode, or by BGP (figure 4.32). Layer 2 mode is pre-selected and needs no external peer. BGP mode advertises through an external peer.

Gateway IP addresses:

One IP address is requested for each Gateway that is configured (figure 4.18). This is the address at which the Gateway is reached.

The field is pre-filled with the highest address on the Kubernetes node network that is not already in use. Addresses assigned to devices, addresses used by the head node for its filesystem mounts, the dynamic range of the network, and ranges already claimed by MetalLB address pools are all excluded. On a cluster with one head node this is typically the address just below the head node itself.

The pre-filled value can be replaced. An address that is already assigned to a device, or that overlaps a range already in use, is rejected with an error.

Figure 4.18: Kubernetes setup TUI session: Gateway IP address

MetalLB IP address pools:

Address pools can be defined for MetalLB to allocate from (figure 4.19). The list is empty by default. Rows are added with <ALT>-<A> and removed with <ALT>-<D>.

Leaving the list empty is correct on the default path. The Gateway does not take its address from a pool defined here. A pool holding only the Gateway address given on the previous screen is generated during the installation itself. Pools are added on this screen only for other services of type LoadBalancer.

Figure 4.19: Kubernetes setup TUI session: MetalLB IP address pools

Exposing the Kubernetes Gateways:

The Gateways can be exposed to the external network by enabling the Gateway proxy on the head nodes and edge directors (figure 4.34). This is enabled by default.

Gateway proxy configuration:

The port that the Gateway proxy listens on is set here (figure 4.35). The default is port 443.

BCM NVIDIA packages:

The cuda-dcgm package can be installed into the software images used by the Kubernetes nodes (figure 4.20). The screen lists the images that would be modified. The default is not to install it.

Figure 4.20: Kubernetes setup TUI session: installing the BCM NVIDIA packages

BCM NVIDIA Container Toolkit package:

The cm-nvidia-container-toolkit package can be installed on the head node and into the software images (figure 4.21). The screen lists the paths that would be modified. The default is not to install it.

Figure 4.21: Kubernetes setup TUI session: installing the BCM NVIDIA Container Toolkit package

Both NVIDIA package screens are skipped if the GPU Operator is selected:

The two preceding screens are presented only when the NVIDIA GPU Operator has not been selected. If it has been selected, then the wizard decides both answers itself and does not ask:

• the BCM NVIDIA packages are installed, because they are almost always wanted alongside the operator

• the BCM NVIDIA Container Toolkit package is not installed, because the GPU Operator deploys and manages the container toolkit itself by default (chapter 6)

Running cm-kubernetes-setup with the --adv-configuration-bcm-nvidia-packages option presents both screens even when the GPU Operator has been selected, so that the answers can be set by hand.

Permission manager:

Kubernetes Permissions Manager (section 4.15) is installed by default (figure 4.22). As the screen states, it is needed only if non-root users are to use the cluster.

Figure 4.22: Kubernetes setup TUI session: installing Kubernetes Permissions Manager

Storage class:

The storage class for the cluster is configured here (figure 4.23). Local path storage is enabled by default, and is set as the default storage class. The alternative is to set no default storage class.

Figure 4.23: Kubernetes setup TUI session: configuring the Kubernetes storage class

Local path storage pool:

If local path storage is enabled, then the directory that backs it is set here (figure 4.24). The default is /cm/shared/apps/kubernetes/<cluster>/var/volumes, where <cluster> is the name of the Kubernetes cluster. A custom provisioner image can also be given, and is optional.

Figure 4.24: Kubernetes setup TUI session: configuring the local path storage pool

Summary:

The final screen offers the deployment and configuration file options (figure 4.25). The configuration can be deployed straight away, or written to a file to be deployed later with -c, or both. Save config & deploy is pre-selected.

The minimal config variants write a configuration file that records only the values that differ from the defaults. The full variants record every value. A minimal file is easier to read and to carry between systems. A full file is a complete record of the deployment, and does not change meaning if a default changes in a later BCM release.

Figure 4.25: Kubernetes setup TUI session: summary screen

4.2.9 Testing Kubernetes

To test that Kubernetes works, the cluster-info command can be run. For example, on the head node, basecm11:

Example

[root@basecm11 ~]# module load kubernetes      #not actually needed -- autoloaded these days
[root@basecm11 ~]# kubectl cluster-info
Kubernetes control plane is running at https://127.0.0.1:10443
CoreDNS is running at https://127.0.0.1:10443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

After cm-kubernetes-setup finishes, and the regular nodes have been rebooted, the state of the nodes can be found by running the get nodes command:

Example

[root@basecm11 ~]# kubectl get nodes
NAME      STATUS   ROLES                  AGE   VERSION
node001   Ready    worker                 18h   v1.35
node002   Ready    worker                 18h   v1.35
basecm11  Ready    control-plane          18h   v1.35

A six node cluster should show the following Kubernetes installed add-ons, when using kubectl with the get all -n kube-system option (some lines truncated):

Example

[root@basecm11 ~]# kubectl get all -n kube-system
NAME                                           READY   STATUS    RESTARTS   AGE
pod/calico-kube-controllers-58497c65d5-skhgw   1/1     Running   0          26m
pod/calico-node-27xj7                          1/1     Running   0          26m
pod/calico-node-6hmm5                          0/1     Running   1          26m
pod/calico-node-987qv                          1/1     Running   0          26m
pod/calico-node-gcbcm                          1/1     Running   0          26m
pod/calico-node-hlsrj                          1/1     Running   0          26m
pod/calico-node-q7k4v                          1/1     Running   0          26m
pod/calico-node-qdbq5                          0/1     Running   0          26m
pod/calico-node-v2dxj                          1/1     Running   0          26m
pod/coredns-6768db756-8l9fs                    1/1     Running   0          26m
pod/coredns-6768db756-cs58q                    1/1     Running   0          26m
pod/kube-state-metrics-758ccc75d6-75dsn        1/1     Running   0          26m
pod/metrics-server-7b477dd7b9-2drkg            1/1     Running   0          26m
pod/metrics-server-7b477dd7b9-z6nch            1/1     Running   0          26m
NAME                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
service/calico-typha         ClusterIP   10.150.121.25    <none>        5473/TCP                 26m
service/kube-dns             ClusterIP   10.150.255.254   <none>        53/UDP,53/TCP,9153/TCP   26m
service/kube-state-metrics   ClusterIP   None             <none>        8080/TCP,8081/TCP        26m
service/metrics-server       ClusterIP   10.150.99.149    <none>        443/TCP                  26m
NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/calico-node   8         8         6       8            6           kubernetes.io/os=linux   26m
NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/calico-kube-controllers   1/1     1            1           26m
deployment.apps/calico-typha              0/0     0            0           26m
deployment.apps/coredns                   2/2     2            2           26m
deployment.apps/kube-state-metrics        1/1     1            1           26m
deployment.apps/metrics-server            2/2     2            2           26m
NAME                                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/calico-kube-controllers-58497c65d5   1         1         1       26m
replicaset.apps/calico-typha-68857595fc              0         0         0       26m
replicaset.apps/coredns-6768db756                    2         2         2       26m
replicaset.apps/kube-state-metrics-758ccc75d6        1         1         1       26m
replicaset.apps/metrics-server-7b477dd7b9            2         2         2       26m

The administrator can now configure the cluster to suit the particular site requirements.

4.3 Using GPUs With Kubernetes: NVIDIA GPUs

4.3.1 Prerequisites

The GPUs have to be recognized by the nodes, and have the appropriate drivers (such as nvidia-driver) installed. Details on how to do this are given in Chapter 9 of the Installation Manual.

To verify the GPUs are recognized and have drivers in place, the nvidia-smi command can be run. The response displayed for a GPU should look similar to the following:

Example

root@node001:~# nvidia-smi
Mon May  5 13:03:52 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.82                 Driver Version: 570.82         CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA Graphics Device         On  |   00000008:01:00.0 Off |                    0 |
| N/A   38C    P0            184W / 1200W |       1MiB / 189471MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA Graphics Device         On  |   00000009:01:00.0 Off |                    0 |
| N/A   36C    P0            160W / 1200W |       1MiB / 189471MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

If a non-BCM Containerd has already been deployed before Kubernetes is deployed, then cm-kubernetes-setup may replace an existing Containerd configuration file in order to enable NVIDIA GPU integration using a Kubernetes CNI plugin. This is because Containerd is configured by cm-kubernetes-setup, overwriting any previous configuration.

4.3.2 New Kubernetes Installation

As part of the setup, cm-kubernetes-setup assigns a new role to the Kubernetes worker nodes: generic::containerd.

The role has a Configurations submode, in which the containerd-cri object can be configured. The entry for Filename specifies the path to the cri.toml file, which contains content used by the container runtime interface on the Kubernetes worker nodes that have been assigned the role.

Example

[basecm11->configurationoverlay[kube-default-worker]->roles]% use generic::containerd
[...]->roles[generic::containerd]]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Name                             generic::containerd
Type                             GenericRole
Add services                     yes
Services                         containerd
Configurations                   <2 in submode>
Environments                     <1 in submode>
Exclude list snippets            <1 in submode>
Data node                        no
[...]->roles[generic::containerd]]% configurations
[...]->roles[generic::containerd]->configurations]% use containerd-cri
[...]->roles[generic::containerd]->configurations[containerd-cri]]% show
Parameter                        Value
-------------------------------- --------------------------------------------------
Name                             containerd-cri
Type                             static
Create directory                 yes
Filename                         /cm/local/apps/containerd/var/etc/conf.d/cri.toml
Filemask directory               0644
User name
Group name
Disabled                         no
Service action on write          RESTART
Service stop on failure          yes
Content                          <645B>
Filemask                         0644

The file with the CRI (Container Runtime Interface) configuration is created in directory:

     /cm/local/apps/containerd/var/etc/conf.d

and included into the main Containerd configuration file:

     /cm/local/apps/containerd/var/etc/config.toml

with the imports statement:

     imports = ["/cm/local/apps/containerd/var/etc/conf.d/*.toml"]

Whatever the container runtime that is selected, if NVIDIA GPU integration is required then the NVIDIA container toolkit is taken care of by the installer. The configuration for Containerd is added as a separate configuration file in the configurations submode, as nvidia-cri.

NVIDIA GPUs are integrated into Kubernetes using the NVIDIA GPU operator. This is discussed further in section 6.7.

4.3.3 Existing Kubernetes Installation

The NVIDIA GPU operator can always be deployed through Helm. The official documentation for the NVIDIA GPU operator is at

https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/overview.html

4.4 Using GPUs With Kubernetes: AMD GPUs

4.4.1 Prerequisites

The GPUs have to be recognized by the node. One way to check this from within BCM is to run sysinfo for the node:

Example

[basecm11->device[basecm11]]% sysinfo | grep GPU
Number of GPUs                           1
GPU Driver Version                       4.18.0-193.el8.x86_64
GPU0 Name                                Radeon Instinct MI25

In order to make Kubernetes aware of nodes that have AMD GPUs, the AMD GPU device plugin has to be deployed as a DaemonSet inside Kubernetes. The official GitHub repository that hosts this plugin can be found at:

https://github.com/RadeonOpenCompute/k8s-device-plugin

The device plugin requires Kubernetes v1.16+, which has been around since BCM version 9.0. With some extra instructions, the plugin can also be made a part of BCM version 8.2.

The DaemonSet YAML file can be deployed with:

Example

kubectl create -f https://raw.githubusercontent.com/RadeonOpenCompute/k8s-device-plugin/v1.16/
k8s-ds-amdgpu-dp.yaml

4.4.2 Managing The YAML File Through CMDaemon

The plugin can be added by the user using the Kubernetes appgroups as an application. In the session that follows, it is given the arbitrary name device-plugin:

Example

[root@basecm11 ~]# wget https://raw.githubusercontent.com/RadeonOpenCompute/k8s-device-plugin/
v1.16/k8s-ds-amdgpu-dp.yaml -O /tmp/k8s-ds-amdgpu-dp.yaml
[root@basecm11 ~]# cmsh
[basecm11]% kubernetes
[basecm11->kubernetes[default]]% appgroups
[basecm11->kubernetes[default]->appgroups]% add amd
[basecm11->kubernetes*[default*]->appgroups*[amd*]]% applications
[basecm11->kubernetes*[default*]->appgroups*[amd*]->applications]% add device-plugin

The configuration of the plugin can be set to the YAML file, by setting the config parameter to take the value of the YAML file path.

Example

[basecm11->...[amd*]->applications*[device-plugin*]]% set config /tmp/k8s-ds-amdgpu-dp.yaml
[basecm11->kubernetes*[default*]->appgroups*[amd*]->applications*[device-plugin*]]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Name                             device-plugin
Revision
Format                           Yaml
Enabled                          yes
Config                           <914B>
Environment                      <0 in submode>
Exclude list snippets            <0 in submode>
[basecm11->kubernetes*[default*]->appgroups*[amd*]->applications*[device-plugin*]]% commit

The YAML file can also be edited within cmsh after it has been set, by running set config without a value.

There are older releases available, starting from Kubernetes v1.10, if needed. Saving this device-plugin YAML should result in pods being scheduled on all the non-tainted nodes, as seen by listing the pods (some columns elided):

[root@basecm11 ~]# kubectl get pod -n kube-system -l name=amdgpu-dp-ds -o wide
NAME                                   READY   STATUS    ... IP               NODE   ...
amdgpu-device-plugin-daemonset-66jl7   1/1     Running   ... 172.29.112.135   gpu001 ...
amdgpu-device-plugin-daemonset-8mh9w   1/1     Running   ... 172.29.152.130   gpu002 ...

4.4.3 Including Head Nodes as part of the DaemonSet

BCM taints head nodes, so that they do not run non-critical pods. The taint can be removed with the “-” operator to allow non-critical pods to run:

Example

kubectl taint nodes basecm11 node-role.kubernetes.io/control-plane-

However, a more specific exception can be configured in the DaemonSet itself.

Within the YAML file, the following existing tolerations definition has to be modified, from:

      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists

to:

      tolerations:
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule
        operator: Exists

The modified toleration tolerates this taint, and therefore has the device plugin run on such tainted nodes.

4.4.3.1 Verifying That AMD GPUs Are Recognized By Kubernetes

If Kubernetes is aware of the AMD GPUs for a node then several mentions of amd.com/gpu are displayed when running the kubectl describe node command for the node. The following session shows output for a node gpu001, ellipsized for clarity:

Example

[root@basecm11 ~]# kubectl describe node gpu001
Name:                 gpu001
...
Capacity:
  amd.com/gpu:        3
  cpu:                64
  ephemeral-storage:  1813510Mi
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             527954676Ki
  pods:               50
...

4.4.4 Running The DaemonSet Only On Specific Nodes

The AMD GPU device plugin, unlike the NVIDIA GPU device plugin Daemonset, is scheduled to run on each Kubernetes host. This means that it runs even if the host has no GPU.

This can be prevented with the following steps:

A LabelSet can be created using cmsh, and the nodes or categories that have GPUs are assigned within the labelsets mode:

Example

[root@basecm11 ~]# cmsh
[basecm11]% kubernetes
[basecm11->kubernetes[default]]% labelsets
[basecm11->kubernetes[default]->labelsets]% use nvidia
[basecm11->kubernetes[default]->labelsets[nvidia]]% ..   #but, we're using AMD GPUs, so let's go back up:
[basecm11->kubernetes[default]->labelsets]% add amd
[basecm11->kubernetes*[default*]->labelsets*[amd*]]% set labels nvidia.com/amd-gpu-accelerator=
[basecm11->kubernetes*[default*]->labelsets*[amd*]]% append categories gpu-nodes
[basecm11->kubernetes*[default*]->labelsets*[amd*]]% commit

This assigns the labels to the nodes with GPUs. This can be verified with:

Example

kubectl get nodes -l nvidia.com/amd-gpu-accelerator=
NAME     STATUS   ROLES    AGE   VERSION
gpu001   Ready    <none>   66m   v1.32.6
gpu002   Ready    <none>   66m   v1.32.6
...

The DaemonSet YAML can now be adjusted to only run the device plugin on nodes with this new label. This can be done by adding an affinity block after the tolerations block:

Example

      tolerations:
      - key: CriticalAddonsOnly  # toleration may be different, if changes were made to it
        operator: Exists
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: 'nvidia.com/amd-gpu-accelerator'
                    operator: Exists

This results in the device plugin pods being removed immediately from all nodes that do not have the label.

4.4.5 Running An Example Workload

An example workload can be run as described in the official AMD GPU Kubernetes device plugin documentation at:

https://github.com/RadeonOpenCompute/k8s-device-plugin/tree/v1.16#example-workload

Thus it should now be possible to run:

[root@basecm11 ~]# kubectl create -f https://raw.githubusercontent.com/RadeonOpenCompute/
k8s-device-plugin/v1.16/example/pod/alexnet-gpu.yaml

The YAML requests only one GPU at the bottom of the YAML file:

apiVersion: v1
kind: Pod
metadata:
  name: alexnet-tf-gpu-pod
  labels:
    purpose: demo-tf-amdgpu
spec:
  containers:
    - name: alexnet-tf-gpu-container
      image: rocm/tensorflow:latest
      workingDir: /root
      env:
      - name: HIP_VISIBLE_DEVICES
        value: "0" # # 0,1,2,...,n for running on GPU and select the GPUs, -1 for running on CPU
      command: ["/bin/bash", "-c", "--"]
    args: ["python3 benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py --model=alexnet;\
trap : TERM INT; sleep infinity & wait"]
      resources:
        limits:
          amd.com/gpu: 1 # requesting a GPU

Container creation might take a while due to the image size. Once scheduled, it prints out that it found exactly one GPU, and proceeds to run a TensorFlow workload.

Example

[root@basecm11 ~]# kubectl logs -f alexnet-tf-gpu-pod
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/compat/v2_compat.py:96:
disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed
in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
2021-01-08 21:03:29.222293: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104]
    CPU Frequency: 2495445000 Hz
2021-01-08 21:03:29.222398: I tensorflow/compiler/xla/service/service.cc:168]
    XLA service 0x39f62f0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-01-08 21:03:29.222420: I tensorflow/compiler/xla/service/service.cc:176]
          StreamExecutor device (0): Host, Default Version
2021-01-08 21:03:29.223754: I tensorflow/stream_executor/platform/default/dso_loader.cc:48]
    Successfully opened dynamic library libamdhip64.so
2021-01-08 21:03:31.635339: I tensorflow/compiler/xla/service/service.cc:168]
    XLA service 0x3a40bb0 initialized for platform ROCM (this does not guarantee that XLA will be used). Devices:
2021-01-08 21:03:31.635363: I tensorflow/compiler/xla/service/service.cc:176]
          StreamExecutor device (0): Device 738c, AMDGPU ISA version: gfx908
2021-01-08 21:03:31.931125: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734]
    Found device 0 with properties:
          pciBusID: 0000:27:00.0 name: Device 738c     ROCm AMD GPU ISA: gfx908
          coreClock: 1.502GHz coreCount: 120 deviceMemorySize: 31.98GiB deviceMemoryBandwidth: 1.12TiB/s
...
TensorFlow:  2.3
Model:       alexnet
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  512 global
             512 per device
Num batches: 100
Num epochs:  0.04
Devices:     ['/gpu:0']
...

Had more GPUs been requested, more would have been made available to the container.

For comparison, a CPU version of the container is also available. The official instructions can be referred to for these, too.

4.5 Kubernetes Configuration Overlays

A list of configuration overlays can be seen from within configurationoverlay mode:

Example

[basecm11->configurationoverlay]% list
Name (key)          Priority Nodes             Categories Roles
------------------- -------- ----------------- ---------- -------------------------...
kube-default-etcd   500      node001..node003             Etcd::Host
kube-default-master 510      node001..node003             generic::containerd, kube...
kube-default-worker 500      node004..node006  default    generic::containerd, kube...
kube-default-netq   501      node001                      generic::netq

The NetQ configuration overlay is only normally displayed if NetQ is deployed.

Configuration overlays can be used to manage the Kubernetes services used with a particular configuration. For example, when managing the Kubernetes services used for a Kubernetes engine within an Auto Scale tracker (section 8.4.9 of the Administrator Manual).

4.6 Removing A Kubernetes Cluster

A Kubernetes cluster can be removed using cm-kubernetes-setup with the --remove and --yes-i-really-mean-it options. Also, if there is more than one cluster present, then the cluster name must be specified using the --cluster parameter.

A removal run looks as follows (some output ellipsized):

Example

[root@basecm11 ~]# cm-kubernetes-setup --remove --cluster default --yes-i-really-mean-it
Connecting to CMDaemon
Executing 20 stages
################### Starting execution for 'Kubernetes Setup'
  - kubernetes
  - docker
## Progress: 0
#### stage: kubernetes: Get Kube Cluster
## Progress: 5
#### stage: kubernetes: Check Kube Cluster Exists
## Progress: 10
#### stage: kubernetes: Find Installed Components
## Progress: 15
#### stage: kubernetes: Find Files On Headnodes
## Progress: 20
#### stage: kubernetes: Firewall Zone Close
## Progress: 25
#### stage: kubernetes: Firewall Interface Close
## Progress: 30
#### stage: kubernetes: Firewall Policy Close
## Progress: 35
#### stage: kubernetes: Nginx Reverse Proxy Close
## Progress: 40
#### stage: kubernetes: IP Ports Close
## Progress: 60
#### stage: kubernetes: Remove Installed Components
## Progress: 65
#### stage: kubernetes: Remove Files On Headnodes
## Progress: 70
#### stage: kubernetes: Remove Etcd Spool
## Progress: 80
#### stage: kubernetes: Set Reboot Required
You need to reboot 2 nodes to cleanup the network configuration
## Progress: 85
#### stage: kubernetes: Collection Update Provisioners
## Progress: 100
Took:     00:08 min.
Progress: 100/100
################### Finished execution for 'Kubernetes Setup', status: completed
Kubernetes Setup finished!

Using the --remove option removes the Kubernetes cluster configuration from BCM, unassigns Kubernetes-related roles—including the EtcdHost role—and removes Kubernetes health checks. The command does not remove packages that were installed with a cm-kubernetes-setup command before that.

After the disabling procedure has finished, the cluster has no Kubernetes configured and running.

4.7 Kubernetes Cluster Configuration Options

Kubernetes allows many Kubernetes clusters to be configured. These are separated sets of hosts with different certificates, users and other global settings.

When carrying out the Kubernetes setup run, a Kubernetes cluster name is asked, and a new object with the cluster settings is then added into the CMDaemon configuration. The administrator can change the settings of the cluster from within the kubernetes mode of cmsh, or within the Kubernetes Clusters options window of Base View accessible using the navigation path Containers > Kubernetes Clusters.

The cmsh equivalent looks like:

Example

[root@basecm11 ~]# cmsh
[basecm11]% kubernetes list
Name (key)
------------------
default
[basecm11]% kubernetes use default
[basecm11->kubernetes[default]]% show
Parameter                            Value
------------------------------------ ------------------------------------------------------
Name                                 default
Revision
Etcd Cluster                         kube-default
Pod Network                          kube-default-pod
Pod Network Node Mask
Internal Network                     internalnet
KubeDNS IP                           10.150.255.254
Kubernetes API server
Kubernetes API server proxy port     10443
App Groups                           <1 in submode>
Label Sets                           <3 in submode>
Notes
Version                              1.27.10-150500.1.1
Trusted domains                      basecm11.openstacklocal,master,localhost,10.141.255.254
Module file template                 <1.01KiB>
Kubeadm init file                    <1.19KiB>
Service Network                      kube-default-service
Kubeadm CERT Key                     *********
Kube CA Cert                         *********
Kube CA Key                          *********
Kubernetes users                     <0 in submode>
External                             no
External Kubernetes Ingress server
External port                        0
Capi template                        no
Capi namespace                       default
Kubernetes management cluster
[basecm11->kubernetes[default]]%

The preceding kubernetes mode parameters are described in table 4.1:

Table 4.1: kubernetes mode parameters

4.8 EtcdCluster

The EtcdCluster mode sets the global etcd cluster settings. It can be accessed using the top level etcd mode of cmsh.

* Boolean (takes yes or no as a value)

Table 4.2: EtcdCluster role parameters and etcd options

4.9 Kubernetes Roles

Kubernetes roles include the following roles:

• EtcdHost (here)

• KubernetesApiServerProxy (here)

• Kubelet (here)

• generic::containerd (here)

• generic::netq (here)

When nodes are configured using Kubernetes roles, then settings in these roles may sometimes use the same values (pointer variables).

Example

[basecm11->configurationoverlay[kube-default-etcd]->roles[Etcd::Host]]% get etcdcluster
kube-default

and

[basecm11->kubernetes[default]]% get etcdcluster
kube-default

Pointer variables such as these have definitions that are shared across the roles, as indicated by the parameter description tables for the roles, and which are described in the following pages.

In cmsh, the roles can be assigned:

• for individual nodes using the roles submode of device mode

• for a category using the roles submode of a category

• for a configuration overlay using the roles submode of configurationoverlay mode

4.9.1 EtcdHost Role

The EtcdHost role is used to configure and manage the etcd service for a node.

The etcd service manages the etcd database, which is a hierarchical distributed key-value database. The database is used by Kubernetes to store its configurations. The EtcdHost role parameters are described in table 4.3:

* Boolean (takes yes or no as a value)

Table 4.3: EtcdHost role parameters and etcd options

The etcd settings are updated by BCM in /cm/local/apps/etcd/current/etc/cm-etcd.conf.

4.9.2 The KubernetesAPIServerProxy Role

The KubernetesApiServerProxy role sets up a proxy that provides the entry point for one or more instances of the Kubernetes API server. The proxy runs on every node of a Kubernetes cluster instance, including the head node.

If multiple Kubernetes control-plane nodes are present, then it enables HA for the Kubernetes control plane components (section 4.1).

This means that local port 10443 on all these nodes load balances over all control-plane node API servers.

Port 10443 can also be opened up for external access. The cm-kubernetes-setup wizard (section 4.2) in one of its screens (figure 4.26) prompts for this.

Figure 4.26: Wizard question

If the prompt is answered with no, then the firewall role on the head node can still be customized with cmsh later on:

Example

root@basecm11:~# cmsh
[basecm11]% device use master
[basecm11->device[basecm11]]% roles
[basecm11->device[basecm11]->roles]% use firewall
[basecm11->device[basecm11]->roles[firewall]]% openports
[basecm11->device[basecm11]->roles[firewall]->openports]% list
Index  Action    Network      Port         Destination  Protocol  Address      Description
------ --------- ------------ ------------ ------------ --------- ------------ ------------------------
[basecm11->device[basecm11]->roles[firewall]->openports]% add ACCEPT net 10443 tcp fw
ok.
[basecm11->device*[basecm11*]->roles*[firewall*]->openports[0]]% commit
[basecm11->device[basecm11]->roles[firewall]->openports[0]]% list
Index  Action    Network      Port         Destination  Protocol  Address      Description
------ --------- ------------ ------------ ------------ --------- ------------ ------------------------
0      ACCEPT    net          10443        fw           TCP       0.0.0.0/0
[basecm11->device[basecm11]->roles[firewall]->openports[0]]%
Thu Oct 31 15:58:56 2024 [notice] basecm11: Service shorewall was restarted

4.9.3 The KubernetesIngressServerProxy Role

This role used to provide the “Ingress Proxy” (also known as “Ingress Server Proxy”) functionality, configuring the head-node NGINX to expose the Kubernetes ingress externally. In BCM 11.33.0 and later the role is almost a placeholder, because the head-node NGINX configuration is now offloaded to the Kubernetes setup itself, which generates and applies it using the Configure Gateway proxy to route traffic from head node(s) to Gateways item of cm-kubernetes-setup (section 4.17.6.3).

The role and the related Kubernetes cluster entity parameters (for example Ingress Proxy Enable, Ingress Proxy Listen Port, Ingress Proxy Backend Port) keep their original names for backward compatibility.

4.9.4 The Kubelet Role

The Kubelet role is used to configure and manage the kubelet service. BCM takes care of joining these nodes with kubeadm join when needed.

Control plane: The Kubelet role has a parameter Control plane that is set to the value yes for control plane nodes. In that case it also runs Kubernetes control plane services, such as:

• Kubernetes API server (kube-apiserver),

• Kubernetes scheduler (kube-scheduler),

• Kubernetes controller manager (kube-controller-manager),

• Kubernetes network proxy (kube-proxy),

• CoreDNS (coredns).

Workers: The Kubelet role has a parameter worker that is set to yes for worker nodes. In that case, the control plane pods will not be running on the node.

The Kubelet role parameters are described in table 4.4:

* Boolean (takes yes or no as a value)

Table 4.4: Kubelet role parameters and kubelet options

Further details on the Kubelet service can be found at https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/.

4.9.4.1 CPU Manager And Topology Manager Policy On GPU Nodes

From BCM 11.34 onwards, GPU worker nodes are automatically configured with a static CPU manager policy and a best-effort topology manager policy. These settings enable workloads running on GPU nodes to request exclusive CPU allocation and benefit from NUMA-aware scheduling. The default kubelet settings (cpuManagerPolicy=none and topologyManagerPolicy=none) are suboptimal for GPU workloads: without CPU pinning, containers share CPUs via time-slicing, which causes cache thrashing and inconsistent throughput in the GPU data pipeline; and without topology awareness, CPU, memory, and GPU allocations may be placed on different NUMA nodes, forcing host → device data transfers to cross the inter-socket interconnect at reduced bandwidth.

The following kubelet settings are applied automatically to nodes that have GPUs:

• cpuManagerPolicy: static enables the static CPU manager, which grants exclusive CPU access to guaranteed-QoS containers that request whole CPUs.

• topologyManagerPolicy: best-effort enables the topology manager, which attempts to align CPU, memory, and device (GPU) allocations to the same NUMA node.

• topologyManagerScope: pod applies topology alignment at the pod level, so that all containers in a pod share the same NUMA node affinity.

• kubeReserved: {cpu: 250m} reserves a small amount of CPU for Kubernetes system components.

Control plane nodes and non-GPU worker nodes use the default kubelet configuration and are not affected by these settings. Clusters deployed before BCM 11.34 are also not affected, since kubeadm upgrade reuses the existing in-cluster kubelet configuration.

Further details on the CPU manager and topology manager policies can be found at https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/ and https://kubernetes.io/docs/tasks/administer-cluster/topology-manager/.

4.9.4.2 The options Submode

[basecm11->configurationoverlay[kube-default-master]->roles[kubelet]]% help options
Name:
options - Manage kubelet container command options
Usage:
options [kubelet]
options [kubelet] <filename>
options [kubelet] <filename> set [name] key=value
options [kubelet] <filename> set [name] key
options [kubelet] <filename> clear [name] key
Examples:
options kube-apiserver.yaml set advertise-address=192.168.200.148
[basecm11->configurationoverlay[kube-default-master]->roles[kubelet]]% options
Filename             Container       Key                   Value
-------------------  --------------- --------------------- ------------------------------------
kube-apiserver.yaml  kube-apiserver  cors-allowed-origins  https://name.run.ai
kube-apiserver.yaml  kube-apiserver  oidc-client-id        runai
kube-apiserver.yaml  kube-apiserver  oidc-issuer-url       https://app.run.ai/auth/realms/name
kube-apiserver.yaml  kube-apiserver  oidc-username-prefix  -