> This page is for version 11 (default). > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Kubernetes Kubernetes is an open-source platform for automating deployment, scaling, and operations of application containers across clusters of hosts. With Kubernetes, it is possible to: * scale applications on the fly * seamlessly update running services * optimize hardware usage by using only the resources that are needed BCM provides the administrator with the required packages, allows Kubernetes to be set up on a cluster, and manages and monitors Kubernetes. More information about the design of Kubernetes, its command line interfaces, and other Kubernetes-specific details, can be found at the official online documentation at [https://kubernetes.io/docs/](https://kubernetes.io/docs/). To deploy most of Kubernetes, NVIDIA Base Command Manager from version 10.0 onwards uses `kubeadm` ([https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/)). BCM runs CoreDNS, the Kubelet component ([https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/)), and the entire control plane inside Kubernetes. The distributed key-value store used for Kubernetes, etcd, is typically run inside of Kubernetes on control planes. However, in BCM etcd is run outside of Kubernetes. Kubernetes integration with BCM is available for Kubernetes v1.34, v1.35, and v1.36 at the time of writing of this paragraph (July 2026). Kubernetes runs on the `x86_64` and `aarch64` architectures for all the BCM-supported Linux distributions. For a more up-to-date status, the feature matrix at [https://service.bcm.nvidia.com/feature-matrix/](https://service.bcm.nvidia.com/feature-matrix/) can be checked. The version that is running on a cluster can be found with: **Example** ```bash root@basecm11:~# kubeadm version -o short v1.34.9 ``` The versions that are available with BCM integration can be found with the `cm-kubernetes-setup` utility (section [4.2](#42-kubernetes-setup)): **Example** ```bash root@basecm11:~# cm-kubernetes-setup --list-versions 1.36 1.35 (NVIDIA AI Enterprise certified) 1.34 (NVIDIA AI Enterprise certified) ``` ## 4.1 Reference Architecture A reference architecture for Kubernetes in BCM comprises: * *etcd nodes*: An etcd cluster—the Kubernetes distributed key-value storage—runs on an odd number (1, 3, 5 ...) of nodes. * *control plane nodes*: typically run on head nodes or on dedicated nodes. 2 or 3 are recommended. * *worker nodes*: typically run on regular nodes that are designated to run user workloads. To avoid a single point of failure and to achieve high availability, a minimum of three nodes is recommended for etcd, and a minimum of two control plane nodes is recommended. **Kubernetes API server proxy:** An NGINX server is configured on the head node(s) and on all other nodes involved in the Kubernetes cluster (control plane and worker nodes). This NGINX server takes care of exposing the Kubernetes API server on a specific port, and load balances requests to the control plane nodes. Should one of those nodes go down it can detect this and stop sending requests until the node comes back up. Since BCM version 8.2, multiple clusters of Kubernetes can be deployed. In such a configuration, the same nodes cannot be shared across different Kubernetes clusters. Because of the NGINX server, a port is reserved on the head node(s) for every Kubernetes cluster. This is required for Kubernetes HA , and it also allows `kubectl` and other tools such as Helm to be used from the head node, to access each Kubernetes cluster. **Gateway Proxy:** The same NGINX server is also used as the entry point on the head node(s) to forward traffic to the Kubernetes `Gateway`s. This functionality is referred to as the “Gateway proxy” and is described in detail in section [4.17](#417-kubernetes-gateway). ### 4.1.1 Hardware Requirements To run properly, Kubernetes deployment requires a per-node minimum of * 16 GB RAM * two CPU cores Without these minimum requirements, deployment may succeed, but further configuration, such as starting up Jupyter kernels, may fail with problems that are hard to diagnose. For example, error reporting may not show a lack of resources because some components are automatically deleted. ## 4.2 Kubernetes Setup The usual, and recommended way, to install Kubernetes with BCM is to install it interactively from a TUI session (section [4.2.8](#428-kubernetes-setup-from-a-tui-session)). An alternative way, typically used for automated non-interactive setups, is to use a plain command line specification (section [4.2.7](#427-kubernetes-setup-from-the-command-line)), which usually results in a long command line. BCM deploys Kubernetes with the `cm-kubernetes-setup` utility, which is part of the `cm-setup` package. Several recent versions of Kubernetes are offered (figure [4.1](#fig-k8s-version-selection)): ![Kubernetes setup TUI session (section 4.2.8): version selection screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/abfe8e91b3b608b84116c97fe48b9be4e1d1489e020a90da74c3d0a2a830ee5a/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/cm-kubernetes-setup-versions.png) *Figure 4.1: Kubernetes setup TUI session (section [4.2.8](#428-kubernetes-setup-from-a-tui-session)): version selection screen* The TUI version selection offers the Kubernetes minor versions that the installed packages provide. The patch ([http://semver.org](http://semver.org)) level within the chosen minor version is then asked for on the following screen (figure [4.5](#fig-kube-urwid-wizard-patch-level)). At the time of writing of this section (August 2026), minor versions 1.34, 1.35, and 1.36 are the recommended and tested versions offered, providing patch levels 1.34.10, 1.35.7, and 1.36.3 respectively. The selected patch version should normally be retained. In special cases, the patch version can be changed in the saved configuration file: * The patch version can be removed to use the latest released patch version. However, because the latest version can change between runs, this can make the installation non-idempotent. * The patch version can be set to a version other than the recommended and tested version. * For an air-gapped deployment (Appendix [B](/containerization/kubernetesairgapinstallation)), each exact Kubernetes version expects a specific combination of system packages and container images, all of which must be present in the air-gapped deployment bundle. From BCM 11.34 onwards, the wizard validates this before installation begins, and fails early with a specific error message if the selected version does not match what was downloaded, instead of failing later during installation. Two kinds of error types can be shown, depending on which mismatch is detected: 1. A package version mismatch names each affected package and its found and expected versions, for both the head node and the default software image: **Example** ```bash Required package versions are not available in the air-gapped package set (kube version or BCM operator package version mismatch): /: - kubeadm==1.36.3 (found: 1.34.10-1.1) - kubectl==1.36.3 (found: 1.34.10-1.1) - kubelet==1.36.3 (found: 1.34.10-1.1) /cm/images/default-image: - kubeadm==1.36.3 (found: 1.34.10-1.1) - kubectl==1.36.3 (found: 1.34.10-1.1) - kubelet==1.36.3 (found: 1.34.10-1.1) ``` 2. A Helm chart version mismatch names the chart and its found and expected versions: **Example** ```bash Deployment is impossible due to missing or mismatched HELM charts: HELM charts version mismatch: network-operator: ['26.1.0'] not in ['26.1.1'] ``` * BCM provides or uses the following Kubernetes-related packages: * `conntrack` (`conntrack-tools` on RHEL-based distributions) and `nginx`: These distribution packages are always installed on the head node(s) and on the control-plane and worker node(s). * `cm-etcd`: This BCM package is installed on the nodes selected for etcd. In a similar way to the case of Harbor or Docker Registry (section [3.2](/containerization/dockerregistries#32-container-registries-setup-and-configuration)), the nodes that run etcd are protected by BCM with the `datanode` option ([here](/admin/provisioning#5444-full-install-confirmation-using-datanode-setting)). For etcd nodes, the option is added during the `cm-kubernetes-setup` installation. As in the case for the registries, the datanode option is set in order to help prevent the administrator from wiping out the existing state of etcd nodes. Wiping out the state of etcd nodes means that the Kubernetes cluster becomes incoherent and that it cannot be restored to where it was just before the etcd nodes were wiped. * `cm-containerd`: This BCM package has the containerd runtime. * `cm-nvidia-container-toolkit`: This BCM package has the NVIDIA container toolkit (includes NVIDIA container runtime). * `cm-kube-diagnose`: This BCM package has Helper tools to diagnose malfunctioning Kubernetes clusters. Kubernetes .rpm and .deb packages themselves are installed from the Kubernetes community-owned software repositories ([https://kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/](https://kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/)). These repositories host Kubernetes versions starting from v1.24.0 at the time of writing of this paragraph (March 2024). ### 4.2.1 Kubernetes Networking Early on during the wizard session (figure [4.2](#fig-k8s-networking)), a name for the cluster is requested. The wizard pre-fills it with `default`, but this should not be confused with the Kubernetes `default` namespace. Here, the name is used instead, inside BCM, to identify the cluster, configuration files, and other resources such as module files. ![Kubernetes setup TUI session: networking selection screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/0b384c522b08af632c0554be391a2178ef5a7a216590614d012989c6015f63fb/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/cm-kubernetes-setup-networking.png) *Figure 4.2: Kubernetes setup TUI session: networking selection screen* This screen also allows the following important choices: * `Kubernetes external FQDN`: This is the FQDN that is placed as one of the entries in the public-facing certificates generated for this Kubernetes cluster. Configuring the public-facing certificate of the `bcm-gateway` `Gateway` (served by Kgateway) is discussed further in section [4.17.7](#4177-configuring-the-gateway-tls-certificate). * `Service network base address` and `Service network netmask bits`: These define the CIDR for the service network. The wizard pre-fills the fields. It also tries to avoid pre-filling them with overlapping network ranges, by taking any existing network known to BCM into account. * `Pod network base address` and `Pod network netmask bits`: These define the CIDR for the pod network. The wizard pre-fills these. It also tries to avoid pre-filling them with overlapping network ranges, by taking any existing network known to BCM into account. By default, entire /24 network ranges are assigned to individual Kubernetes nodes from the pod CIDR. The packages are installed automatically from the repository when the administrator runs `cm-kubernetes-setup` from the command line. The log file produced by the setup can be found in `/var/log/cm-kubernetes-setup.log`. ### 4.2.2 Kubernetes Core Add-ons During setup, some critical add-on components such as the Networking Component for Kubernetes are automatically deployed. Most components are in the `kube-system` namespace, but others have their own namespaces. In BCM some add-ons are treated as Kubernetes applications (Chapter [5](/containerization/kubernetesapps)), and belong to the default app group, `system`. In the future Helm is expected to manage most, if not all, components instead of BCM Kubernetes applications. The user is prompted for a Networking Component, Kubernetes operators (managed using Helm), Kubernetes add-ons (managed using Kubernetes applications), and more. They all result in either Helm charts being deployed, or in Kubernetes applications being managed from BCM. A `cmsh` treeview illustrating the hierarchy to access these applications is: ```bash [cmsh] |-- ... | |-- kubernetes[default] | `-- appgroups[system] | `-- applications |-- ... ``` Helm charts can be found using the `helm` command. For example using `helm list -A` on a cluster (after loading the correct module file). In the past the DNS component was also provided as an add-on. However since BCM version 10.0, `kubeadm` is used to bundle CoreDNS as part of the default control plane. Kubernetes apps are discussed further in Chapter [5](/containerization/kubernetesapps). Kubernetes operators are discussed further in Chapter [6](/containerization/kubernetesoperators). #### 4.2.2.1 Kubeadm Components Components deployed as part of the default Kubernetes control plane are: * Kubernetes API server * Kubernetes Scheduler * Kubernetes Controller Manager * Kubernetes Proxy * Core DNS These are deployed through the “static pod” mechanism ([https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/](https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/)). Kubernetes control plane components are described at: [https://kubernetes.io/docs/concepts/overview/components/](https://kubernetes.io/docs/concepts/overview/components/). **CoreDNS:** CoreDNS is the DNS server add-on for internal service discovery. It reads the IP addresses of services and pods from etcd, and resolves domain names for them. If a domain name is not found because the domain is external to the Kubernetes cluster, then CoreDNS forwards the request to the main DNS server. BCM uses CoreDNS version 1.13.1 with Kubernetes version 1.35. #### 4.2.2.2 Networking Component An important component managed through BCM’s Kubernetes Apps is the Networking component. This adheres to the Container Networking Interface, or CNI. At the time of writing the available CNIs in the setup wizard are either Calico or Flannel. However BCM does support other CNIs, for example Cilium, as described in the knowledge base article at [https://enterprise-support.nvidia.com/s/article/installing-cilium-networking-for-kubernetes](https://enterprise-support.nvidia.com/s/article/installing-cilium-networking-for-kubernetes). **Flannel:** Flannel is a simple and easy way to configure a layer 3 network fabric designed for Kubernetes. **Calico:** Calico is a popular open source networking and network security solution for Kubernetes. It provides network connectivity between workloads and security policies features. It can be configured to use eBPF. Further details on Calico can be found at [https://docs.projectcalico.org/](https://docs.projectcalico.org/). If the Kubernetes cluster is composed of more than 50 nodes, then the Calico component Typha is also automatically deployed for better scalability. The number of Typha replicas is calculated by allocating one Typha replica per 150 nodes, with a minimum of 3 (above 50 nodes) and a maximum of 20. If an initial deployment of the Kubernetes cluster has fewer than 50 nodes, but nodes are then added to the Kubernetes cluster so that the 50 node threshold is exceeded, then Typha is not automatically enabled. In this case, Typha can be enabled manually using `cmsh` as follows: **Example** ```bash [basecm11->kubernetes[default]->appgroups[system]->applications[calico]]% environment [basecm11->kubernetes[default]->appgroups[system]->applications[calico]->environment]% list Name (key) Value Nodes environment ---------------------- ---------------- ------------------ calico_typha_replicas 0 no calico_typha_service none no cidr 10.141.0.0/16 no [basecm11->kubernetes...[calico]->environment]% set calico_typha_service value calico-typha [basecm11->kubernetes*...[calico*]->environment*]% set calico_typha_replicas value 3 [basecm11->kubernetes*[default*]->appgroups*[system*]->applications*[calico*]->environment*]% commit [basecm11->kubernetes[default]->appgroups[system]->applications[calico]->environment]% ``` **Cilium:** Cilium ([https://cilium.io/](https://cilium.io/)) is a networking, observability, and security solution with an eBPF-based dataplane. ### 4.2.3 Kubernetes Optional Components The following components are installed by default unless otherwise noted. However, the user can choose to skip some or all of them during the setup. #### 4.2.3.1 Kgateway Kgateway is the implementation that BCM uses to fulfill the Kubernetes Gateway API specification. It is discussed further in section [4.17](#417-kubernetes-gateway). #### 4.2.3.2 MetalLB MetalLB is a bare-metal load balancer that allocates external IP addresses to any `LoadBalancer`-type `Service` on the cluster. BCM deploys it alongside Kgateway to back the Kubernetes `Gateway`s (section [4.17.4.1](#41741-gateway-ip-address-assignment-and-routability)), but it can also be used outside the Gateway scope. #### 4.2.3.3 Kubernetes Metrics Server The Kubernetes Metrics Server is an add-on that is a replacement for Heapster. It aggregates metrics, and provides container monitoring and performance analysis. It exposes metrics using an API. #### 4.2.3.4 Kubernetes State Metrics `kube-state-metrics` is an add-on agent to generate and expose cluster-level Kubernetes metrics ([section G.1.14 of the *Administrator Manual*](/admin/metricshealthchecksactions#g114-kubernetes-metrics)) for objects. The project is not focused on the health of the individual Kubernetes components, but rather on the health of the various objects inside, such as deployments, nodes and pods. ### 4.2.4 Kubernetes Deprecated And Retired Components The following components were part of BCM’s Kubernetes installation in earlier releases. They are no longer installed by default, and are documented here for reference and to point users to their replacements. #### 4.2.4.1 Kubernetes Dashboard (retired) The Kubernetes Dashboard provided a web UI for cluster administration and metrics visualization in BCM versions before 11.32.0. It was retired upstream and has been removed in BCM version 11.32.0. #### 4.2.4.2 Ingress NGINX Controller (deprecated) The Ingress NGINX Controller (`ingress-nginx`) provided ingress routing and web load balancing in BCM versions before 11.33.0 using standard Kubernetes `Ingress` resources, which were consumed together with `ingress-nginx`-specific annotations, such as `nginx.ingress.kubernetes.io/*`. The Ingress NGINX Controller was deployed as a `ingress-nginx` Helm release. In BCM, it was exposed externally using the legacy Ingress proxy on the non-standard ports `30080` (HTTP) and `30443` (HTTPS), and the TLS certificate for externally-exposed services was served on `30443`. It was deprecated upstream and in BCM starting with version 11.32.0, and is disabled by default from BCM 11.33.0, where it is replaced by Kgateway (section [4.17](#417-kubernetes-gateway)). For BCM clusters that were originally deployed with the Ingress NGINX Controller, the procedure to migrate to the Gateway-based stack is described in section [4.17.8](#4178-migrating-an-existing-kubernetes-cluster-from-ingress-to-gateway). The wizard marks the Ingress NGINX Controller as deprecated on the operator selection screen (section [4.2.8](#428-kubernetes-setup-from-a-tui-session)). It should not be selected for a new deployment without a good reason to do so. The Ingress NGINX Controller should not be confused with the Ingress proxy. The controller runs inside Kubernetes. The Ingress proxy is an NGINX instance that runs outside Kubernetes, on the head node, and forwards external traffic into the cluster (section [4.9.3](#493-the-kubernetesingressserverproxy-role)). The two are configured independently, so the `Ingress Proxy Enable` parameter being set says nothing about whether the Ingress NGINX Controller is in use. ### 4.2.5 Helm Kubernetes Package Manager Helm is an add-on that manages *charts*, which are packages of pre-configured Kubernetes resources. The Helm component is installed and properly configured with BCM’s Kubernetes installation by default. It is initialized and ready for use by every Kubernetes user when the Kubernetes module is loaded. BCM uses Helm version 4.2.3. When the Helm binary is installed during the Kubernetes setup, an offer is made to deploy charts during the setup process. Most operators are documented in Chapter [6](/containerization/kubernetesoperators). Other parts of the wizard also determine which Helm charts are deployed as part of the setup for the following: * Kyverno and Kyverno Policies (section [4.10.1](#4101-kyverno)) * BCM Permissions Manager (section [4.15](#415-kubernetes-permission-manager)) * BCM Local Path Provisioner (section [4.20](#420-local-path-storage-class)) **Example** ```bash root@basecm11:~# helm list -A NAME NAMESPACE CHART APP VERSION agentgateway agentgateway-system ... agentgateway-v1.3.1 v1.3.1 agentgateway-crds agentgateway-system ... agentgateway-crds-v1.3.1 v1.3.1 calico tigera-operator ... tigera-operator-v3.32.1 v3.32.1 calico-crds tigera-operator ... crd.projectcalico.org.v1-v3.32.1 v3.32.1 cm-jupyter-kernel-operator cm ... cm-jupyter-kernel-operator-0.3.12 0.3.12 cm-kubernetes-mpi-operator cm ... mpi-operator-0.8.0 0.8.0 gpu-operator gpu-operator ... gpu-operator-v26.3.3 v26.3.3 grafana-operator grafana ... grafana-operator-5.24.0 v5.24.0 k8s-nim-operator nim-operator ... k8s-nim-operator-3.1.1 3.1.1 kgateway kgateway-system ... kgateway-2.3.6 2.3.6 kgateway-crds kgateway-system ... kgateway-crds-2.3.6 2.3.6 knative-operator knative-operator ... knative-operator-1.22.3 kube-prometheus-stack prometheus ... kube-prometheus-stack-87.18.0 v0.92.1 kube-state-metrics kube-system ... kube-state-metrics-8.0.0 2.19.1 kyverno kyverno ... kyverno-3.8.2 v1.18.2 kyverno-policies kyverno ... kyverno-policies-3.8.2 v1.18.2 local-path-provisioner cm ... local-path-provisioner-0.0.36 v0.0.36 lws lws-system ... lws-v0.9.0 v0.9.0 metallb metallb-system ... metallb-0.16.1 v0.16.1 metrics-server kube-system ... metrics-server-3.13.1 0.8.1 network-operator network-operator ... network-operator-26.1.1 v26.1.1 permissions-manager cm ... cm-kubernetes-permissions-manager-0.7.0 0.7.0 postgres-operator postgres-operator ... postgres-operator-1.15.1 1.15.1 prometheus-adapter prometheus ... prometheus-adapter-5.3.0 v0.12.0 spark-operator spark-operator ... spark-operator-2.5.1 2.5.1 ``` ### 4.2.6 Note On Run:ai Operator Dependencies #### 4.2.6.1 Selecting The Run:ai Operator Dependencies When the `cm-kubernetes-setup` wizard is run, with the intent of running Run:ai as well, the operator selection screen must include the dependencies that Run:ai requires. For Run:ai 2.25, these are: * NVIDIA GPU Operator * Grafana Operator * Kgateway * Knative Operator * Kubeflow Training Operator * Kubernetes Metrics Server * Kubernetes MPI Operator * Kubernetes State Metrics * LeaderWorkerSet Operator * MetalLB * Network Operator (technically optional, but recommended) * NIM Operator (optional) * Prometheus Adapter * Prometheus Operator Stack The Run:ai operator and its prerequisites are described in more detail in section [6.11](/containerization/kubernetesoperators#611-the-runai-operator). ### 4.2.7 Kubernetes Setup From The Command Line The `cm-kubernetes-setup` command line utility has the following usage synopsis: ```bash root@basecm11:~# cm-kubernetes-setup --help usage: Kubernetes Setup cm-kubernetes-setup [-c ] [--list-versions] [--list-operators-versions] [--cluster CLUSTER_NAME] [--skip-image-update] [--skip-kube-version-check] [--skip-dns-configuration-check] [--skip-ping-check] [--skip-mtu-check] [--skip-package-manager-update-check] [--skip-install-repos] [--adv-configuration-bcm-nvidia-packages] [--add-user USERNAME_ADD] [--list-users] [--get-user GET_USER] [--modify-user USERNAME_MODIFY] [--remove-user USERNAME_REMOVE] [--namespace NAMESPACE] [--add-to-namespace] [--remove-from-namespace] [--role {edit,admin,view,cluster-admin}] [--runas-uid RUNAS_UID] [--runas-gids RUNAS_GIDS] [--user-paths USER_PATHS] [--allow-all-uids] [--operators OPERATORS] [--backup-permissions FILE] [--restore-permissions FILE] [--list-operators] [--add-operators | --remove-operators] [--patch-kube-prometheus-stack] [--configure-gateway-proxy] [--remove] [--yes-i-really-mean-it] [--pull] [--images IMAGES] [--nodes NODES] [--node-selector NODE_SELECTOR] [--pull-registry-server PULL_REGISTRY_SERVER] [--pull-registry-username PULL_REGISTRY_USERNAME] [--pull-registry-email PULL_REGISTRY_EMAIL] [--pull-registry-password PULL_REGISTRY_PASSWORD] [--docker-io-username DOCKER_IO_USERNAME] [--docker-io-password DOCKER_IO_PASSWORD] [--kubeadm-image-repository KUBEADM_IMAGE_REPOSITORY] [--github-token GITHUB_TOKEN] [--containerd-sandbox-image CONTAINERD_SANDBOX_IMAGE] [--enable-ovn] [--no-cni] [--latest-versions] [--skip-ovn-prerequisites-checks] [--skip-netq-prerequisites-checks] [--provision-node PROVISION_NODE] [--provision-software-image PROVISION_SOFTWARE_IMAGES] [--airgap-registry AIRGAP_REGISTRY] [--airgap-registry-as-helm | --airgap-helm-repo AIRGAP_HELM_REPO] [--airgap-install-system-packages | --airgap-push- container-images | --airgap-push-helm-charts] [--ca-file CA_FILE] [--airgap-registry-username AIRGAP_REGISTRY_USERNAME] [--airgap-registry-password AIRGAP_REGISTRY_PASSWORD] [-v] [--store-name-aliases] [--no-distro-checks] [--json] [--output-remote-execution-runner] [--on-error-action {debug,remotedebug,undo,abort}] [--skip-reboot] [--skip-packages] [--min-reboot-timeout ] [--allow-running-from-secondary] [--dev] [-h] options: --cluster CLUSTER_NAME Name of the referred Kubernetes cluster -h, --help Print this screen common: Common arguments -c Load runtime configuration for plugins from a YAML config file Supported versions information: --list-versions Show supported Kubernetes versions --list-operators-versions List available for selection Kubernetes operators versions installing Kubernetes clusters: Flags for installing or managing Kubernetes clusters --skip-image-update Skip the image update steps. --skip-kube-version-check Skip Kubernetes version check before install (setup will not continue if it finds unsupported version). --skip-dns-configuration-check Skip DNS configuration check before install (setup will not continue if it finds DNS malfunctioning). --skip-ping-check Skip Ping check before install (setup will not continue if there is no inter-node connectivity across the Kubernetes cluster). --skip-mtu-check Skip MTU check before install (inconsistent MTU sizes on Kubernetes nodes can result in packet loss). --skip-package-manager-update-check Skip Package Manager update/refresh check before install (setup will not continue if 'apt update' fails on Ubuntu for example). --skip-install-repos Skip installation of kubernetes and helm package repos. (assumes they have been manually configured, e.g., in the case of air-gapped). --adv-configuration-bcm-nvidia-packages Allow the option to override the selection of BCM NVIDIA packages to be installed when setting up the NVIDIA GPU operator user management: Flags for adding a new user in a Kubernetes cluster --add-user USERNAME_ADD Create a new user in a Kubernetes cluster --list-users Get information about configured Kubernetes users --get-user GET_USER Get information about configured Kubernetes users --modify-user USERNAME_MODIFY Modify user in a Kubernetes cluster --remove-user USERNAME_REMOVE Remove existing user from a Kubernetes cluster --namespace NAMESPACE Specify namespace for user (--get-user, --modify-user) role binding --add-to-namespace Indicate if permissions to manage namespace needs to be granted for a given user (--modify-user) --remove-from-namespace Indicate if permissions to manage namespace needs to be revoked for a given user (--modify-user) --role {edit,admin,view,cluster-admin} Specify role for the new (--add-user) and existing (--modify-user) role binding (Default: edit) For 'cluster-admin' namespace flag is ignored --runas-uid RUNAS_UID UID is allowed to be used in unprivileged pods (--add-user, --modify-user) --runas-gids RUNAS_GIDS Comma-separated list of GIDs allowed to be used in unprivileged pods (--add-user, --modify-user) --user-paths USER_PATHS Comma-separated list of paths user is able to mount in pods (--add- user, --modify-user) --allow-all-uids Allow user to run processes in pods as any user (--add-user, --modify-user) hostPath volumes will be disabled for such pods --operators OPERATORS Comma-separated list of operators user has access to (--add-user, --modify-user) backup or restore Permission Manager user configurations: Flag for managing permission manager user configuration --backup-permissions FILE Save permissions to file --restore-permissions FILE Restore permissions from file. Workload which is already run by users in their namespaces will be affected list available operators: Flag to list available Kubernetes operators --list-operators List available Kubernetes operators modify operators: Flags for config-driven Day 2 operator management on an existing cluster --add-operators Add operators to an existing Kubernetes cluster. Requires a config file (-c); only operators it explicitly marks install_helm: true are added --remove-operators Remove operators from an existing Kubernetes cluster. Requires a config file (-c); only operators it explicitly marks install_helm: false are removed update kubernetes addons: Flags for updating Kubernetes addons --patch-kube-prometheus-stack Patch to improve kube-prometheus-stack helm chart (scrape config) gateway proxy: Flags for configuring the Gateway proxy --configure-gateway-proxy Configure the Gateway proxy to forward traffic from the head node(s) to the Kubernetes Gateways removing Kubernetes clusters: Flags for removing a Kubernetes cluster --remove Remove a Kubernetes cluster --yes-i-really-mean-it Required for additional safety pulling images to the nodes: Flags for pulling images to the nodes --pull Pull images to the nodes --images IMAGES Comma-separated list of images to pull (--pull) --nodes NODES Comma-separated list of nodes to pull images to (--pull) --node-selector NODE_SELECTOR Selector (label query) to filter on, supports '=', '==', and '!='.(e.g. key1=value1,key2=value2) (--pull) --pull-registry-server PULL_REGISTRY_SERVER Registry server to authenticate --pull-registry-username PULL_REGISTRY_USERNAME Registry username --pull-registry-email PULL_REGISTRY_EMAIL Email to authenticate --pull-registry-password PULL_REGISTRY_PASSWORD Registry password docker.io: Flags for authenticating to docker hub --docker-io-username DOCKER_IO_USERNAME Username --docker-io-password DOCKER_IO_PASSWORD Password kubeadm: Flags for configuring kubeadm --kubeadm-image-repository KUBEADM_IMAGE_REPOSITORY Custom imageRepository GitHub: Flags for configuring GitHub API access --github-token GITHUB_TOKEN GitHub token for authenticated release metadata requests. Stored base64-encoded in Kubernetes config. containerd: Flags for configuring containerd --containerd-sandbox-image CONTAINERD_SANDBOX_IMAGE Custom sandbox image (e.g. my-registry:5000/pause:3.8) ovn: Flags for configuring OVN CNI --enable-ovn Allow selection of OVN CNI during Kubernetes setup (currently not shown by default). --no-cni Instead of deploying a CNI, the setup will exit early after having setup Kubernetes without a CNI, the user is expected to configure a CNI, and other components. --latest-versions Dynamically determine and use latest versions of OVN Kubernetes CNI and Dashboard charts. --skip-ovn-prerequisites-checks Provide this flag to skip precondition checks such as checking for Network Layouts, Linux distro requirements for OVN CNI. netq: Flags for configuring NetQ --skip-netq-prerequisites-checks Provide this flag to skip precondition checks such as checking for disk size, Linux distro, etc. provisioning: Flags for provisioning Kubernetes packages and configuration --provision-node PROVISION_NODE Provision a node with Kube packages and prepare other configuration --provision-software-image PROVISION_SOFTWARE_IMAGES Provision a software image with Kube packages (excludes NVIDIA GPU (driver, toolkit, ...) packages) and prepares other configuration. Please ensure that software images are prepared before kubelet role assignment. If manual provisioning of nodes is required, an imageupdate is insufficient, either use --provision-node, or give the node an additional reboot to make it join the Kubernetes cluster. Air-gapped deployment: --airgap-registry AIRGAP_REGISTRY AirGap container registry address --airgap-registry-as-helm Use AirGap container registry as helm repository --airgap-helm-repo AIRGAP_HELM_REPO Helm repository address Air-gapped deployment preparation: --airgap-install-system-packages Install pre-downloaded system packages --airgap-push-container-images Push pre-downloaded container images to the AirGap registry --airgap-push-helm-charts Push pre-downloaded helm charts to the AirGap registry --ca-file CA_FILE path to CA file for AirGap registry TLS verification --airgap-registry-username AIRGAP_REGISTRY_USERNAME AirGap registry username --airgap-registry-password AIRGAP_REGISTRY_PASSWORD AirGap registry password advanced: Various *advanced* configuration options flags. -v, --verbose Verbose output --store-name-aliases Store hostname aliases for head nodes (active and passive) and default category --no-distro-checks Disable distribution checks based on ds.json --json Use json formatting for log lines printed to stdout --output-remote-execution-runner Format output for CMDaemon --on-error-action {debug,remotedebug,undo,abort} Upon encountering a critical error, instead of asking the user for choice, setup will do selected action. --skip-reboot Skip the any stages which reboot compute nodes. --skip-packages Skip the any stages which install packages. Requires packages to be already installed. --min-reboot-timeout How long to wait for nodes to finish reboot (default and minimum allowed: 300 seconds). --allow-running-from-secondary Allow to run the wizard from the secondary when it is the active head node. --dev Enables additional command line arguments ``` The preceding output does not list every argument that the utility accepts. Some valid arguments are not shown by `--help`. For example, `--skip-headnode-check` is accepted by the utility but is not listed. It is a development argument, and it is not part of any documented BCM procedure. An argument that is absent from the `--help` output may be valid, but is typically not intended for use by the end user. The `cm-kubernetes-setup` utility should be executed on the console. #### 4.2.7.1 Dealing With A Pre-existing Docker Installation Docker (Chapter [2](/containerization/dockerengine)) is no longer a requirement for Kubernetes configured by BCM. This is because Kubernetes can directly interface with containerd through its Container Runtime Interface (CRI). Docker can co-exist with Kubernetes, and can be set up as discussed in section [2.1](/containerization/dockerengine#21-docker-setup). ### 4.2.8 Kubernetes Setup From A TUI Session When the Kubernetes installation is carried out using `cm-kubernetes-setup` without any options, a TUI wizard starts up. The administrator can answer several questions in the wizard. Questions that are asked include questions about which of the node categories or which of the individual nodes should be configured to run the Kubernetes services. There are also questions about the service and pod networks parameters, the port numbers that will be configured for the daemons, which operators to install, and so on. After the wizard has been completed, a configuration file can be saved that can be used to set up Kubernetes, by default `cm-kubernetes-setup.conf`. The configuration file can be deployed immediately from the wizard, or it can be deployed later by specifying it as an option to `cm-kubernetes-setup`, in the form `-c` \<*`file`*>. The layout of the configuration file can change between BCM releases. For example, the Flannel settings are stored under `modules.kubernetes.flannel` in BCM 11.33, and under `modules.kubernetes.operators.flannel` in BCM 11.34. A configuration file that was saved by one release should therefore not be assumed to work with another. If no deployment has been carried out earlier, then the main operations screen of the wizard shows just two options, `Deploy` and `Exit`. If deployment has already been carried out, then the further options that are available are also displayed (figure [4.3](#fig-kube-urwid-wizard-operations)): **Example** ```bash [root@basecm11 ~]# cm-kubernetes-setup TUI session starts up: ``` ![Kubernetes setup TUI session: main operations screen after a deployment](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/911835e99b7128aa0fdfea5ebb16618686fdee7d3605fc74353667d091964520/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/kube-urwid-wizard-operations.png) *Figure 4.3: Kubernetes setup TUI session: main operations screen after a deployment* One of the further options shown after a deployment is a `Modify operators` item, for adding operators to, or removing operators from, an existing cluster without re-running the full setup. This is described in section [6.15](/containerization/kubernetesoperators#615-adding-and-removing-operators-on-an-existing-cluster). The deployment using CLI or using TUI assigns the appropriate roles, and adds the new Kubernetes cluster. The deployment adds health checks to the monitoring configuration, and it generates certificates for the Kubernetes daemons. **Operating the wizard:** The wizard is a terminal UI, where * the arrow keys (up, down, left, right) move between fields * the `` key moves between widgets such as the field area and the buttons along the bottom * the `` bar selects an option on screens that present a list of choices * the `` key, if pressed on a screen with a list of choices, advances to the next screen without changing the selection, which can take some getting used to * the `< Help >` button on the screen, if present, describes the setting in more detail than the screen itself. The screens that follow are the ones presented when the wizard is run with the default operator selection. That is the path through the wizard, not a recommendation to accept every pre-filled value: some screens, such as the DockerHub registry mirror, need an answer that suits the site. Screens that appear only when a particular operator has been selected are not described here. Those are covered with the operator that they belong to, in chapter [6](/containerization/kubernetesoperators). **Kubernetes operations:** If no deployment has been carried out earlier, then this screen offers only `Deploy` and `Exit` as options (figure [4.4](#fig-kube-urwid-wizard-operations-initial)). The fuller set of options shown in figure [4.3](#fig-kube-urwid-wizard-operations) appears only after a cluster exists. ![Kubernetes setup TUI session: main operations screen before any deployment](/base-command-manager/product-manuals/_fern-img/d1e2437f1ecca9007517d0c11306998d28e4f43ae2a55f055ac5e2df027ec63c.webp) *Figure 4.4: Kubernetes setup TUI session: main operations screen before any deployment* **Choosing a Kubernetes version:** The wizard lists the Kubernetes minor versions available in the installed packages (figure [4.1](#fig-k8s-version-selection)). Versions certified for NVIDIA AI Enterprise are marked with an asterisk, and the legend on the screen explains the marking. **Choosing a Kubernetes patch level:** The version chosen on the preceding screen is a minor version. This screen selects the patch level within it (figure [4.5](#fig-kube-urwid-wizard-patch-level)). Two entries are offered: `latest`, and the specific patch release that the installed packages provide. The specific release is pre-selected. Choosing `latest` means that the patch level is resolved when the deployment runs, rather than being fixed now. For an air-gapped cluster the specific release should be chosen, because only the patch level that was mirrored into the tarball is available (section [B.2](/containerization/kubernetesairgapinstallation#b2-head-node-preparation-steps)). ![Kubernetes setup TUI session: choosing the Kubernetes patch level](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/ba28fb5f55256392696d9604584c13b42c0cd03ccdf5566eb78d262ef8a2ef10/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/kube-urwid-wizard-patch-level.png) *Figure 4.5: Kubernetes setup TUI session: choosing the Kubernetes patch level* **DockerHub registry mirror:** A registry mirror can be specified so that images are pulled from it rather than from DockerHub (figure [4.6](#fig-kube-urwid-wizard-dockerhub-mirror)). The field is pre-filled with an NVIDIA-internal host that is not reachable from customer sites. The value should therefore be replaced with the address of a mirror that the cluster can reach, or it should be cleared if no mirror is used. For an air-gapped cluster, the mirror is the local container registry that is set up as part of the air-gapped procedure (section [B.2](/containerization/kubernetesairgapinstallation#b2-head-node-preparation-steps)). ![Kubernetes setup TUI session: DockerHub registry mirror](/base-command-manager/product-manuals/_fern-img/c1f6cd1ff6ad9f6a8fa118f21a84e35a0c20d971557731ea51b649398c37b78a.webp) *Figure 4.6: Kubernetes setup TUI session: DockerHub registry mirror* **Basic cluster values:** The cluster name, domain, external FQDN, and the service and pod network CIDRs are set here (figure [4.2](#fig-k8s-networking)). The wizard pre-fills the network fields and tries to avoid ranges that overlap networks already known to BCM. These fields are described in detail in section [4.2.1](#421-kubernetes-networking). **Exposing the Kubernetes API server:** The Kubernetes API server can be exposed to the external network (figure [4.7](#fig-kube-urwid-wizard-expose-api-server)). **The wizard offers `yes` as the pre-selected answer, so accepting the default exposes the API server.** Leaving it unexposed means that the port is not opened to the external network, which is the safer choice. The API server is still reached through the NGINX proxy on the head node, so `kubectl` and Helm continue to work there. It should be exposed only if clients outside the cluster need to reach the API server directly. ![Kubernetes setup TUI session: exposing the Kubernetes API server](/base-command-manager/product-manuals/_fern-img/8ef9ecdeaf451772a267fcc14a58ed65d21a99d7844ccffc725408f7294e97ba.webp) *Figure 4.7: Kubernetes setup TUI session: exposing the Kubernetes API server* **Selecting the internal network:** This is the network that Kubernetes nodes use to reach each other (figure [4.8](#fig-kube-urwid-wizard-internal-network)). One network must be selected, using ``. On a cluster configured for high availability a third option, `failovernet`, is offered alongside `externalnet` and `internalnet`. If the nodes do not share an IP address on the selected network, then their internal networks are configured as fallback networks. ![Kubernetes setup TUI session: selecting the internal network](/base-command-manager/product-manuals/_fern-img/1a72084f92f11f2f4559c64a243de82f2f28941b7298ccf6f09a0102baf6b479.webp) *Figure 4.8: Kubernetes setup TUI session: selecting the internal network* **Node assignment:** The control-plane, worker, and etcd nodes are assigned on four consecutive screens (figures [4.9](#fig-kube-urwid-wizard-master-nodes)–[4.12](#fig-kube-urwid-wizard-etcd-nodes)). The head node is pre-selected as a control-plane node and as an etcd node. Worker nodes are assigned on two of those screens. The first selects whole node categories, and the second selects individual nodes. Selecting a category is sufficient on its own. If the intended worker nodes are all members of a category that has been selected, then the screen for individual nodes can be left with nothing selected, and it lists only the nodes that the category selection did not already cover. An odd number of etcd nodes should be selected, as the screen itself states. An even-numbered cluster still establishes a quorum, but it tolerates no more failures than the odd number below it: a three-node and a four-node cluster both keep working with one node down, so the fourth node adds cost without adding fault tolerance. ![Kubernetes setup TUI session: assigning control-plane nodes](/base-command-manager/product-manuals/_fern-img/32a8c08f081227049e5d3c0a0ecf334d493627ff9e0ecb06ed9451dbbad15fb7.webp) *Figure 4.9: Kubernetes setup TUI session: assigning control-plane nodes* ![Kubernetes setup TUI session: assigning worker nodes by category](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/576ec81e5317fbbc144ea90d90e6a8dceaaefd19fa7a43a598da23bcf16418af/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/kube-urwid-wizard-worker-categories.png) *Figure 4.10: Kubernetes setup TUI session: assigning worker nodes by category* ![Kubernetes setup TUI session: assigning individual worker nodes](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/6963b6362a28813eb10908dcbcb75283ce7e772581975b7045634b1c92f295be/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/kube-urwid-wizard-worker-nodes.png) *Figure 4.11: Kubernetes setup TUI session: assigning individual worker nodes* ![Kubernetes setup TUI session: etcd node assignment](/base-command-manager/product-manuals/_fern-img/7ba2541251459379807bf5c32a11e25291278e8f01c1480740b59db5650df175.webp) *Figure 4.12: Kubernetes setup TUI session: etcd node assignment* **Main Kubernetes component values:** The port numbers configured for the daemons have default assignments, but can be re-assigned as needed, and the `etcd` spool directory can be set (figure [4.13](#fig-kube-urwid-wizard-component-values)). The defaults are port `10443` for the API server proxy, and `/var/lib/etcd` for the `etcd` spool directory. ![Kubernetes setup TUI session: main Kubernetes component values](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/0735cf31b35d08ed2c5cd43a39c9d89ca366b2926fb88776d2a71427f92eb825/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/kube-urwid-wizard-component-values.png) *Figure 4.13: Kubernetes setup TUI session: main Kubernetes component values* **Kubernetes network plug-in:** The CNI network plug-in is chosen here (figure [4.14](#fig-kube-urwid-wizard-network-plugin-cni)). Calico, installed by way of the Tigera Operator, is pre-selected and is the recommended choice. Flannel is the alternative. These are the only two that the wizard offers. Both are described in section [4.2.1](#421-kubernetes-networking). ![Kubernetes setup TUI session: selecting the CNI network plug-in](/base-command-manager/product-manuals/_fern-img/990dbea4ea10ca3ac79c8ea9ee621a6aba450c9805ddf73228a07e7a8352c33b.webp) *Figure 4.14: Kubernetes setup TUI session: selecting the CNI network plug-in* **Kyverno policy engine:** The Kyverno policy engine is installed by default (figure [4.27](#fig-kube-urwid-wizard-kyverno-install)). Kyverno is what BCM uses to enforce Pod Security Standards. PodSecurityPolicy provided that enforcement in older releases, and was removed in Kubernetes v1.25 (section [4.10.1](#4101-kyverno)). If the policy engine is not installed, then Kubernetes Permissions Manager (section [4.15](#415-kubernetes-permission-manager)) still creates the namespace and the role bindings for each user, so users are separated from each other by RBAC. What is not created are the policies that restrict what a user may do inside their own namespace. On Kubernetes v1.25 and later there is no fallback for this, because the PodSecurityPolicy that used to provide it no longer exists. A user in that situation can run privileged pods, and can mount arbitrary host paths. Those pods are created in the user’s own namespace, but neither capability is confined to it: a privileged pod and a host path mount reach the node filesystem and the container runtime. The exposure is therefore at node level, and through the node it extends to other workloads and to cluster data. The RBAC separation described above does not prevent this. On a cluster with one administrator and no other users this can be acceptable. On a cluster that is shared between users it usually is not. **Kyverno policies:** The Kyverno policies are installed by default (figure [4.29](#fig-kube-urwid-wizard-kyverno-policies)). The policies are what implement the Pod Security Standards (section [4.14.2](#4142-kyverno-policies)). The engine on its own enforces nothing. On a cluster configured for high availability, a further screen asks whether Kyverno itself is to be set up for high availability (figure [4.28](#fig-kube-urwid-wizard-kyverno-ha)). That screen is not presented on a cluster with a single head node. **Namespaces excluded from the Kyverno policies:** Namespaces can be excluded from each of the cluster-wide policies (figure [4.15](#fig-kube-urwid-wizard-kyverno-namespaces)). Pods running in an excluded namespace are not subject to that policy. Each of the five policies takes its own comma-separated list, and each is pre-filled with the `default` namespace. Further exclusions are added automatically during installation, based on the features that have been configured and the operators that have been selected. A namespace named on this screen is not created by the wizard, as the screen itself states. ![Kubernetes setup TUI session: namespaces excluded from the Kyverno policies](/base-command-manager/product-manuals/_fern-img/d07a8ad26fc3910ce36eea13f254df41296d0ac7c476ac96ec8af18bda6ae4d1.webp) *Figure 4.15: Kubernetes setup TUI session: namespaces excluded from the Kyverno policies* **Selecting the operators:** Operator packages are application managers, and are described further in chapter [6](/containerization/kubernetesoperators). The operators that can be selected are listed below, in the order in which the wizard presents them: * NVIDIA GPU Operator * AgentGateway * AgentGateway CRDs * Argo CD * Grafana Alloy * Grafana Loki * Grafana Operator * Ingress NGINX Controller (deprecated, section [4.2.4.2](#4242-ingress-nginx-controller-deprecated)) * Jupyter Kernel Operator (`cm-jupyter-kernel-operator`) * Kgateway * Knative Operator * KubeFlow Training Operator * Kubernetes Metrics Server * Kubernetes MPI Operator (`cm-kubernetes-mpi-operator`) * Kubernetes State Metrics * LeaderWorkerSet operator * MetalLB * Network Operator * NIM Operator * PostgreSQL Operator (`postgresql-operator`) * Prometheus Adapter * Prometheus Operator Stack * Spark Operator (`spark-operator`) Four of these are selected by default: Kgateway, Kubernetes Metrics Server, Kubernetes State Metrics, and MetalLB (figure [4.16](#fig-kube-urwid-wizard-operators-select)). The rest are installed only if they are selected here. The NVIDIA GPU Operator is not selected by default. It manages the Kubernetes device plugin that exposes NVIDIA GPUs as schedulable resources (section [6.7](/containerization/kubernetesoperators#67-the-nvidia-gpu-operator)), and the wizard provides no other way to install that plugin. A cluster with NVIDIA GPUs therefore cannot run GPU workloads until the operator is installed. The wizard does not check whether the cluster has GPUs, and does not warn if the operator is left unselected. It does not have to be installed here: it can also be added to a running cluster (section [6.7.2](/containerization/kubernetesoperators#672-installing-the-nvidia-gpu-operator-on-an-existing-kubernetes-cluster)). The CNI network plug-in is not part of this list. It is chosen on the earlier screen shown in figure [4.14](#fig-kube-urwid-wizard-network-plugin-cni). Each operator that is selected adds its own configuration screens later in the wizard. Those screens are described with the operator concerned, in chapter [6](/containerization/kubernetesoperators). ![Kubernetes setup TUI session: selecting the operators to install](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/dbd1cda53e362428e5f3f5237cc8d42965cadadd6525f19795f617f3e9a758f0/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/kube-urwid-wizard-operators-select.png) *Figure 4.16: Kubernetes setup TUI session: selecting the operators to install* Prior to BCM 11.33, Run:ai was installed as part of `cm-kubernetes-setup` (figure [4.17](#fig-kube-urwid-wizard-operators-runai)). From BCM 11.33 onwards, Run:ai is deployed separately using the `cm-runai-setup` wizard after Kubernetes setup is complete (section [6.11.2.2](/containerization/kubernetesoperators#61122-from-bcm-1133-using-cm-runai-setup)). ![Kubernetes setup TUI session: Run:ai operator selection (prior to BCM 11.33)](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/d1d81f8059af732e3396ab300d48d77dabd619bbd5deac73796c3feed2700950/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/kube-urwid-wizard-operators-runai.png) *Figure 4.17: Kubernetes setup TUI session: Run:ai operator selection (prior to BCM 11.33)* **MetalLB mode:** MetalLB advertises the Gateway addresses either by ARP, in layer 2 mode, or by BGP (figure [4.32](#fig-kube-gateway-metallb-mode)). Layer 2 mode is pre-selected and needs no external peer. BGP mode advertises through an external peer. **Gateway IP addresses:** One IP address is requested for each Gateway that is configured (figure [4.18](#fig-kube-urwid-wizard-gateway-ip)). This is the address at which the Gateway is reached. The field is pre-filled with the highest address on the Kubernetes node network that is not already in use. Addresses assigned to devices, addresses used by the head node for its filesystem mounts, the dynamic range of the network, and ranges already claimed by MetalLB address pools are all excluded. On a cluster with one head node this is typically the address just below the head node itself. The pre-filled value can be replaced. An address that is already assigned to a device, or that overlaps a range already in use, is rejected with an error. ![Kubernetes setup TUI session: Gateway IP address](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/b0538a05a1c7788033ffd445dee29ee7f7a7607fabcb2ae7051c365a9e1efcb7/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/kube-urwid-wizard-gateway-ip.png) *Figure 4.18: Kubernetes setup TUI session: Gateway IP address* **MetalLB IP address pools:** Address pools can be defined for MetalLB to allocate from (figure [4.19](#fig-kube-urwid-wizard-metallb-address-pools)). The list is empty by default. Rows are added with `-` and removed with `-`. Leaving the list empty is correct on the default path. The Gateway does not take its address from a pool defined here. A pool holding only the Gateway address given on the previous screen is generated during the installation itself. Pools are added on this screen only for other services of type `LoadBalancer`. ![Kubernetes setup TUI session: MetalLB IP address pools](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/9f43d9c9162c2b28618d448825e28ec55566a0f6a5fd9d9d18483d065f277d88/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/kube-urwid-wizard-metallb-address-pools.png) *Figure 4.19: Kubernetes setup TUI session: MetalLB IP address pools* **Exposing the Kubernetes Gateways:** The Gateways can be exposed to the external network by enabling the Gateway proxy on the head nodes and edge directors (figure [4.34](#fig-kube-external-ingress)). This is enabled by default. **Gateway proxy configuration:** The port that the Gateway proxy listens on is set here (figure [4.35](#fig-kube-external-gateway-proxy-port)). The default is port `443`. **BCM NVIDIA packages:** The `cuda-dcgm` package can be installed into the software images used by the Kubernetes nodes (figure [4.20](#fig-kube-urwid-wizard-bcm-nvidia-packages)). The screen lists the images that would be modified. The default is not to install it. ![Kubernetes setup TUI session: installing the BCM NVIDIA packages](/base-command-manager/product-manuals/_fern-img/9e34950fe54c4419cb8bbe5d3bd3f430995d42182f89987b7b1afc7a2f8dedcc.webp) *Figure 4.20: Kubernetes setup TUI session: installing the BCM NVIDIA packages* **BCM NVIDIA Container Toolkit package:** The `cm-nvidia-container-toolkit` package can be installed on the head node and into the software images (figure [4.21](#fig-kube-urwid-wizard-bcm-nvidia-container-toolkit)). The screen lists the paths that would be modified. The default is not to install it. ![Kubernetes setup TUI session: installing the BCM NVIDIA Container Toolkit package](/base-command-manager/product-manuals/_fern-img/40ff4af02f9ef29bfb537a61fa6b5d8850d8668b56ec100a211339dbbaa343ef.webp) *Figure 4.21: Kubernetes setup TUI session: installing the BCM NVIDIA Container Toolkit package* **Both NVIDIA package screens are skipped if the GPU Operator is selected:** The two preceding screens are presented only when the NVIDIA GPU Operator has not been selected. If it has been selected, then the wizard decides both answers itself and does not ask: * the BCM NVIDIA packages are installed, because they are almost always wanted alongside the operator * the BCM NVIDIA Container Toolkit package is not installed, because the GPU Operator deploys and manages the container toolkit itself by default (chapter [6](/containerization/kubernetesoperators)) Running `cm-kubernetes-setup` with the `--adv-configuration-bcm-nvidia-packages` option presents both screens even when the GPU Operator has been selected, so that the answers can be set by hand. **Permission manager:** Kubernetes Permissions Manager (section [4.15](#415-kubernetes-permission-manager)) is installed by default (figure [4.22](#fig-kube-urwid-wizard-permissions-manager)). As the screen states, it is needed only if non-root users are to use the cluster. ![Kubernetes setup TUI session: installing Kubernetes Permissions Manager](/base-command-manager/product-manuals/_fern-img/6d74523d8c8a0a2ca76840a395974796257388d55bce86b9fc3cd25836f38a8f.webp) *Figure 4.22: Kubernetes setup TUI session: installing Kubernetes Permissions Manager* **Storage class:** The storage class for the cluster is configured here (figure [4.23](#fig-kube-urwid-wizard-storage-class)). Local path storage is enabled by default, and is set as the default storage class. The alternative is to set no default storage class. ![Kubernetes setup TUI session: configuring the Kubernetes storage class](/base-command-manager/product-manuals/_fern-img/45168ab43abd8173368223f953772dc3f509388d8019f98d0e8938e39bc35838.webp) *Figure 4.23: Kubernetes setup TUI session: configuring the Kubernetes storage class* **Local path storage pool:** If local path storage is enabled, then the directory that backs it is set here (figure [4.24](#fig-kube-urwid-wizard-local-path-pool)). The default is `/cm/shared/apps/kubernetes/`\<*cluster*>`/var/volumes`, where \<*cluster*> is the name of the Kubernetes cluster. A custom provisioner image can also be given, and is optional. ![Kubernetes setup TUI session: configuring the local path storage pool](/base-command-manager/product-manuals/_fern-img/cf304ff73a330e06a20ad9736fb9cfc75b9603c7f8b4ce62c20dc43eabbb5638.webp) *Figure 4.24: Kubernetes setup TUI session: configuring the local path storage pool* **Summary:** The final screen offers the deployment and configuration file options (figure [4.25](#fig-kube-urwid-wizard-summary)). The configuration can be deployed straight away, or written to a file to be deployed later with `-c`, or both. `Save config & deploy` is pre-selected. The `minimal config` variants write a configuration file that records only the values that differ from the defaults. The full variants record every value. A minimal file is easier to read and to carry between systems. A full file is a complete record of the deployment, and does not change meaning if a default changes in a later BCM release. ![Kubernetes setup TUI session: summary screen](/base-command-manager/product-manuals/_fern-img/b3377a4c2f5a653c15f448230b8c2bf951f3138f80a94f724eaad089d34eddd7.webp) *Figure 4.25: Kubernetes setup TUI session: summary screen* ### 4.2.9 Testing Kubernetes To test that Kubernetes works, the `cluster-info` command can be run. For example, on the head node, `basecm11`: **Example** ```bash [root@basecm11 ~]# module load kubernetes #not actually needed -- autoloaded these days [root@basecm11 ~]# kubectl cluster-info Kubernetes control plane is running at https://127.0.0.1:10443 CoreDNS is running at https://127.0.0.1:10443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. ``` After `cm-kubernetes-setup` finishes, and the regular nodes have been rebooted, the state of the nodes can be found by running the `get nodes` command: **Example** ```bash [root@basecm11 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION node001 Ready worker 18h v1.35 node002 Ready worker 18h v1.35 basecm11 Ready control-plane 18h v1.35 ``` A six node cluster should show the following Kubernetes installed add-ons, when using `kubectl` with the `get all -n kube-system` option (some lines truncated): **Example** ```bash [root@basecm11 ~]# kubectl get all -n kube-system NAME READY STATUS RESTARTS AGE pod/calico-kube-controllers-58497c65d5-skhgw 1/1 Running 0 26m pod/calico-node-27xj7 1/1 Running 0 26m pod/calico-node-6hmm5 0/1 Running 1 26m pod/calico-node-987qv 1/1 Running 0 26m pod/calico-node-gcbcm 1/1 Running 0 26m pod/calico-node-hlsrj 1/1 Running 0 26m pod/calico-node-q7k4v 1/1 Running 0 26m pod/calico-node-qdbq5 0/1 Running 0 26m pod/calico-node-v2dxj 1/1 Running 0 26m pod/coredns-6768db756-8l9fs 1/1 Running 0 26m pod/coredns-6768db756-cs58q 1/1 Running 0 26m pod/kube-state-metrics-758ccc75d6-75dsn 1/1 Running 0 26m pod/metrics-server-7b477dd7b9-2drkg 1/1 Running 0 26m pod/metrics-server-7b477dd7b9-z6nch 1/1 Running 0 26m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/calico-typha ClusterIP 10.150.121.25 5473/TCP 26m service/kube-dns ClusterIP 10.150.255.254 53/UDP,53/TCP,9153/TCP 26m service/kube-state-metrics ClusterIP None 8080/TCP,8081/TCP 26m service/metrics-server ClusterIP 10.150.99.149 443/TCP 26m NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/calico-node 8 8 6 8 6 kubernetes.io/os=linux 26m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/calico-kube-controllers 1/1 1 1 26m deployment.apps/calico-typha 0/0 0 0 26m deployment.apps/coredns 2/2 2 2 26m deployment.apps/kube-state-metrics 1/1 1 1 26m deployment.apps/metrics-server 2/2 2 2 26m NAME DESIRED CURRENT READY AGE replicaset.apps/calico-kube-controllers-58497c65d5 1 1 1 26m replicaset.apps/calico-typha-68857595fc 0 0 0 26m replicaset.apps/coredns-6768db756 2 2 2 26m replicaset.apps/kube-state-metrics-758ccc75d6 1 1 1 26m replicaset.apps/metrics-server-7b477dd7b9 2 2 2 26m ``` The administrator can now configure the cluster to suit the particular site requirements. ## 4.3 Using GPUs With Kubernetes: NVIDIA GPUs ### 4.3.1 Prerequisites The GPUs have to be recognized by the nodes, and have the appropriate drivers (such as `nvidia-driver`) installed. Details on how to do this are given in [Chapter 9 of the *Installation Manual*](/installation/nvidiagpus). To verify the GPUs are recognized and have drivers in place, the `nvidia-smi` command can be run. The response displayed for a GPU should look similar to the following: **Example** ```bash root@node001:~# nvidia-smi Mon May 5 13:03:52 2025 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 570.82 Driver Version: 570.82 CUDA Version: 12.8 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA Graphics Device On | 00000008:01:00.0 Off | 0 | | N/A 38C P0 184W / 1200W | 1MiB / 189471MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA Graphics Device On | 00000009:01:00.0 Off | 0 | | N/A 36C P0 160W / 1200W | 1MiB / 189471MiB | 0% Default | | | | Disabled | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+ ``` If a non-BCM Containerd has already been deployed before Kubernetes is deployed, then `cm-kubernetes-setup` may replace an existing Containerd configuration file in order to enable NVIDIA GPU integration using a Kubernetes CNI plugin. This is because Containerd is configured by `cm-kubernetes-setup`, overwriting any previous configuration. ### 4.3.2 New Kubernetes Installation As part of the setup, `cm-kubernetes-setup` assigns a new role to the Kubernetes worker nodes: `generic::containerd`. The role has a `Configurations` submode, in which the `containerd-cri` object can be configured. The entry for `Filename` specifies the path to the `cri.toml` file, which contains content used by the container runtime interface on the Kubernetes worker nodes that have been assigned the role. **Example** ```bash [basecm11->configurationoverlay[kube-default-worker]->roles]% use generic::containerd [...]->roles[generic::containerd]]% show Parameter Value -------------------------------- ------------------------------------------------ Name generic::containerd Type GenericRole Add services yes Services containerd Configurations <2 in submode> Environments <1 in submode> Exclude list snippets <1 in submode> Data node no [...]->roles[generic::containerd]]% configurations [...]->roles[generic::containerd]->configurations]% use containerd-cri [...]->roles[generic::containerd]->configurations[containerd-cri]]% show Parameter Value -------------------------------- -------------------------------------------------- Name containerd-cri Type static Create directory yes Filename /cm/local/apps/containerd/var/etc/conf.d/cri.toml Filemask directory 0644 User name Group name Disabled no Service action on write RESTART Service stop on failure yes Content <645B> Filemask 0644 ``` The file with the CRI (Container Runtime Interface) configuration is created in directory: ```bash /cm/local/apps/containerd/var/etc/conf.d ``` and included into the main Containerd configuration file: ```bash /cm/local/apps/containerd/var/etc/config.toml ``` with the `imports` statement: ```bash imports = ["/cm/local/apps/containerd/var/etc/conf.d/*.toml"] ``` Whatever the container runtime that is selected, if NVIDIA GPU integration is required then the NVIDIA container toolkit is taken care of by the installer. The configuration for Containerd is added as a separate configuration file in the `configurations` submode, as `nvidia-cri`. NVIDIA GPUs are integrated into Kubernetes using the NVIDIA GPU operator. This is discussed further in section [6.7](/containerization/kubernetesoperators#67-the-nvidia-gpu-operator). ### 4.3.3 Existing Kubernetes Installation The NVIDIA GPU operator can always be deployed through Helm. The official documentation for the NVIDIA GPU operator is at [https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/overview.html](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/overview.html) ## 4.4 Using GPUs With Kubernetes: AMD GPUs ### 4.4.1 Prerequisites The GPUs have to be recognized by the node. One way to check this from within BCM is to run `sysinfo` for the node: **Example** ```bash [basecm11->device[basecm11]]% sysinfo | grep GPU Number of GPUs 1 GPU Driver Version 4.18.0-193.el8.x86_64 GPU0 Name Radeon Instinct MI25 ``` In order to make Kubernetes aware of nodes that have AMD GPUs, the AMD GPU device plugin has to be deployed as a DaemonSet inside Kubernetes. The official GitHub repository that hosts this plugin can be found at: [https://github.com/RadeonOpenCompute/k8s-device-plugin](https://github.com/RadeonOpenCompute/k8s-device-plugin) The device plugin requires Kubernetes v1.16+, which has been around since BCM version 9.0. With some extra instructions, the plugin can also be made a part of BCM version 8.2. The DaemonSet YAML file can be deployed with: **Example** ```bash kubectl create -f https://raw.githubusercontent.com/RadeonOpenCompute/k8s-device-plugin/v1.16/ k8s-ds-amdgpu-dp.yaml ``` ### 4.4.2 Managing The YAML File Through CMDaemon The plugin can be added by the user using the Kubernetes appgroups as an application. In the session that follows, it is given the arbitrary name `device-plugin`: **Example** ```bash [root@basecm11 ~]# wget https://raw.githubusercontent.com/RadeonOpenCompute/k8s-device-plugin/ v1.16/k8s-ds-amdgpu-dp.yaml -O /tmp/k8s-ds-amdgpu-dp.yaml [root@basecm11 ~]# cmsh [basecm11]% kubernetes [basecm11->kubernetes[default]]% appgroups [basecm11->kubernetes[default]->appgroups]% add amd [basecm11->kubernetes*[default*]->appgroups*[amd*]]% applications [basecm11->kubernetes*[default*]->appgroups*[amd*]->applications]% add device-plugin ``` The configuration of the plugin can be set to the YAML file, by setting the `config` parameter to take the value of the YAML file path. **Example** ```bash [basecm11->...[amd*]->applications*[device-plugin*]]% set config /tmp/k8s-ds-amdgpu-dp.yaml [basecm11->kubernetes*[default*]->appgroups*[amd*]->applications*[device-plugin*]]% show Parameter Value -------------------------------- ------------------------------------------------ Name device-plugin Revision Format Yaml Enabled yes Config <914B> Environment <0 in submode> Exclude list snippets <0 in submode> [basecm11->kubernetes*[default*]->appgroups*[amd*]->applications*[device-plugin*]]% commit ``` The YAML file can also be edited within `cmsh` after it has been set, by running `set config` without a value. There are older releases available, starting from Kubernetes v1.10, if needed. Saving this device-plugin YAML should result in pods being scheduled on all the non-tainted nodes, as seen by listing the pods (some columns elided): ```bash [root@basecm11 ~]# kubectl get pod -n kube-system -l name=amdgpu-dp-ds -o wide NAME READY STATUS ... IP NODE ... amdgpu-device-plugin-daemonset-66jl7 1/1 Running ... 172.29.112.135 gpu001 ... amdgpu-device-plugin-daemonset-8mh9w 1/1 Running ... 172.29.152.130 gpu002 ... ``` ### 4.4.3 Including Head Nodes as part of the DaemonSet BCM taints head nodes, so that they do not run non-critical pods. The taint can be removed with the “`-`” operator to allow non-critical pods to run: **Example** ```bash kubectl taint nodes basecm11 node-role.kubernetes.io/control-plane- ``` However, a more specific exception can be configured in the DaemonSet itself. Within the YAML file, the following existing `tolerations` definition has to be modified, from: ```bash tolerations: - key: CriticalAddonsOnly operator: Exists ``` to: ```bash tolerations: - key: node-role.kubernetes.io/control-plane effect: NoSchedule operator: Exists ``` The modified toleration tolerates this taint, and therefore has the device plugin run on such tainted nodes. #### 4.4.3.1 Verifying That AMD GPUs Are Recognized By Kubernetes If Kubernetes is aware of the AMD GPUs for a node then several mentions of `amd.com/gpu` are displayed when running the `kubectl describe node` command for the node. The following session shows output for a node `gpu001`, ellipsized for clarity: **Example** ```bash [root@basecm11 ~]# kubectl describe node gpu001 Name: gpu001 ... Capacity: amd.com/gpu: 3 cpu: 64 ephemeral-storage: 1813510Mi hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 527954676Ki pods: 50 ... ``` ### 4.4.4 Running The DaemonSet Only On Specific Nodes The AMD GPU device plugin, unlike the NVIDIA GPU device plugin Daemonset, is scheduled to run on each Kubernetes host. This means that it runs even if the host has no GPU. This can be prevented with the following steps: A LabelSet can be created using `cmsh`, and the nodes or categories that have GPUs are assigned within the `labelsets` mode: **Example** ```bash [root@basecm11 ~]# cmsh [basecm11]% kubernetes [basecm11->kubernetes[default]]% labelsets [basecm11->kubernetes[default]->labelsets]% use nvidia [basecm11->kubernetes[default]->labelsets[nvidia]]% .. #but, we're using AMD GPUs, so let's go back up: [basecm11->kubernetes[default]->labelsets]% add amd [basecm11->kubernetes*[default*]->labelsets*[amd*]]% set labels nvidia.com/amd-gpu-accelerator= [basecm11->kubernetes*[default*]->labelsets*[amd*]]% append categories gpu-nodes [basecm11->kubernetes*[default*]->labelsets*[amd*]]% commit ``` This assigns the labels to the nodes with GPUs. This can be verified with: **Example** ```bash kubectl get nodes -l nvidia.com/amd-gpu-accelerator= NAME STATUS ROLES AGE VERSION gpu001 Ready 66m v1.32.6 gpu002 Ready 66m v1.32.6 ... ``` The DaemonSet YAML can now be adjusted to only run the device plugin on nodes with this new label. This can be done by adding an affinity block after the tolerations block: **Example** ```bash tolerations: - key: CriticalAddonsOnly # toleration may be different, if changes were made to it operator: Exists affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: 'nvidia.com/amd-gpu-accelerator' operator: Exists ``` This results in the device plugin pods being removed immediately from all nodes that do not have the label. ### 4.4.5 Running An Example Workload An example workload can be run as described in the official AMD GPU Kubernetes device plugin documentation at: [https://github.com/RadeonOpenCompute/k8s-device-plugin/tree/v1.16#example-workload](https://github.com/RadeonOpenCompute/k8s-device-plugin/tree/v1.16#example-workload) Thus it should now be possible to run: ```bash [root@basecm11 ~]# kubectl create -f https://raw.githubusercontent.com/RadeonOpenCompute/ k8s-device-plugin/v1.16/example/pod/alexnet-gpu.yaml ``` The YAML requests only one GPU at the bottom of the YAML file: ```bash apiVersion: v1 kind: Pod metadata: name: alexnet-tf-gpu-pod labels: purpose: demo-tf-amdgpu spec: containers: - name: alexnet-tf-gpu-container image: rocm/tensorflow:latest workingDir: /root env: - name: HIP_VISIBLE_DEVICES value: "0" # # 0,1,2,...,n for running on GPU and select the GPUs, -1 for running on CPU command: ["/bin/bash", "-c", "--"] args: ["python3 benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py --model=alexnet;\ trap : TERM INT; sleep infinity & wait"] resources: limits: amd.com/gpu: 1 # requesting a GPU ``` Container creation might take a while due to the image size. Once scheduled, it prints out that it found exactly one GPU, and proceeds to run a TensorFlow workload. **Example** ```bash [root@basecm11 ~]# kubectl logs -f alexnet-tf-gpu-pod WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version. Instructions for updating: non-resource variables are not supported in the long term 2021-01-08 21:03:29.222293: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2495445000 Hz 2021-01-08 21:03:29.222398: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x39f62f0 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2021-01-08 21:03:29.222420: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2021-01-08 21:03:29.223754: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libamdhip64.so 2021-01-08 21:03:31.635339: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3a40bb0 initialized for platform ROCM (this does not guarantee that XLA will be used). Devices: 2021-01-08 21:03:31.635363: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Device 738c, AMDGPU ISA version: gfx908 2021-01-08 21:03:31.931125: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 0 with properties: pciBusID: 0000:27:00.0 name: Device 738c ROCm AMD GPU ISA: gfx908 coreClock: 1.502GHz coreCount: 120 deviceMemorySize: 31.98GiB deviceMemoryBandwidth: 1.12TiB/s ... TensorFlow: 2.3 Model: alexnet Dataset: imagenet (synthetic) Mode: training SingleSess: False Batch size: 512 global 512 per device Num batches: 100 Num epochs: 0.04 Devices: ['/gpu:0'] ... ``` Had more GPUs been requested, more would have been made available to the container. For comparison, a CPU version of the container is also available. The official instructions can be referred to for these, too. ## 4.5 Kubernetes Configuration Overlays A list of configuration overlays can be seen from within `configurationoverlay` mode: **Example** ```bash [basecm11->configurationoverlay]% list Name (key) Priority Nodes Categories Roles ------------------- -------- ----------------- ---------- -------------------------... kube-default-etcd 500 node001..node003 Etcd::Host kube-default-master 510 node001..node003 generic::containerd, kube... kube-default-worker 500 node004..node006 default generic::containerd, kube... kube-default-netq 501 node001 generic::netq ``` The NetQ configuration overlay is only normally displayed if NetQ is deployed. Configuration overlays can be used to manage the Kubernetes services used with a particular configuration. For example, when managing the Kubernetes services used for a Kubernetes engine within an Auto Scale tracker ([section 8.4.9 of the *Administrator Manual*](/admin/cm-scale#849-auto-scaling-a-job-on-premises-to-a-workload-manager-and-kubernetes)). ## 4.6 Removing A Kubernetes Cluster A Kubernetes cluster can be removed using `cm-kubernetes-setup` with the `--remove` and `--yes-i-really-mean-it` options. Also, if there is more than one cluster present, then the cluster name must be specified using the `--cluster` parameter. A removal run looks as follows (some output ellipsized): **Example** ```bash [root@basecm11 ~]# cm-kubernetes-setup --remove --cluster default --yes-i-really-mean-it Connecting to CMDaemon Executing 20 stages ################### Starting execution for 'Kubernetes Setup' - kubernetes - docker ## Progress: 0 #### stage: kubernetes: Get Kube Cluster ## Progress: 5 #### stage: kubernetes: Check Kube Cluster Exists ## Progress: 10 #### stage: kubernetes: Find Installed Components ## Progress: 15 #### stage: kubernetes: Find Files On Headnodes ## Progress: 20 #### stage: kubernetes: Firewall Zone Close ## Progress: 25 #### stage: kubernetes: Firewall Interface Close ## Progress: 30 #### stage: kubernetes: Firewall Policy Close ## Progress: 35 #### stage: kubernetes: Nginx Reverse Proxy Close ## Progress: 40 #### stage: kubernetes: IP Ports Close ## Progress: 60 #### stage: kubernetes: Remove Installed Components ## Progress: 65 #### stage: kubernetes: Remove Files On Headnodes ## Progress: 70 #### stage: kubernetes: Remove Etcd Spool ## Progress: 80 #### stage: kubernetes: Set Reboot Required You need to reboot 2 nodes to cleanup the network configuration ## Progress: 85 #### stage: kubernetes: Collection Update Provisioners ## Progress: 100 Took: 00:08 min. Progress: 100/100 ################### Finished execution for 'Kubernetes Setup', status: completed Kubernetes Setup finished! ``` Using the `--remove` option removes the Kubernetes cluster configuration from BCM, unassigns Kubernetes-related roles—including the EtcdHost role—and removes Kubernetes health checks. The command does not remove packages that were installed with a `cm-kubernetes-setup` command before that. After the disabling procedure has finished, the cluster has no Kubernetes configured and running. ## 4.7 Kubernetes Cluster Configuration Options Kubernetes allows many Kubernetes clusters to be configured. These are separated sets of hosts with different certificates, users and other global settings. When carrying out the Kubernetes setup run, a Kubernetes cluster name is asked, and a new object with the cluster settings is then added into the CMDaemon configuration. The administrator can change the settings of the cluster from within the `kubernetes` mode of `cmsh`, or within the `Kubernetes Clusters` options window of Base View accessible using the navigation path `Containers` > `Kubernetes Clusters`. The `cmsh` equivalent looks like: **Example** ```bash [root@basecm11 ~]# cmsh [basecm11]% kubernetes list Name (key) ------------------ default [basecm11]% kubernetes use default [basecm11->kubernetes[default]]% show Parameter Value ------------------------------------ ------------------------------------------------------ Name default Revision Etcd Cluster kube-default Pod Network kube-default-pod Pod Network Node Mask Internal Network internalnet KubeDNS IP 10.150.255.254 Kubernetes API server Kubernetes API server proxy port 10443 App Groups <1 in submode> Label Sets <3 in submode> Notes Version 1.27.10-150500.1.1 Trusted domains basecm11.openstacklocal,master,localhost,10.141.255.254 Module file template <1.01KiB> Kubeadm init file <1.19KiB> Service Network kube-default-service Kubeadm CERT Key ********* Kube CA Cert ********* Kube CA Key ********* Kubernetes users <0 in submode> External no External Kubernetes Ingress server External port 0 Capi template no Capi namespace default Kubernetes management cluster [basecm11->kubernetes[default]]% ``` The preceding `kubernetes` mode parameters are described in table [4.1](#kubernetes-mode-parameters): | **Parameter** | **Description** | | :----------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `App Groups` | Groups of Kubernetes add-ons managed by CMDaemon. | | `CAPI namespace` | Namespace where CAPI is deployed (optional). | | `CAPI template` | Is this Kubernetes cluster configuration a template for CAPI workload clusters? | | `Etcd Cluster` | The etcd cluster instance. | | `External` | Is this Kubernetes cluster configuration for an external Kubernetes instance (e.g., cloud)? | | `External Kubernetes Ingress server` | The Ingress endpoint for the external Kubernetes. | | `External port` | Additional port used by external Kubernetes cluster (unused). | | `Internal Network` | Network to back the internal communications. | | `Kube CA Cert` | Path to PEM-encoded RSA or ECDSA certificate used for the CA | | `Kube CA Key` | Path to PEM-encoded RSA or ECDSA private key used for the CA | | `KubeDNS IP` | CoreDNS IP Address. | | `Kubeadm Cert Key` | Key used to encrypt the control plane certificates in the kubeadm-certs Secret. | | `Kubeadm Init file` | Contents of the init configuration file provided to Kubeadm to initialize or join nodes. | | `Kubernetes API Server` | Kubernetes API server address (format: `https://` \<*host*>`:<`*port number*>). | | `Kubernetes API Server Proxy Port` | Kubernetes API server proxy (NGINX LoadBalancer) port (default: 10443). | | `Kubernetes Management cluster` | Relevant to CAPI only: the Kubernetes cluster managing this CAPI workload cluster (optional). | | `Kubernetes Users` | Submode to manage users for this Kubernetes cluster (e.g.: whether or not to manage their Kube config files) | | `Label Sets` | Submode to manage assignment of labels to Kubernetes nodes | | `Module file template` | Template used for writing the Kubernetes TCL module file. | | `Name` | The name, identifier, or label for the Kubernetes cluster. | | `Pod Network` | Network where pod IP addresses are assigned from | | `Pod Network Node Mask` | Corresponding subnet mask used to define the size of the address space allocated to each node for its pods | | `Service Network` | Network from which the service cluster IP addresses are assigned, in IPv4 CIDR format. Must not overlap with any IP address ranges assigned to nodes for pods. Default: `172.29.0.0/16` | | `Trusted Domains` | Trusted domains to be included in Kubernetes-related certificates as Alternative Subject Names. | | `Version` | Version of the Kubernetes cluster. | *Table 4.1: `kubernetes` mode parameters* ## 4.8 EtcdCluster The EtcdCluster mode sets the global etcd cluster settings. It can be accessed using the top level `etcd` mode of `cmsh`.
**Parameter** **Description** **Option to `etcd`**
`Name` etcd cluster name. `--initial-cluster-token`
`Election Timeout` Election timeout, in milliseconds. `--election-timeout`
`Heart Beat Interval` Heart beat interval, in milliseconds. `--heartbeat-interval`
`CA` The certificate authority (CA) Certificate path for etcd, used to generate certificates for etcd. `--peer-trusted-ca-file`
`CA Key` The CA Key path for etcd, used to generate certificates for etcd.
`Member Certificate` The certificate path to use for etcd cluster members, signed with the etcd CA. The EtcdHost role can specify a member CA as well, and in that case it overwrites any value set here. `--peer-cert-file`
`Member Certificate Key` The key path to use for etcd cluster members, signed with the etcd CA. The EtcdHost role can specify a member CA as well, and in that case it overwrites any value set here. `--peer-key-file`
`Client CA` The CA used for client certificates. When set it is assumed client certificate and key are generated and signed with this CA by another party. etcd still expects the path to be correct for the client certificate and key. `--trusted-ca-file`
`Client Certificate` The client certificate, used by etcdctl, for example. `--cert-file`
`Client Certificate Key` The client certificate key, used by etcdctl for example. `--key-file`
\* Boolean (takes `yes` or `no` as a value) *Table 4.2: EtcdCluster role parameters and etcd options* ## 4.9 Kubernetes Roles Kubernetes roles include the following roles: * `EtcdHost` ([here](#491-etcdhost-role)) * `KubernetesApiServerProxy` ([here](#492-the-kubernetesapiserverproxy-role)) * `Kubelet` ([here](#494-the-kubelet-role)) * `generic::containerd` ([here](#495-containerd-role)) * `generic::netq` ([here](#45-kubernetes-configuration-overlays)) When nodes are configured using Kubernetes roles, then settings in these roles may sometimes use the same values (pointer variables). **Example** ```bash [basecm11->configurationoverlay[kube-default-etcd]->roles[Etcd::Host]]% get etcdcluster kube-default ``` and ```bash [basecm11->kubernetes[default]]% get etcdcluster kube-default ``` Pointer variables such as these have definitions that are shared across the roles, as indicated by the parameter description tables for the roles, and which are described in the following pages. In `cmsh`, the roles can be assigned: * for individual nodes using the `roles` submode of `device` mode * for a category using the `roles` submode of a category * for a configuration overlay using the `roles` submode of `configurationoverlay` mode ### 4.9.1 EtcdHost Role The EtcdHost role is used to configure and manage the `etcd` service for a node. The `etcd` service manages the `etcd` database, which is a hierarchical distributed key-value database. The database is used by Kubernetes to store its configurations. The EtcdHost role parameters are described in table [4.3](#etcdhost-role-parameters-table):
**Parameter** **Description** **Option to `etcd`**
`Member Name` The human-readable name for this `etcd` member ( `$hostname` is replaced by the node hostname) `--name`
`Spool` Path to the data directory (default: `/var/lib/etcd` ) `--data-dir`
`Advertise Client` List of client URLs for this member to `--advertise-client-urls`
`URLs` advertise publicly (default: `http://$hostname:5001` )
`Advertise Peers` List of peer URLs for this member to `--initial-advertise-peer-urls`
`URLs` advertise to the rest of the cluster (default: `http://$hostname:5002` )
`Listen Client URLs` List of URLs to listen on for client traffic (default: `http://$hostname:5001` , [http://127.0.0.1:2379](http://127.0.0.1:2379) ) `--listen-client-urls`
`Listen Peer URLs` List of URLs to listen on for peer traffic (default: `http://$hostname:5002` ) `--listen-peer-urls`
`Snapshot Count` Number of committed transactions that trigger a snapshot to disk (default: `5000` ) `--snapshot-count`
`Debug` \* Drop the default log level to DEBUG for all subpackages? (default: `no` ) `--debug`
`Member Certificate` etcd member certificate, signed with CA specified in the etcd cluster. Setting it overrules the value set in the EtcdCluster object ( `etcd` mode, section [4.8](#48-etcdcluster) ). Default empty. `--peer-cert-file`
`Member Certificate Key` etcd member certificate key, signed with CA specified in the etcd cluster. Setting it overrules the value set in the EtcdCluster object ( `etcd` mode, section [4.8](#48-etcdcluster) ). Default empty. `--peer-key-file`
`Options` Additional parameters for the `etcd` daemon (empty by default)
\* Boolean (takes `yes` or `no` as a value) *Table 4.3: EtcdHost role parameters and `etcd` options* The `etcd` settings are updated by BCM in `/cm/local/apps/etcd/current/etc/cm-etcd.conf`. ### 4.9.2 The KubernetesAPIServerProxy Role The KubernetesApiServerProxy role sets up a proxy that provides the entry point for one or more instances of the Kubernetes API server. The proxy runs on every node of a Kubernetes cluster instance, including the head node. If multiple Kubernetes control-plane nodes are present, then it enables HA for the Kubernetes control plane components (section [4.1](#41-reference-architecture)). This means that local port 10443 on all these nodes load balances over all control-plane node API servers. Port 10443 can also be opened up for external access. The `cm-kubernetes-setup` wizard (section [4.2](#42-kubernetes-setup)) in one of its screens (figure [4.26](#fig-kube-external-network)) prompts for this. ![Wizard question](/base-command-manager/product-manuals/_fern-img/8508b0bb27ab04fc3ff780a930ac6cef46458b0519b33d8a70b87b9ed370984f.webp) *Figure 4.26: Wizard question* If the prompt is answered with `no`, then the firewall role on the head node can still be customized with `cmsh` later on: **Example** ```bash root@basecm11:~# cmsh [basecm11]% device use master [basecm11->device[basecm11]]% roles [basecm11->device[basecm11]->roles]% use firewall [basecm11->device[basecm11]->roles[firewall]]% openports [basecm11->device[basecm11]->roles[firewall]->openports]% list Index Action Network Port Destination Protocol Address Description ------ --------- ------------ ------------ ------------ --------- ------------ ------------------------ [basecm11->device[basecm11]->roles[firewall]->openports]% add ACCEPT net 10443 tcp fw ok. [basecm11->device*[basecm11*]->roles*[firewall*]->openports[0]]% commit [basecm11->device[basecm11]->roles[firewall]->openports[0]]% list Index Action Network Port Destination Protocol Address Description ------ --------- ------------ ------------ ------------ --------- ------------ ------------------------ 0 ACCEPT net 10443 fw TCP 0.0.0.0/0 [basecm11->device[basecm11]->roles[firewall]->openports[0]]% Thu Oct 31 15:58:56 2024 [notice] basecm11: Service shorewall was restarted ``` ### 4.9.3 The KubernetesIngressServerProxy Role This role used to provide the “Ingress Proxy” (also known as “Ingress Server Proxy”) functionality, configuring the head-node NGINX to expose the Kubernetes ingress externally. In BCM 11.33.0 and later the role is almost a placeholder, because the head-node NGINX configuration is now offloaded to the Kubernetes setup itself, which generates and applies it using the `Configure Gateway proxy to route traffic from head node(s) to Gateways` item of `cm-kubernetes-setup` (section [4.17.6.3](#41763-the-configure-gateway-proxy-menu-item)). The role and the related Kubernetes cluster entity parameters (for example `Ingress Proxy Enable`, `Ingress Proxy Listen Port`, `Ingress Proxy Backend Port`) keep their original names for backward compatibility. ### 4.9.4 The Kubelet Role The Kubelet role is used to configure and manage the `kubelet` service. BCM takes care of joining these nodes with `kubeadm join` when needed. **Control plane:** The Kubelet role has a parameter `Control plane` that is set to the value `yes` for control plane nodes. In that case it also runs Kubernetes control plane services, such as: * [Kubernetes API server (`kube-apiserver`)](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/), * [Kubernetes scheduler (`kube-scheduler`)](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/), * [Kubernetes controller manager (`kube-controller-manager`)](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/), * [Kubernetes network proxy (`kube-proxy`)](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/), * [CoreDNS (`coredns`)](https://coredns.io/manual/toc/#what-is-coredns). **Workers:** The Kubelet role has a parameter `worker` that is set to `yes` for worker nodes. In that case, the control plane pods will not be running on the node. The Kubelet role parameters are described in table [4.4](#kubelet-role-parameters-table):
**Parameter** **Description**
`Kubernetes Cluster` The Kubernetes cluster instance (a pointer)
`Control plane` Is Kubelet running services on this node, making it a control plane node?
`Worker` Kubelet is a worker flag
`Max Pods` Configuration to change max number of pods on Kubelet
`Options` Submode that allows configuration of additional flags to containers specified in manifest `/etc/kubernetes/manifests`
\* Boolean (takes `yes` or `no` as a value) *Table 4.4: Kubelet role parameters and `kubelet` options* Further details on the Kubelet service can be found at [https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/). #### 4.9.4.1 CPU Manager And Topology Manager Policy On GPU Nodes From BCM 11.34 onwards, GPU worker nodes are automatically configured with a static CPU manager policy and a best-effort topology manager policy. These settings enable workloads running on GPU nodes to request exclusive CPU allocation and benefit from NUMA-aware scheduling. The default kubelet settings (`cpuManagerPolicy=none` and `topologyManagerPolicy=none`) are suboptimal for GPU workloads: without CPU pinning, containers share CPUs via time-slicing, which causes cache thrashing and inconsistent throughput in the GPU data pipeline; and without topology awareness, CPU, memory, and GPU allocations may be placed on different NUMA nodes, forcing host → device data transfers to cross the inter-socket interconnect at reduced bandwidth. The following kubelet settings are applied automatically to nodes that have GPUs: * `cpuManagerPolicy: static` enables the static CPU manager, which grants exclusive CPU access to guaranteed-QoS containers that request whole CPUs. * `topologyManagerPolicy: best-effort` enables the topology manager, which attempts to align CPU, memory, and device (GPU) allocations to the same NUMA node. * `topologyManagerScope: pod` applies topology alignment at the pod level, so that all containers in a pod share the same NUMA node affinity. * `kubeReserved: {cpu: 250m}` reserves a small amount of CPU for Kubernetes system components. Control plane nodes and non-GPU worker nodes use the default kubelet configuration and are not affected by these settings. Clusters deployed before BCM 11.34 are also not affected, since `kubeadm upgrade` reuses the existing in-cluster kubelet configuration. Further details on the CPU manager and topology manager policies can be found at [https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/](https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/) and [https://kubernetes.io/docs/tasks/administer-cluster/topology-manager/](https://kubernetes.io/docs/tasks/administer-cluster/topology-manager/). #### 4.9.4.2 The `options` Submode ```bash [basecm11->configurationoverlay[kube-default-master]->roles[kubelet]]% help options Name: options - Manage kubelet container command options Usage: options [kubelet] options [kubelet] options [kubelet] set [name] key=value options [kubelet] set [name] key options [kubelet] clear [name] key Examples: options kube-apiserver.yaml set advertise-address=192.168.200.148 [basecm11->configurationoverlay[kube-default-master]->roles[kubelet]]% options Filename Container Key Value ------------------- --------------- --------------------- ------------------------------------ kube-apiserver.yaml kube-apiserver cors-allowed-origins https://name.run.ai kube-apiserver.yaml kube-apiserver oidc-client-id runai kube-apiserver.yaml kube-apiserver oidc-issuer-url https://app.run.ai/auth/realms/name kube-apiserver.yaml kube-apiserver oidc-username-prefix - ``` The full path to the files should be specified for the `options` command. For example, `kube-apiserver.yaml` may be specified as `/etc/kubernetes/manifests/kube-apiserver.yaml`. ### 4.9.5 Containerd Role The Containerd role is used to configure and manage the `containerd` daemon. This is done through a generic role in BCM. Generic roles under Kubernetes are found under the `cmsh` path indicated by: `cmsh > configurationoverlay[`*`kube-object`*`] > roles[`*`generic::role`*`]` **Example** ```bash [basecm11->configurationoverlay[kube-default-master]->roles[generic::containerd]]% show Parameter Value -------------------------------- ------------------------------------------------ Name generic::containerd Revision Type GenericRole Add services yes Services containerd Configurations <2 in submode> Environments <1 in submode> Exclude list snippets <2 in submode> Data node no ``` The role has several submodes: * The `configurations` submode: This contains various configuration drop-ins. Configuration drop-ins that BCM may manage in the `generic::containerd` role are: * NVIDIA Container Toolkit configuration. * Docker registry certs directory configuration. * Docker Hub credentials configuration. * Harbor Registry Mirror configuration. * CRI cgroup configuration. * CNI bin dir configuration. * CRI registry configuration. * CRI custom sandbox image configuration. The drop-ins that are actually available depend on choices made by the cluster administrator during deployment. * The `environment` submode: This has optional environment variables that are made available to the configuration files and that are to be used as templates. * The `excludelistsnippets` submode: This has files related to containerd that need to be excluded, such as the most obvious directory `/var/lib/containerd`. ## 4.10 Security Model The Kubernetes security model allows authentication using a certificate authority (CA), with the user and daemon certificates signed by a Kubernetes CA. The Kubernetes CA should not be confused with the BCM CA. BCM lets `kubeadm` create a CA specifically for issuing all Kubernetes-related certificates. The certificates are put into `/etc/kubernetes/pki//` by default. In Kubernetes terminology a user is a unique identity accessing the Kubernetes API server. The user may be a human or an automated process. For example an admin or a developer is a human user, but kubelet represents an infrastructure user. Both types of users are authorized and authenticated in the same way against the API server. Kubernetes uses client certificates, tokens, or HTTP basic authentication methods to authenticate users for API calls. BCM configures client certificate usage by default. The authentication is performed by the API server which validates the user certificate using the common name part of the certificate subject. In Kubernetes, authorization happens as a separate step from authentication. Authorization applies to all HTTP accesses on the main (secure) API server port. BCM by default enables RBAC (Role-Based Access Control) combined with Node Authorization. The authorization check for any request thus takes the common name and/or organization part of the certificate subject to determine which roles the user or service has associated. Roles carry a certain set of privileges for resources within Kubernetes. ### 4.10.1 Kyverno BCM has support for the Kyverno policy engine ([https://kyverno.io/](https://kyverno.io/)). If Kyverno is installed, then Kubernetes Permissions Manager (section [4.15](#415-kubernetes-permission-manager)) creates policy manifests packed as a Helm chart for every user added to Kubernetes using `cm-kubernetes-setup`. In addition, a kyverno-policy chart is installed in enforce mode to implement Pod Security Standards ([https://kyverno.io/policies/](https://kyverno.io/policies/)). During installation, some exclusions are added to the policies automatically to make chosen features of the Kubernetes cluster work. For every created user the following defaults are applied: * The user has an associated service account with the same name * A \<*username*>-restricted namespace is created. So, for a user `john` the namespace is `john-restricted`. * An `edit` cluster role is bound to the service account in \<*username*>-restricted namespace. The user is allowed to create pods, services, configmaps, etc. in the namespace * The user is allowed to list nodes in the cluster * Kyverno policies are applied to the resources in \<*username*>-restricted namespace or to pods created or updated by the associated user * If hostPath is not the home directory of the user (of the format `/home/` \<*username*>) then the creation of the resource is denied * The UID and GID of the running process are set to the same value as the UID and GID of the PAM user Modifications from the defaults are: * If the `Allow any UID process in pods` checkbox is ticked, or if the `--allow-all-uids` argument is specified, then the UID and GID of the running process becomes the user’s UID and GID only if the hostPath volume is specified. Otherwise it can be set to any UID and GID. * Cluster roles can be set not only to * `edit` but also to * `view` * `admin` * `cluster-admin` More details on these roles can be found at:\ [https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles). * In addition, the user can be given access to custom CRDs, such as Postgres Operator, Jupyter Operator or Google Spark Operator ### 4.10.2 PodSecurityPolicy PodSecurityPolicy (PSP) was available in older versions of BCM. However, PSP support was completely removed in Kubernetes v1.25. BCM therefore now uses Kyverno (section [4.10.1](#4101-kyverno)) for an equivalent functionality. ## 4.11 Addition Of New Kubernetes Users BCM users can use Kubernetes by making them Kubernetes users. This means having Kubernetes configuration and access set up for them. This can be carried out using the `cm-kubernetes-setup` TUI utility, and choosing the `Add user` option (figure [4.3](#fig-kube-urwid-wizard-operations)). The utility then prompts for * a Kubernetes cluster * a user name * a namespace that the privileges are to be assigned to * a role for the user, with choices provided from: * `cluster-admin`: cluster-wide administrator * `admin`: administrator * `edit`: regular user * `view`: read-only user * a switch if the user is allowed to run as any user, including root, inside pods * a comma-separated list of paths that the user is able to mount to pods * the UIDs and GIDs for user processes in pods * a list of the Kubernetes operators that a user can use Based on the input, a YAML for the Kubernetes Permission Manager is generated. This in turn, creates a Helm chart with all the required roles, role bindings, and Kyverno rules. Creation of the user also triggers CMDaemon to create certificate and configuration files in the `~/.kube` directory ### 4.11.1 Adding Users Non-Interactively With `cm-kubernetes-setup` The user-management options are included in the complete `cm-kubernetes-setup -h` output in section [4.2.7](#427-kubernetes-setup-from-the-command-line). The user has to be a user that exists on the cluster already and available using PAM. If `--add-to-namespace` is specified, then the namespace has to exist on the Kubernetes cluster already. **Example** ```bash cm-kubernetes-setup --add-user john ``` The preceding example creates a user `john` for the default `john-restricted` namespace. It also assigns the `edit` role, and gives permission to run processes in the pod with the current UID/GIDs of the user. The ability to mount `~/john` as a hostPath is also provided. A way to assign any of the default Kubernetes user-facing roles is also provided by using `--role` key, as documented at [https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) The possible roles are: `view`, `edit`, `admin`, and `cluster-admin`. **Example** ```bash cm-kubernetes-setup --add-user john --role view ``` The preceding example creates a user `john` with `view` privileges only, for the default `john-restricted` namespace. **Example** ```bash cm-kubernetes-setup --add-user john --user-paths /home/john,/scratch --allow-all-uids \ --operators cm-jupyter-kernel-operator ``` The preceding example creates a user `john` with the following privileges: * `edit` privileges * able to mount `/home/john` and `/scratch` as hostPath volume, when the process runs with UID/GIDs taken from the PAM subsystem on the moment of creation * able to run as any user, including root (attempts to mount any hostPath volume will be rejected) * access to the Jupyter Kernel Operator, i.e. with access to the resource `kind: CmKubernetesOperatorPermissionsJupyterKernel` ## 4.12 Getting Information And Modifying Existing Kubernetes Users It is possible to edit user properties and permissions. `cm-kubernetes-setup` provides 2 ways of doing it: interactively or using CLI options. Modifying users can be done interactively by choosing `Modify User` in the `cm-kubernetes-setup` main menu. Guidance is then given on choosing the cluster, users, and on modifying permissions. Modifying users can also be done using CLI options, by specifying the `--modify-user` argument: **Example** ```bash cm-kubernetes-setup --add-user john --user-paths /home/john \ --allow-all-uids --operators cm-jupyter-kernel-operator ``` The default `john-restricted` namespace is created as user `john` is added, along with the settings specified by the other options. The following example then adds permission to mount the `/scratch` hostPath into pods, and gives access to the Postgres Operator: ```bash cm-kubernetes-setup --modify-user john --user-paths /home/john,/scratch --namespace john-restricted\ --allow-all-uids --operators cm-jupyter-kernel-operator,cm-kubernetes-postgresql-operator ``` Information about existing users can be found with: **Example** ```bash cm-kubernetes-setup --list-users ``` Permission for user `john` to operate in the `dev` namespace can be added with: **Example** ```bash kubectl create namespace dev cm-kubernetes-setup --modify-user john --namespace dev --add-to-namespace ``` Permission for the user `john` to operate in the `dev` namespace can be revoked with: **Example** ```bash kubectl create namespace dev cm-kubernetes-setup --modify-user john --namespace dev --remove-from-namespace ``` ## 4.13 List Of Resources Defined For Users These resources are rendered by the Permission Manager Operator, and can therefore be found inside Kubernetes. #### 4.13.0.1 The Role Bindings Deployed For Every User By Default By default, the role bindings deployed for the user `john` created in the preceding section are: * `ClusterRole/john-nodes` (in namespace `john-restricted`) * `ClusterRoleBinding/john-nodes` (in namespace `john-restricted`) User `john` is given read-only rights for the Nodes resource (for `kubectl get nodes`). #### 4.13.0.2 The Secure Namespace Related Resources The secure namespace for user `john` is: * `Namespace/john-restricted` The service account used by `john`: * `ServiceAccount/john` (in namespace `john-restricted`) This is found referenced, for example, in `john`’s `$HOME/.kube/config`. The PodSecurityPolicy that defines the user can run non-privileged pods, and use only ports above `1024`, and so on: * `PodSecurityPolicy/john-restricted` (in namespace `john-restricted`) More details on this can be found in section [4.10](#410-security-model), [here](#4102-podsecuritypolicy). This policy will only do something as soon as the `PodSecurityPolicy` Admission Controller is enabled in the API server. A PodSecurityPolicy that defines the user can run as root as well, but *without* hostPath volumes: * `PodSecurityPolicy/john-restricted-root` (in namespace `john-restricted`) To give the aforementioned privileges to `john`’s secure namespace, so that `john` can run workloads, execute `kubectl get all`, and more: * `Role/john-restricted` (in namespace `john-restricted`) * `RoleBinding/john-restricted` (in namespace `john-restricted`) The RoleBinding assigns it to the user `john` and ServiceAccount account for `john`. The upstream documentation at [https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin](https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin) has more details on this. The user `john` can be given the ability to use the PodSecurityPolicy defined earlier in his secure namespace, but also in other namespaces: * `ClusterRole/john-psp` (in namespace `john-restricted`) * `ClusterRoleBinding/john-psp` (in namespace `john-restricted`) The same ability can be given for the second `root but no hostPath` PodSecurityPolicy: * `ClusterRole/john-psp-root` (in namespace `john-restricted`) * `ClusterRoleBinding/john-psp-root` (in namespace `john-restricted`) If the Kyverno engine is installed then several policies are added: * `clusterpolicies.kyverno.io/john-*-drop-privs-w-hostpath*`: Policy to modify pod manifests to run process with specified UID/GID * `clusterpolicies.kyverno.io/john-*-limit-hostpath-vols`: Policy to deny pods if hostPath volumes do not match specified paths The full content of all the documents created for the user can be viewed by checking the generated Helm manifest: **Example** ```bash helm get manifest -n cm-permissions john-XXXXXX ``` ## 4.14 Kyverno Kyverno ([https://kyverno.io/](https://kyverno.io/)) is a policy engine designed for Kubernetes. With Kyverno, policies are managed as Kubernetes resources, and no new language is required to write policies. This allows the use of familiar tools such as kubectl, git, and kustomize to manage policies. Kyverno policies can validate, mutate, and generate Kubernetes resources, as well as ensure container image supply chain security. ### 4.14.1 Kyverno Installation Kyverno engine and Kyverno policy Helm charts can be installed as a part of `cm-kubernetes-setup`: ![Choosing Kyverno installation](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/37316dd5f2e1517f88be38ca8d4242bb6a08208537e524af07b5701f3331fbf5/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/kube-urwid-wizard-kyverno-install.png) *Figure 4.27: Choosing Kyverno installation* ![Kyverno high availability setup](/base-command-manager/product-manuals/_fern-img/abfa260e1a9656fc4e59d52acab0d2a6d746bedaa9a648cb022fd2bdbdd33d10.webp) *Figure 4.28: Kyverno high availability setup* ![Kyverno policy setup](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/e6c0414c2de982b79f3cd9fa29a68c7d99ef44b26c8cda14fac503e0fc072fb4/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/kube-urwid-wizard-kyverno-policies.png) *Figure 4.29: Kyverno policy setup* The installation adds 2 Helm charts in the namespace ’kyverno’: ```bash [root@basecm11 ~]# module load kubernetes/ [root@basecm11 ~]# helm list -n kyverno NAME NAMESPACE STATUS ... CHART APP VERSION kyverno kyverno deployed ... kyverno-v2.5.2 v1.7.2 kyverno-policies kyverno deployed ... kyverno-policies-v2.5.2 v1.7.2 ``` If the HA option is chosen, then the replica count value is set to 3. ```bash [root@basecm11 ~]# helm get values -n kyverno kyverno USER-SUPPLIED VALUES: replicaCount: 3 ``` This means that at any given time Kubernetes scheduler tries to run 3 pods at the same time: ```bash [root@basecm11 ~]# kubectl get pods -n kyverno NAME READY STATUS RESTARTS AGE kyverno-5bfb99b9c9-ddmmw 1/1 Running 0 1h kyverno-5bfb99b9c9-hgfsc 1/1 Running 0 1h kyverno-5bfb99b9c9-n67rv 1/1 Running 0 1h ``` ### 4.14.2 Kyverno Policies It is also recommended to install Kyverno policies in order to enforce Pod Security Standards [https://kyverno.io/policies/](https://kyverno.io/policies/). BCM configures Kyverno policies in ’enforce’ mode, adding service namespaces during `cm-kubernetes-setup` as exclusions (figure [4.30](#fig-kube-urwid-wizard-kyverno-policies-excluding-namespaces)). ![Kyverno policy: Defining the excluded namespaces](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/700142eca233976e66cbc1803c8e47eabbe49051e974d56680000d874ba3d8fd/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/kube-urwid-wizard-kyverno-excluded-namespaces.png) *Figure 4.30: Kyverno policy: Defining the excluded namespaces* The list of namespaces to be excluded from particular policies depends on features selected during installation: ```bash [root@basecm11 ~]# helm get values -n kyverno kyverno-policies USER-SUPPLIED VALUES: validationFailureAction: enforce policyExclude: disallow-host-namespaces: any: - resources: kinds: - Pod namespaces: - default - prometheus disallow-host-path: any: - resources: kinds: - Pod namespaces: - default - local-path-storage - '*-restricted' - prometheus - kube-system - gpu-operator disallow-host-ports: any: - resources: kinds: - Pod namespaces: - default - prometheus ``` In the preceding output, all namespaces that match the wildcard `*-restricted` are excluded from the policy named ’disallow-host-path’ ([https://kyverno.io/policies/pod-security/baseline/disallow-host-path/disallow-host-path/](https://kyverno.io/policies/pod-security/baseline/disallow-host-path/disallow-host-path/)). This means that, without additional restrictions, all pods in the user namespaces can mount any host path from an underlying node. To prevent that Kubernetes Permission Manager creates a Kyverno Cluster Policy for every newly-created user, and restricts the hostPath to only the home directory of the user: ```bash [root@basecm11 ~]# kubectl get clusterpolicies.kyverno.io | grep john john-n730sr0-drop-privs-w-hostpath false enforce true john-n730sr0-drop-privs-w-hostpath-containers false enforce true john-n730sr0-drop-privs-w-hostpath-initcontainers false enforce true john-n730sr0-limit-hostpath-vols false enforce true ``` ## 4.15 Kubernetes Permission Manager The Kubernetes permission manager is a custom operator based on Helm. It helps to manage user and system account permissions, roles, role bindings and pod security policies. The operator itself is packed and distributed as a Helm chart, so it can be installed during Kubernetes cluster creation using the `cm-kubernetes-setup` TUI. The Helm chart for the operator is located in `/cm/shared/apps/kubernetes-permissions-manager/current/helm`. The output to the following command shows if it is installed: ```bash [root@basecm11 ~]# module load kubernetes/ [root@basecm11 ~]# helm list -n cm NAME NAMESPACE STATUS ... CHART APP VERSION local-path-provisioner cm deployed ... local-path-provisioner-0.0.36 v0.0.36 permissions-manager cm deployed ... cm-kubernetes-permissions-manager-0.7.0 0.7.0 ``` The Helm chart of the operator includes custom resource definitions (CRD), and makes it possible for the administrator to manage resources using the `kubectl` tool: **Example** ```bash [root@basecm11 ~]# cat > permissions.yaml<kubernetes[default]]% get trusteddomains basecm11.example.com kubernetes kubernetes.default kubernetes.default.svc master localhost ``` In the preceding example, the FQDN of the cluster is `basecm11.example.com`. The cluster administrator managing their own cluster will have another FQDN, and not this FQDN. The port on which the Kubernetes API server is exposed externally is the Kubernetes API server proxy port. It can be retrieved from the Kubernetes cluster entity as follows: ```bash [basecm11->kubernetes[default]]% get kubernetesapiserverproxyport 10443 ``` This port is opened automatically by Shorewall on the head node when the exposing the Kubernetes API server to the external network option is selected during `cm-kubernetes-setup`, as mentioned earlier. ### 4.16.2 External Access To Kubernetes-hosted Applications And Services Applications and services running in Kubernetes are exposed externally through the Gateway components, which are fully covered in section [4.17](#417-kubernetes-gateway). For any service that needs its own subdomain (for example, `myapp.`, used as a hostname in a Gateway API route resource—such as an `HTTPRoute` or `TLSRoute`—attached to a Kubernetes `Gateway`), a corresponding DNS entry under the external FQDN must be added. ## 4.17 Kubernetes Gateway Starting with BCM 11.33.0, support for the Kubernetes Gateway API (currently version `1.5.x`) is introduced, provided by Kgateway and MetalLB. External traffic is exposed using the Gateway proxy on its listening port (`443` by default) on the head node(s), which forwards that traffic to the configured `Gateway`s served by Kgateway (the `bcm-gateway` `Gateway` by default, and any additional `Gateway`s that have been added). The [4.17](#417-kubernetes-gateway) BCM Kubernetes Gateway components are: *Kgateway* ([https://kgateway.dev/](https://kgateway.dev/)): the controller that fulfills the Kubernetes Gateway API ([https://gateway-api.sigs.k8s.io/](https://gateway-api.sigs.k8s.io/)) specification on the cluster, reconciling `Gateway`, `HTTPRoute`, `TLSRoute`, and related resources, and spinning up Kgateway proxy pods that actually handle traffic for each `Gateway`. *MetalLB* ([https://metallb.universe.tf/](https://metallb.universe.tf/)): provides a `LoadBalancer` `Service` IP address for each `Gateway` in the same way that a cloud provider’s load balancer would do for Kubernetes running in a managed cloud, and makes that IP address routable from outside the cluster. The *Gateway proxy*: an NGINX server running on the head node(s) and edge directors, provided by the `KubernetesIngressServerProxy` role (section [4.9.3](#493-the-kubernetesingressserverproxy-role)). It serves as the single entry point on the head node(s) for external clients, and forwards incoming traffic on its listening port to the appropriate `Gateway` based on the TLS Server Name Indication (SNI) hostname. ### 4.17.1 Gateway API The Kubernetes Gateway API is a Kubernetes-defined API specification, not a controller. The `Gateway`, `HTTPRoute`, `TLSRoute`, and related resources follow this specification, and are intended to be implementation-agnostic. BCM provides Gateway API 1.5.x. Kgateway is the implementation that BCM uses with the specification, and the resources that the applications and operators define according to the specification are, in most cases, portable across any conformant Gateway API implementation. The official documentation for the Kubernetes Gateway API explains the model in detail. ### 4.17.2 Kgateway With Kgateway the control plane and the data plane are split: * The Kgateway operator itself is the *control plane*. It watches the Gateway API resources (`Gateway`, `HTTPRoute`, etc.) and reconciles them. * The *data plane* is created on demand, per `Gateway`, in the form of one or more Kgateway proxy pods that actually handle incoming traffic. A consequence of this split is that only installing the Kgateway operator does not, by itself, expose any traffic: a `Gateway` (with a corresponding `GatewayClass`) must also be created so that the Kgateway operator can spin up the proxy pod(s) for it. ### 4.17.3 The `bcm-gateway` Gateway All BCM-shipped applications and operators expose their Kubernetes-hosted services through the configured `Gateway`s. The configured gateways are the `bcm-gateway` `Gateway` by default, and any additional `Gateway`s that have been added (section [4.17.5](#4175-creating-additional-gateways)) using Gateway API `HTTPRoute` or `TLSRoute` resources attached to those `Gateway`s. The configured routes and the `Gateway`s themselves can be inspected with `kubectl`: ```bash root@basecm11:~# kubectl get gateway -A NAMESPACE NAME CLASS ADDRESS PROGRAMMED AGE bcm-gateway bcm-gateway kgateway-ha2 10.141.255.253 True 9s root@basecm11:~# kubectl get httproute -A NAMESPACE NAME HOSTNAMES AGE bcm-gateway landingpage 17s loki loki 16s prometheus grafana 16s prometheus prometheus 16s root@basecm11:~# kubectl get tlsroute -A NAMESPACE NAME AGE default kubernetes 21s ``` The full YAML of an individual route can be inspected with `kubectl`. For example, the BCM-shipped `HTTPRoute` `landingpage` attached to the default `bcm-gateway` `Gateway` can be viewed with: ```bash root@basecm11:~# kubectl get httproute landingpage -n bcm-gateway -o yaml apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: creationTimestamp: "2026-04-21T10:17:37Z" generation: 1 name: landingpage namespace: bcm-gateway spec: parentRefs: - group: gateway.networking.k8s.io kind: Gateway name: bcm-gateway namespace: bcm-gateway rules: - backendRefs: - group: gateway.kgateway.dev kind: Backend name: landingpage weight: 1 matches: - path: type: PathPrefix value: / status: ... ``` The `spec.parentRefs` entry binds this `HTTPRoute` to the default `bcm-gateway` `Gateway` (named `bcm-gateway` in the `bcm-gateway` namespace), and the `spec.rules` entry forwards traffic matching the `/` path prefix to a Kgateway `Backend` named `landingpage`, which is configured as a static Backend pointing to `master:80`, where the BCM landing page service runs on the head node(s) and is served by `apache2` or `httpd`. Multiple backends and path-based routing can be configured through additional `HTTPRoute` resources attached to the same `bcm-gateway` `Gateway`. #### 4.17.3.1 Attaching Application Routes To The `bcm-gateway` Gateway Applications can attach their own `HTTPRoute` or `TLSRoute` resources to the `bcm-gateway` `Gateway` alongside the BCM-shipped routes shown in the preceding text. This is the simplest way to expose an application externally, as it avoids having to allocate and configure an extra IP address. For an application to be allowed to attach routes, its namespace must first be added under the relevant listener’s `allowedRoutes.namespaces.selector.matchExpressions` on the `bcm-gateway` `Gateway`. ### 4.17.4 MetalLB #### 4.17.4.1 Gateway IP Address Assignment And Routability Each `Gateway` is exposed using a `LoadBalancer` `Service` that requires its own IP address, which is unique across all `Gateway`s, and also unique across different Kubernetes clusters. In a bare-metal BCM deployment there is no cloud load balancer to allocate such IP addresses, so this responsibility is delegated to MetalLB ([https://metallb.universe.tf/](https://metallb.universe.tf/)), which is installed alongside Kgateway. MetalLB allocates an IP address from a configured pool to each `Gateway`’s `LoadBalancer` `Service` and makes that IP address routable from outside the cluster. MetalLB supports two modes for advertising the assigned IP addresses: * `L2` mode (Layer 2 / ARP): No special network equipment configuration is required, which makes `L2` mode the simplest to deploy. It provides basic failover-style HA, but not true load-balancing. The main limitation is that traffic can only reach the Gateway through nodes that are in the same L2 network as the client. When the Gateway proxy is enabled, this means that at least one Kubernetes node must be in the same L2 network as the head node(s), so incoming traffic can be forwarded through the head node(s) to reach Kubernetes services. Further details can be found at [https://metallb.universe.tf/concepts/layer2/](https://metallb.universe.tf/concepts/layer2/). * In `L2` mode, `cm-kubernetes-setup` suggests sensible default IP addresses on a best-effort basis by filtering out many known used or assigned IP addresses and IP ranges. These suggestions may still conflict with the environment into which the setup is carried out. The customer must ensure that each `Gateway` uses a unique, routable, IP address. * `BGP` mode: The MetalLB speakers establish BGP sessions with the network’s BGP-capable routers and advertise the `Service` IP addresses as routes. This provides a higher level of HA together with true load balancing, since traffic is distributed across nodes. This requires a BGP peer to be available upfront—typically a network router, switch, or similar device. Further details can be found at [https://metallb.universe.tf/concepts/bgp/](https://metallb.universe.tf/concepts/bgp/). During the initial `cm-kubernetes-setup` session, the wizard prompts for the MetalLB mode to use for Kubernetes Gateways (figure [4.32](#fig-kube-gateway-metallb-mode)). ![Wizard prompt: select the MetalLB mode for Kubernetes Gateways](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/24178f5391cd1205385ee2af34069d7b044d4af39015232b0a25925c8a825b9a/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/kube-gateway-metallb-mode.png) *Figure 4.32: Wizard prompt: select the MetalLB mode for Kubernetes Gateways* If BGP mode is selected, the wizard then prompts for the specific BGP settings (figure [4.33](#fig-kube-gateway-metallb-bgp)). ![Wizard prompt: MetalLB BGP settings for Kubernetes Gateways](/base-command-manager/product-manuals/_fern-img/bcf4721a782b241b05dea2b253eff1cb4d0051e4cf057db6747a97cc6352af57.webp) *Figure 4.33: Wizard prompt: MetalLB BGP settings for Kubernetes Gateways* The MetalLB mode is configured *per Kubernetes cluster* and therefore applies to all `Gateway`s in that cluster. The number of Kgateway proxy pods, by contrast, is determined *per `Gateway`* using the chosen `GatewayClass` (section [4.17.5.1](#41751-provided-gateway-classes)), so different applications within the same cluster can scale their data plane independently. The same `Gateway`s work with either MetalLB mode. #### 4.17.4.2 L2 Mode: Restricting Which Nodes Participate In The Gateway Data Plane The L2-mode limitation described in the preceding text means that some care is needed in deciding which Kubernetes nodes participate in the Gateway data plane. BCM manages this automatically when the Gateway proxy is enabled, but the cluster administrator should still validate whether other BCM compute nodes need to reach the Gateway from a different L2 network. BCM uses a dedicated labelset in `cmsh`, `exclude-lb`, backed by the standard upstream Kubernetes node label `node.kubernetes.io/exclude-from-external-load-balancers` ([https://kubernetes.io/docs/reference/labels-annotations-taints/](https://kubernetes.io/docs/reference/labels-annotations-taints/)). When this label is present on a node: * MetalLB does not use that node to advertise any `LoadBalancer` `Service` IP address. * Kgateway does not schedule any of its proxy pods on that node. The label thus keeps the Gateway IP address advertisement and the corresponding proxy pods co-located on the same set of nodes, which avoids the failure mode where an IP address is announced from a node that does not actually host the proxy pod. `cm-kubernetes-setup` manages the `exclude-lb` labelset as follows: * In `L2` mode with the Gateway proxy enabled, the labelset is applied to Kubernetes nodes that are not on a network shared with head nodes. This ensures that MetalLB advertisement and Kgateway proxy pod scheduling occur only on the other nodes that do share networks with the head node(s). * In `L2` mode with the Gateway proxy disabled, the labelset is not applied to any node. It is then up to the cluster administrator to optionally add nodes to the labelset based on node needs. * In `BGP` mode no nodes are added to the labelset either. This means that all Kubernetes nodes can participate in the Gateway data plane, and no further action is needed from the cluster administrator. Nodes can be added to or removed from the `exclude-lb` labelset in `cmsh`, for example: **Example** ```bash [root@basecm11 ~]# cmsh [basecm11]% kubernetes use default [basecm11->kubernetes[default]]% labelsets [basecm11->kubernetes[default]->labelsets]% use exclude-lb [basecm11->kubernetes[default]->labelsets[exclude-lb]]% append nodes node001 [basecm11->kubernetes*[default*]->labelsets*[exclude-lb*]]% commit [basecm11->kubernetes[default]->labelsets[exclude-lb]]% removefrom nodes node001 [basecm11->kubernetes*[default*]->labelsets*[exclude-lb*]]% commit ``` ### 4.17.5 Creating Additional Gateways Applications that need to be exposed externally can either attach their routes to the default `bcm-gateway` `Gateway` (section [4.17.3.1](#41731-attaching-application-routes-to-the-bcm-gateway-gateway)), or create an additional `Gateway` as described in the following subsections. #### 4.17.5.1 Provided Gateway Classes A `Gateway` references a `GatewayClass` that determines how the Gateway is implemented (which controller fulfills it, and with how many proxy pods). BCM provisions three Kgateway-backed `GatewayClass` resources, differing only in the number of Kgateway proxy pods that back the corresponding `Gateway`: * `kgateway`: 1 Kgateway proxy pod (suitable for non-HA deployments) * `kgateway-ha2`: 2 Kgateway proxy pods * `kgateway-ha3`: 3 Kgateway proxy pods The default `bcm-gateway` `Gateway` is provisioned against the `kgateway-ha2` class, giving it two Kgateway proxy pods. Additional `Gateway`s can reference any of the provided classes. The available classes can be listed with: ```bash bash$ kubectl get gatewayclass NAME CONTROLLER ACCEPTED AGE kgateway kgateway.dev/kgateway True 104m kgateway-ha2 kgateway.dev/kgateway True 104m kgateway-ha3 kgateway.dev/kgateway True 104m ``` #### 4.17.5.2 MetalLB Resources When an additional `Gateway` is created, the MetalLB resources need to be created as well, in order to assign and advertise an IP address for that `Gateway`. This subsection describes the manifests that are applied, mirroring what `cm-kubernetes-setup` creates for the default `bcm-gateway` `Gateway` (section [4.17.4.1](#41741-gateway-ip-address-assignment-and-routability)). For each additional `Gateway`, two MetalLB resources are required, both in the `metallb-system` namespace: * An `IPAddressPool` that holds the IP address (or range of IP addresses) that MetalLB can assign to the `Gateway`, and—optionally—restricts the pool to that one `Gateway` using `serviceAllocation`. * An *advertisement* resource that exposes the assigned IP address to the network. The value of `kind` sets the cluster’s MetalLB mode (section [4.17.4.1](#41741-gateway-ip-address-assignment-and-routability)): * `L2Advertisement` for L2 mode * `BGPAdvertisement` for BGP mode. In BGP mode the `BGPPeer` that BCM provisions in the `metallb-system` namespace is cluster-wide, and does not need to be duplicated for additional `Gateway`s. The new `BGPAdvertisement` reuses it automatically. The manifests that follow are a valid starting point, but additional `Gateway`s can deviate from this shape. For example: * a wider CIDR or a range in `spec.addresses` can be used, which means that the pool can serve multiple `LoadBalancer` `Service`s * omitting `serviceAllocation` makes the pool generally available, * services can be selected on different labels or namespaces. The upstream MetalLB documentation has the full schema of each resource. **Detecting the MetalLB mode:** Whether BGP mode is available for an additional `Gateway` can be checked by looking for the BCM-managed `BGPPeer` named `gateway` in the `metallb-system` namespace: **Example** ```bash root@basecm11:~# kubectl -n metallb-system get bgppeer gateway NAME AGE gateway 2d ``` If the `BGPPeer` exists, then the cluster was set up with BGP mode and an additional `Gateway` can reuse that peer by applying a `BGPAdvertisement`. BGP mode is generally the better choice when it is available—the trade-offs between the two modes are described in section [4.17.4.1](#41741-gateway-ip-address-assignment-and-routability). `L2` mode is always usable as well, regardless of whether the `BGPPeer` exists, by applying an `L2Advertisement` instead. **Naming convention:** The `gateway-` name prefix in the `metallb-system` namespace is reserved for MetalLB resources that `cm-kubernetes-setup` creates and manages itself, so the names of MetalLB resources for an additional `Gateway` must not start with that prefix. A common convention is to reuse the `Gateway`’s own name (or `-`) as the MetalLB resource name. **IP address selection:** The IP address, or range chosen for the additional `Gateway` must: * be unique across all `Gateway`s (and across other Kubernetes clusters that may share the same network), as described in section [4.17.4.1](#41741-gateway-ip-address-assignment-and-routability) * not overlap with any other `IPAddressPool` in the cluster, including the ones created by `cm-kubernetes-setup` * be routable from the intended clients, taking into account the L2-mode restriction on which nodes participate in the `Gateway` data plane (section [4.17.4.2](#41742-l2-mode-restricting-which-nodes-participate-in-the-gateway-data-plane)) **IPAddressPool:** Apply an `IPAddressPool` containing the chosen IP address as a `/32` entry, and use `serviceAllocation` to bind the pool to the specific additional `Gateway` using the `gateway.networking.k8s.io/gateway-name` label that Kgateway sets on the `LoadBalancer` `Service`: **Example** ```bash apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: namespace: metallb-system spec: addresses: - /32 autoAssign: true serviceAllocation: namespaces: - serviceSelectors: - matchExpressions: - key: gateway.networking.k8s.io/gateway-name operator: In values: - ``` The `serviceAllocation` stanza restricts this pool so that it is only used for the additional `Gateway`. It does not restrict the additional `Gateway` to this pool. If another `IPAddressPool` on the cluster is generally available (`autoAssign: true`, with no `serviceAllocation` restricting it away from this `Service`), then MetalLB can still assign an address from that pool instead of this dedicated one, bypassing the intended assignment. For `serviceAllocation` to have its intended effect, every other `IPAddressPool` that could serve the same `Service` must either set `autoAssign: false` or carry its own `serviceAllocation` restriction. This must be in place before the `Service` is created: an address already assigned to a `Service` is not reassigned automatically. **L2Advertisement (L2 mode):** In `L2` mode, apply an `L2Advertisement` that references the `IPAddressPool`: **Example** ```bash apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: name: namespace: metallb-system spec: ipAddressPools: - ``` **BGPAdvertisement (BGP mode):** In `BGP` mode, apply a `BGPAdvertisement` instead, again referencing the `IPAddressPool`. No additional `BGPPeer` is needed—the `BGPPeer` that `cm-kubernetes-setup` provisioned for the default `bcm-gateway` `Gateway` handles the BGP session with the upstream router: **Example** ```bash apiVersion: metallb.io/v1beta1 kind: BGPAdvertisement metadata: name: namespace: metallb-system spec: ipAddressPools: - ``` ### 4.17.6 Gateway Proxy The Gateway proxy is implemented by NGINX on the head node(s) and edge directors. Its configuration is generated and applied by `cm-kubernetes-setup`, which also opens the Gateway proxy port (`443` by default) in the firewall. The related parameters (such as `Ingress Proxy Enable` and `Ingress Proxy Listen Port`) live on the Kubernetes cluster entity in `cmsh`, and the corresponding head-node role is the `KubernetesIngressServerProxy` role (section [4.9.3](#493-the-kubernetesingressserverproxy-role)). #### 4.17.6.1 Architecture The Gateway proxy listens on the head node(s) on its listening port and uses TLS Server Name Indication (SNI) to decide where to forward each incoming connection: * Connections that present a SNI hostname matching a hostname declared by one of the configured `Gateway`s’ listeners are forwarded to that specific `Gateway`’s `LoadBalancer` `Service` IP address. * Connections without a SNI hostname (for example, when a client connects to the head node by IP address), and connections with a SNI hostname that does not match any of the configured `Gateway` listeners, are forwarded to the default `bcm-gateway` `Gateway`. On the `Gateway` side, the matching listener must be on port `443`. The backend port that the Gateway proxy connects to on each `Gateway`’s `LoadBalancer` `Service` is exposed by the `Ingress Proxy Backend Port` parameter on the Kubernetes cluster entity in `cmsh`, which defaults to `443`. This value is used internally by the Gateway proxy and is not intended to be customized. #### 4.17.6.2 Configuration During Setup During the initial `cm-kubernetes-setup` session, two wizard screens drive the Gateway proxy configuration: * A yes/no prompt (figure [4.34](#fig-kube-external-ingress)) asking whether to expose the Kubernetes `Gateway`s to the external network by enabling the Gateway proxy on the head node(s) and edge director(s). Selecting `yes` (recommended) enables it. Selecting `no` skips it; the Gateway proxy can still be configured later using the menu item described in section [4.17.6.3](#41763-the-configure-gateway-proxy-menu-item), or by following the manual steps in section [4.17.6.4](#41764-manual-configuration). ![Wizard prompt: expose the Kubernetes Gateways by enabling the Gateway proxy?](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/c0217d261f3d1d3e891f907a7f8263fe956a9237661a61cd25629da869c395be/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/kube-external-ingress.png) *Figure 4.34: Wizard prompt: expose the Kubernetes Gateways by enabling the Gateway proxy?* * A follow-up `Customize Gateway proxy configuration` screen (figure [4.35](#fig-kube-external-gateway-proxy-port)) asking on which TCP port the Gateway proxy should listen. The default is `443`, which matches the standard HTTPS port and is what the rest of this documentation assumes. ![Wizard prompt: Customize Gateway proxy configuration (Gateway proxy port)](/base-command-manager/product-manuals/_fern-img/9533296c70639ac7ac64bc97e9ccb3b4a893dcdd3bddf78efa46715430a23939.webp) *Figure 4.35: Wizard prompt: Customize Gateway proxy configuration (Gateway proxy port)* #### 4.17.6.3 The `Configure Gateway proxy` Menu Item After the initial setup, the same head-node NGINX configuration can be regenerated and re-applied at any time using the `Configure Gateway proxy to route traffic from head node(s) to Gateways` item, found under the `Gateway configuration` menu of `cm-kubernetes-setup`. Selecting that item inspects the `Gateway`s currently present on the cluster, collects their `LoadBalancer` `Service` IP addresses assigned by MetalLB and the hostnames declared on their listeners, and then refreshes the head-node NGINX Gateway proxy configuration accordingly so that the SNI-based routing described in section [4.17.6.1](#41761-architecture) reflects the current set of `Gateway`s. This item is therefore re-run whenever a new `Gateway` is added or an existing one is changed. #### 4.17.6.4 Manual Configuration If the wizard prompt about exposing the Kubernetes `Gateway`s by enabling the Gateway proxy was answered with `no` during the initial `cm-kubernetes-setup` session, the Gateway proxy can still be enabled afterwards by carrying out the following steps. **Step 1: ensure the Gateway proxy port is not already in use:** If the configured Gateway proxy port is already in use (for example, by the Gateway proxy of another Kubernetes cluster, or by the BCM landing page when served on that port using `apache2` or `httpd`), then that port must be changed. If the Gateway proxy port is the default `443`, then for Ubuntu-based systems this means commenting out `Listen 443` in `/etc/apache2/ports.conf`. For RHEL-based systems that line is located at `/etc/httpd/conf.d/ssl.conf`. If a different Gateway proxy port was configured, then the corresponding `Listen` directive for that port must be adjusted instead. The web service should then be restarted (`apache2` for Ubuntu, `httpd` for RHEL and Rocky). **Step 2: enable the Gateway proxy in the Kubernetes cluster entity:** On the Kubernetes cluster entity in `cmsh`, `Ingress Proxy Enable` should be set to `yes`, and `Ingress Proxy Listen Port` should be set to the desired Gateway proxy port. The parameter `gateway_proxy_enabled` must also be set. It is an additional entry for BCM 11 versions later than [point release version](/admin/softwareimagemanagement#912-bcm-package-point-release-versions-and-the-cm-package-release-info-command) 11.33.0, required as the Gateway proxy replaces the legacy Ingress. The example below uses the default port `443`. **Example** ```bash root@basecm11:~# cmsh [basecm11]% kubernetes use default [basecm11->kubernetes[default]]% set -e "gateway_proxy_enabled" true [basecm11->kubernetes*[default*]]% set ingressproxyenable yes [basecm11->kubernetes*[default*]]% set ingressproxylistenport 443 [basecm11->kubernetes*[default*]]% set ingressproxybackendport 443 [basecm11->kubernetes*[default*]]% commit [basecm11->kubernetes[default]]% ``` **Step 3: assign the role to the head nodes:** This can be done as follows in `cmsh`, and should be repeated for the passive head node in case of an HA BCM cluster. **Example** ```bash root@basecm11:~# cmsh [basecm11]% device use master [basecm11->device[basecm11]]% roles [basecm11->device[basecm11]->roles]% list Name (key) -------------------------------------------------- Kubernetes::ApiServerProxy [overlay:kube-default-worker] generic::containerd [overlay:kube-default-worker] kubelet backup boot firewall headnode monitoring nginx provisioning storage [basecm11->device[basecm11]->roles]% assign Kubernetes::IngressServerProxy Creating new role 'Kubernetes::IngressServerProxy' ... succeeded. [basecm11->device*[basecm11*]->roles*[Kubernetes::IngressServerProxy*]]% show Parameter Value -------------------------------- ------------------------------------------------ Name Kubernetes::IngressServerProxy Revision Type KubernetesIngressServerProxyRole Add services yes Worker connections 1024 Send file yes TCP no push yes TCP no delay yes Keep alive timeout 1m 5s Types hash max size 2048 Listen Port 443 Kubernetes Ingress Server Port 0 [basecm11->device*[basecm11*]->roles*[Kubernetes::IngressServerProxy*]]% commit [basecm11->device[basecm11]->roles[Kubernetes::IngressServerProxy]]% Thu Oct 31 16:29:51 2024 [notice] basecm11: Service nginx was reloaded ``` As the preceding session shows, assigning the role immediately affects the `/etc/nginx/nginx.conf` file and the `nginx` service, but the actual Gateway proxy routing configuration is generated and applied in Step 4. **Step 4: run the `Configure Gateway proxy` menu item:** This is described in section [4.17.6.3](#41763-the-configure-gateway-proxy-menu-item). **Step 5: open the Gateway proxy port in the firewall:** The port in the default firewall, Shorewall ([section 7.2 of the *Installation Manual*](/installation/thirdparty#72-shorewall)), is managed using the `firewall` role in `cmsh`. Only the Gateway proxy port needs to be opened, in addition to the Kubernetes API server proxy port. The example below uses the default Gateway proxy port `443`; if a different port was configured in Step 2, use that value instead. ```bash root@basecm11:~# cmsh [basecm11]% device use master [basecm11->device[basecm11]]% roles [basecm11->device[basecm11]->roles]% use firewall [basecm11->device[basecm11]->roles[firewall]]% openports [basecm11->device[basecm11]->roles[firewall]->openports]% list -v Index Action Network Port Count Destination Protocol Address Description ------ --------- ------------ ------ ------ ------------ --------- ------------ ------------------------ 0 ACCEPT net 10443 1 fw TCP 0.0.0.0/0 Kubernetes 'default' API Server proxy [basecm11->device[basecm11]->roles[firewall]->openports]% add ACCEPT net 443 tcp fw ok. [basecm11->device*[basecm11*]->roles*[firewall*]->openports*[1]]% commit [basecm11->device[basecm11]->roles[firewall]->openports[1]]% list -v Index Action Network Port Count Destination Protocol Address Description ------ --------- ------------ ------ ------ ------------ --------- ------------ ------------------------ 0 ACCEPT net 10443 1 fw TCP 0.0.0.0/0 Kubernetes 'default' API Server proxy 1 ACCEPT net 443 1 fw TCP 0.0.0.0/0 [basecm11->device[basecm11]->roles[firewall]->openports[1]]% Thu Oct 31 16:35:19 2024 [notice] basecm11: Service shorewall was restarted ``` The pre-existing `10443` entry shown above with the description `Kubernetes 'default' API Server proxy` is automatically opened by the `KubernetesApiServerProxy` role when exposing the Kubernetes API server to the external network was selected during `cm-kubernetes-setup`. In the example above, the newly-added port `443` entry has no description because it was added manually; it can be commented or otherwise tagged later if needed. As shown above, the change is reflected immediately in the `/etc/shorewall/rules` file and `shorewall` service. ### 4.17.7 Configuring The Gateway TLS Certificate Kubernetes applications that are exposed externally on BCM use HTTPS using the Gateway proxy port, served by the configured `Gateway`s (the `bcm-gateway` `Gateway` by default, and any additional `Gateway`s that have been added). The TLS certificate that the `bcm-gateway` `Gateway` presents is the certificate that needs to be configured for external access. For the `bcm-gateway` `Gateway`, it is assumed that * the cluster administrator has provided the cluster with a domain name, for example `my-cluster.nvidia.local` * there is a DNS entry present that makes this domain name resolve to the cluster IP address * a matching `bcm-gateway.key` and `bcm-gateway.crt` TLS certificate key pair file has been provided #### 4.17.7.1 Gateway Self-signed Certificate For testing purposes it is possible to create a self-signed certificate with a private certificate authority (CA) owned by the cluster administrator, and configure the `bcm-gateway` `Gateway` to present it. That is, using that instead of a trusted (third-party) CA. However the effort needed to configure this means that the procedure is not generally recommended. Reasons to avoid using a self-signed certificate are: * each user needs the self-signed CA certificate pairs working on their system between the appropriate applications * each user also needs to override a warning about the untrusted certificate by accepting the self-signed certificate on browsers such as Chrome. * it is very easy to run into subtle issues later on that are hard to uncover That said, appendix [A](/containerization/selfsigned) has instructions on setting up a self-signed certificate for testing purposes. The steps that are needed to also make the certificates trusted on any given subsystem or application such as the web browser, are outside the scope of the manuals and BCM support. It is assumed that this has already been set up by the organization. #### 4.17.7.2 Gateway Trusted CA Certificate In case no self-signed certificate is to be used—for example if there is already a trusted CA certificate to use for this purpose—then another Kubernetes wizard session can be run from the active head node to configure the `bcm-gateway` `Gateway` TLS certificate. The cluster administrator carries out the following procedure: * uses SSH to get into the active head node * copies certificate pair files, `bcm-gateway.crt` and `bcm-gateway.key`, over to the head node (for example: to the `/root` directory) * invokes `cm-kubernetes-setup`, opens the `Gateway configuration` menu, and then chooses (figure [4.36](#fig-setup-ingress-https)): `Configure TLS Certificate for Gateway` * answers `yes` when the wizard prompts: `Do you want to configure an existing, properly signed certificate pair?` * sets paths to the certificate pair files; the wizard then stores the certificate pair as the Kubernetes TLS `Secret` `gateway-bcm-gateway-cert` in the `bcm-gateway` namespace, and references that `Secret` from the `bcm-gateway` `Gateway` listener * confirms that the HTTPS certificate is working correctly, using the SAN check (described shortly) ![The Configure TLS Certificate for Gateway item under the Gateway configuration menu](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/0776e7ac83d3381ddb0c8aa522aeee64176f44c2c567576c0ba3ac24552d8a63/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/setup-ingress-https.png) *Figure 4.36: The Configure TLS Certificate for Gateway item under the Gateway configuration menu* **The SAN check:** The following command can be used to inspect the SAN (Subject Alternative Name) part of the certificate currently presented by the `bcm-gateway` `Gateway` using the Gateway proxy port: **Example** ```bash [root@basecm11 ~]# openssl s_client -connect localhost:443 < /dev/null 2>/dev/null | \ openssl x509 -noout -text | grep -A1 'Subject Alternative Name' X509v3 Subject Alternative Name: DNS:my-cluster.nvidia.local, DNS:*.apps.my-cluster.nvidia.local, DNS:master.cm.cluster ``` ### 4.17.8 Migrating An Existing Kubernetes Cluster From Ingress To Gateway For BCM clusters that were originally deployed with the legacy Ingress NGINX Controller (BCM versions before 11.33.0), the following procedure migrates the cluster to Kubernetes Gateway without recreating the cluster. **Prerequisites:** Before starting the migration: * All applications and operators that currently rely on Kubernetes `Ingress` resources should either already support, or be ready to be migrated to, the Kubernetes Gateway API. * If the legacy Ingress proxy is currently used for external access, a maintenance window should be planned, since that external traffic will be interrupted during the migration. **Step 1: deploy the Gateway components:** The `cm-kubernetes-setup` utility is started on the active head node. From its `Gateway configuration` menu, the `Install Gateway components and replacements for legacy BCM Ingresses` item is selected. The exact actions performed by this step are summarized by the wizard confirmation dialog. This step is intentionally non-destructive in the sense that it does *not* uninstall the Ingress NGINX Controller, and it does *not* delete any existing `Ingress` resources. However, it does reconfigure the legacy Ingress proxy on the head node(s) to behave as the Gateway proxy, so external traffic through the legacy Ingress path no longer works after this step. **Step 2: adjust external load balancer configuration:** If any external load balancer currently sends traffic to the legacy Ingress proxy ports (`30080` and/or `30443`, with `30443` the default HTTPS port), it should be updated to use the Gateway proxy port instead. This ensures that external traffic reaches the head-node Gateway proxy rather than the legacy Ingress proxy path while the Gateway-based path is being validated. **Step 3: validate traffic for the BCM-shipped services:** With the Gateway components running, the BCM-shipped services that were previously fronted by Ingress—the landing page, the Kubernetes API at the configured external FQDN, Prometheus, Grafana, and Loki, where applicable—should be verified through the new `bcm-gateway` `Gateway` using the Gateway proxy port. The `Gateway` and its `HTTPRoute`/`TLSRoute` resources can be inspected with: * `kubectl get gateway -A` * `kubectl get httproute -A` * `kubectl get tlsroute -A` **Step 4: migrate non-BCM `Ingress` resources to Gateway API:** Any non-BCM-provided application that still relies on Kubernetes `Ingress` resources is now migrated. For each such `Ingress`: * an equivalent `HTTPRoute` (or `TLSRoute` for TLS pass-through) is authored, targeting the same backend `Service` * the new route is attached to a `Gateway`—either to the default `bcm-gateway` `Gateway` (section [4.17.3.1](#41731-attaching-application-routes-to-the-bcm-gateway-gateway)), or to a dedicated `Gateway` created in the application’s namespace (section [4.17.5](#4175-creating-additional-gateways)) * the original `Ingress` resource is left in place for the time being, so traffic keeps flowing through the legacy path while the new path is being validated **Step 5: refresh the Gateway proxy for non-BCM `Gateway`s:** After non-BCM `Gateway`s have been created on the cluster * `cm-kubernetes-setup` is started again * the `Gateway configuration` menu is opened * the `Configure Gateway proxy to route traffic from head node(s) to Gateways` item is selected (section [4.17.6.3](#41763-the-configure-gateway-proxy-menu-item)). This refreshes the head-node NGINX Gateway proxy so that traffic for the new `Gateway`s is forwarded from the head node(s) to their respective `LoadBalancer` `Service` IP addresses assigned by MetalLB, using the SNI-based routing described in section [4.17.6.1](#41761-architecture). Connections that do not match any non-BCM `Gateway` listener (including connections without a SNI hostname) continue to be forwarded to the default `bcm-gateway` `Gateway`. **Step 6: validate traffic for the non-BCM services:** Each migrated application is verified through its `HTTPRoute` or `TLSRoute` on the `bcm-gateway` `Gateway` (or on its dedicated `Gateway`). At this point, both the BCM-shipped services and the non-BCM services are reachable through the Gateway-based path, while the legacy Ingress NGINX Controller `NodePort`s are still in place as a fallback. **Step 7: remove the legacy Ingress NGINX controller:** Once all traffic has been confirmed on the Gateway-based path, the legacy controller is removed. To carry this out, `cm-kubernetes-setup` is started again, the `Gateway configuration` menu is opened and the `Remove Ingress NGINX Controller (ingress-nginx) and legacy BCM Ingresses` item is selected. The exact actions performed by the removal are summarized by the wizard confirmation dialog. This step does *not* delete any `Ingress` resources that were not provided by BCM. Any non-BCM `Ingress` resources that have not been removed yet remain present on the cluster. **Step 8: complete the cleanup:** Once the previous steps have all completed successfully, any non-BCM `Ingress` resources that are still present on the cluster can be safely deleted by the cluster administrator. For example: ```bash root@basecm11:~# kubectl get ingress -A root@basecm11:~# kubectl delete ingress -n ``` After this, the cluster is fully migrated to the Kubernetes Gateway API, and the legacy Ingress NGINX Controller is no longer in use on the cluster. ## 4.18 Networking Model Kubernetes expects all pods to have unique IP addresses, which are reachable from within the cluster. This can be implemented in several ways, including adding pod network interfaces to a network bridge created on each host, or by using 3rd party tools to manage pod virtual networks. With BCM the default pod network provider is Calico ([https://www.projectcalico.org/](https://www.projectcalico.org/)). Calico uses the Border Gateway Protocol (BGP) to distribute routes for every Kubernetes pod. This allows the Kubernetes cluster to be integrated without the need for overlays (IP-in-IP). Calico is particularly suitable for large Kubernetes deployments on bare metal, or in private clouds. This is because for larger deployments the performance and complexity costs of overlay networks can become significant. ## 4.19 Kubernetes Monitoring When `cm-kubernetes-setup` is run, it configures the following Kubernetes-related health checks: 1. `KubernetesChildNode`: checks if all the expected agents and services are up and running for active nodes 2. `KubernetesComponentsStatus`: checks if all the daemons running on a node are healthy 3. `KubernetesNodesStatus`: checks if Kubernetes nodes have a status of `Ready` 4. `KubernetesPodsStatus`: checks if all the pods are in one of these states: `Running`, `Succeeded`, or `Pending` ## 4.20 Local Path Storage Class For storage, instead of creating Kubernetes PersistentVolumes every time, a modern and practical way is to use the StorageClass feature. Further documentation on StorageClass is available at: * [http://blog.kubernetes.io/2016/10/dynamic-provisioning-and-storage-in-kubernetes.html](http://blog.kubernetes.io/2016/10/dynamic-provisioning-and-storage-in-kubernetes.html) * [https://kubernetes.io/docs/concepts/storage/persistent-volumes/#storageclasses](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#storageclasses) As a part of initial installation it is possible to choose a Local Path Storage class to utilize the shared storage mounted on every node of the Kubernetes cluster. The two wizard screens that do this, the storage class screen and the local path storage pool screen, are described in the walkthrough in section [4.2.8](#428-kubernetes-setup-from-a-tui-session). Possible options include any POSIX shared filesystems, such as NFS, BeeGFS, LustreFS, etc. If a class is enabled, then the installation wizard asks for a path for where Kubernetes physical volumes (PV) will be physically located. This path should be located on a shared filesystem accessible from all nodes. After installation, the storage class can be seen to be available with: ```bash # kubectl get storageclasses.storage.k8s.io NAME PROVISIONER ... VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION local-path (default) cluster.local/local-path-provisioner ... WaitForFirstConsumer true ``` Users of the cluster can then freely create persistent volume claims (PVC) resources and use them in running pods. ## 4.21 Integration With Harbor In order to spawn pods that use images from the Harbor registry, a secret must first be created with the credentials: ```bash [root@basecm11 ~]# kubectl create secret docker-registry myregistrykey \ --docker-server=node001:9443 --docker-username=admin --docker-password=Harbor12345 ``` The secret must then be referenced from the pod: ```bash apiVersion: v1 kind: Pod metadata: name: foo spec: containers: - name: foo image: node001:9443/library/nginx imagePullSecrets: - name: myregistrykey ``` Further information on this is available at [https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod](https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod). ## 4.22 Kubernetes Upgrades This section assumes that an upgrade to a BCM Kubernetes is being considered. This means that it is assumed to have been set up with `cm-kubernetes-setup`. Upgrades to Kubernetes can be done by following the principles behind the official upstream documentation at [https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/). The following points must be kept in mind when upgrading a BCM Kubernetes cluster: * Nodes that are part of the cluster may be provisioned through their software image. * The Kubernetes version in BCM must manually be updated to match the version of the upgrade. * Air-gapped upgrades follow the principles of this section, but have some extra considerations. Air-gapped upgrades are discussed in Appendix [C](/containerization/kubernetesairgapupgrade), and the instructions there can be followed after reading this section (section [4.22](#422-kubernetes-upgrades)) first. * Only some Kubernetes versions are supported. A list of supported available versions can be retrieved with: **Example** ```bash [root@basecm11 ~]# cm-kubernetes-setup --list-versions 1.36 1.35 (NVIDIA AI Enterprise certified) 1.34 (NVIDIA AI Enterprise certified) ``` In the example that follows, a BCM cluster is upgraded from Kubernetes 1.31 to Kubernetes 1.32. [A](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/)t the time of writing of this section (March 2026), the URL for the upstream documentation redirects the readers to the upgrade instructions for, among others, 1.33 to 1.34. ### 4.22.1 Upgrade Prerequisites Kubernetes is not a single entity for upgrade purposes. There is a single driver, the `kubeadm` program, which is used to plan and execute the upgrade, but it does it only for itself and a handful of components central to Kubernetes’ functionality. Kubernetes does not have ways to manage its third-party components with `kubeadm`’s upgrade planning and execution. This means that multiple Kubernetes components that are essential to Kubernetes are not involved in the upgrade process. If these components are neglected during an upgrade, then they can either prevent successful completion of the upgrade, or can themselves fail as a result of the upgrade. Some noteworthy components that are excluded by upgrades are the etcd database, CNIs, CSIs, container runtimes, as well as individual pods not managed by higher-level entities such as `Deployment`, `ReplicaSet` and so on. The Kubernetes documentation at [https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets) describes the workings of the eviction process in greater detail. Kubernetes upgrades assume that various entities, in particular, pods, can migrate away from the node being upgraded. This assumption may not be true because of various constraints on pod scheduling. The constraints need to be satisfied or worked around in order for migration to work. Particular care has to be taken when dealing with persistent volumes or physical resources such as GPUs that cannot migrate, but are needed for a pod to be deployed. #### 4.22.1.1 Hot And Cold Upgrades * A hot upgrade is one that allows the system to stay online at all times. * A cold upgrade is one that requires at least a brief service interruption. Since Kubernetes is aimed at providing 100% system uptime, the upgrade process is designed to be hot. The cluster administrator carrying out the hot upgrade does however need to make an effort for the instances where Kubernetes cannot ensure service uptime. **Upgrading components that cannot be stopped in a regular manner:** One common problem with hot upgrades is the need to migrate or to stop `DaemonSet` pods. These are intended to run on all Kubernetes nodes, even those being drained. Kubernetes can be told to ignore `DaemonSet` pods by using the `--ignore-daemonsets` option. The problem with doing that is that it avoids upgrading those pods, and also typically avoids upgrading the third-party component that created them. To ensure upgrade completion, the administrator has to carry out a separate upgrade action, appropriate for each component ignored in this way. **Upgrading gradually:** Kubernetes upgrades come without any tools to perform upgrades at scale. The available tools can only upgrade a single node at a time. This means that, especially for larger clusters, the administrator needs to carry out the upgrade gradually, and has to plan and allocate extra capacity to accommodate pods migrated from nodes being upgraded. Special precautions need to be taken when dealing with deployments which require larger replica counts. Administrators may need to perform special procedures. Third-party components in particular may require the simultaneous presence of multiple entities across multiple nodes, in order to be able to upgrade those components without a service interruption. Finally, it is possible that some components are simply not designed for hot upgrades. In that case, administrators have to remove those components from the system, and install the newer version, with the downtime depending on each individual case. ### 4.22.2 Example RHEL9 Cluster A cluster with 3 nodes that run the control plane, and based on RHEL9, is considered as a reference example. One control plane node is the single head node, and the other two control plane nodes are two regular (compute) nodes. There are four additional worker-only nodes, and all the non-head nodes share the same category and software image. ```bash [root@basecm11 ~]# module load kubernetes [root@basecm11 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION node001 Ready control-plane,worker 8m25s v1.27.13 node002 Ready control-plane,worker 8m25s v1.27.13 node003 Ready worker 8m43s v1.27.13 node004 Ready worker 8m41s v1.27.13 node005 Ready worker 8m42s v1.27.13 node006 Ready worker 8m41s v1.27.13 basecm11 Ready control-plane 9m22s v1.27.13 ``` ### 4.22.3 Before Starting The Upgrade The upstream instructions at [https://v1-32.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/#before-you-begin](https://v1-32.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/#before-you-begin), on preparing for the upgrade should be read. A node that is to be upgraded must be drained. ### 4.22.4 Updating The First Control Plane Node The first control plane node is upgraded in this section. Instead of going into much detail on upgrades—the official documentation at [https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/) does a good job—an example covering an upgrade to a Rocky cluster running Kubernetes version 1.27 is shown that uses the official documentation for guidance. Instructions for Ubuntu differ slightly but can be carried out in a similar way. Typically the control plane is on the head node, so that the cluster administrator starts with updating the control plane being run by the head node. However, for completeness, the two subsections that follow describe updating the control plane if starting with: * either the head node * or a compute node #### 4.22.4.1 Upgrading A Control Plane Node Starting With The Head Node To upgrade the control plane when the head node is the first node to be upgraded, the repository entry for picking up packages from the Kubernetes repository should have the new version set. **Example** ```bash [root@basecm11 ~]# grep -ir kubernetes /etc/yum.repos.d/* #is anything already there? /etc/yum.repos.d/kubernetes.repo:[kubernetes] /etc/yum.repos.d/kubernetes.repo:name=Kubernetes /etc/yum.repos.d/kubernetes.repo:exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni [root@basecm11 ~]# grep v /etc/yum.repos.d/kubernetes.repo #what version is fetched? baseurl=https://pkgs.k8s.io/core:/stable:/v1.31/rpm/ gpgkey=https://pkgs.k8s.io/core:/stable:/v1.31/rpm/repodata/repomd.xml.key ``` The `grep` outputs confirm there is a repository file, and that it is set to pick up Kubernetes version 1.31. The .31 indicates what is called the minor version in Kubernetes. To instead pick up version 1.32, the upstream documentation instructs changing the minor version in the repository file to the next minor version upgrade number, so here from `v1.31` to `v1.32`. Skipping minor versions for upgrades is unsupported. An additional check to confirm the running version is: ```bash [root@basecm11 ~]# kubeadm version kubeadm version: &version.Info{Major:"1", Minor:"31", GitVersion:"v1.31.10", GitCommit: ... ``` The minor version in `kubernetes.repo` is changed from `v1.31` to `v1.32` using a text editor. The installation for a new `kubeadm` can then be carried out with: ```bash [root@basecm11 ~]# yum install -y kubeadm --disableexcludes=kubernetes ... [root@basecm11 ~]# kubeadm version #should now show we are at version 1.32 kubeadm version: &version.Info{Major:"1", Minor:"32", GitVersion:"v1.32.6", ... ``` The upgrade plan can be inspected: ```bash [root@basecm11 ~]# kubeadm upgrade plan ... You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.32.6 ... ``` In this case it gives a recommended upgrade command to run on the head node: The upgrade is now made to version `1.32.6`, as suggested. The curious cluster administrator can view the list of available versions can be viewed with the `--showduplicates` option to `yum`: **Example** ```bash [root@basecm11 ~]# yum list --showduplicates kubeadm --disableexcludes=kubernetes ... Installed Packages kubeadm.x86_64 1.32.6-150500.1.1 @kubernetes Available Packages kubeadm.aarch64 1.32.6-150500.1.1 kubernetes kubeadm.ppc64le 1.32.6-150500.1.1 kubernetes kubeadm.s390x 1.32.6-150500.1.1 kubernetes kubeadm.src 1.32.6-150500.1.1 kubernetes kubeadm.x86_64 1.32.6-150500.1.1 kubernetes ... ``` Most cluster administrators stick to the recommendation. ```bash [root@basecm11 ~]# kubeadm upgrade apply v1.32.6 ... [upgrade/version] You have chosen to change the cluster version to "v1.32.6" [upgrade/versions] Cluster version: v1.31.10 [upgrade/versions] kubeadm version: v1.32.6 [upgrade] Are you sure you want to proceed? [y/N]: y ... [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.32.6". Enjoy! [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so. ``` The host should now be drained, if it has not already been drained: ```bash [root@basecm11 ~]# kubectl drain $(hostname) --ignore-daemonsets ``` As suggested, the kubelets are now upgraded. Updating the `kubectl` version at the same time is also a good idea: ```bash [root@basecm11 ~]# yum install -y kubelet kubectl --disableexcludes=kubernetes ``` The kubelets can be restarted so that the new versions run: ```bash [root@basecm11 ~]# kubelet --version Kubernetes v1.32.6 [root@basecm11 ~]# systemctl status kubelet ... Active: active (running) since ...; 21h ago #need to reload kubelets to new versions ... [root@basecm11 ~]# systemctl daemon-reload [root@basecm11 ~]# systemctl restart kubelet && systemctl status kubelet | grep Active Active: active (running) since ... 14ms ago ``` This ends the procedure of upgrading the control plane by first starting on a head node. Upgrading the control plane by first starting with a regular node instead is described next. #### 4.22.4.2 Upgrading The Control Plane Node Starting With A Compute Node This section describes the list of commands for upgrading the control plane when it is the first control plane node to be upgraded. For example `node001`, rather than a head node. The appropriate software image and related information for the node must first be found. For `node001` this can be done with: ```bash [root@basecm11 ~]# cmsh [basecm11]% device use node001 [basecm11->device[node001]]% get softwareimage default-image (category:default) [basecm11->device[node001]]% softwareimage [basecm11->softwareimage]% use default-image [basecm11->softwareimage[default-image]]% get path /cm/images/default-image ``` The software image is then entered and modified using a chroot jail. The repository file of the image, with an absolute path outside the chroot jail of `/cm/images/default-image/etc/yum.repos.d/kubernetes.repo`, has its minor version changed appropriately: ```bash [root@basecm11 ~]# cm-chroot-sw-img /cm/images/default-image [root@default-image /]# grep v /etc/yum.repos.d/kubernetes.repo #what version would be fetched? /etc/yum.repos.d/kubernetes.repo:baseurl=https://pkgs.k8s.io/core:/stable:/v1.31/rpm/ /etc/yum.repos.d/kubernetes.repo:gpgkey=https://pkgs.k8s.io/core:/stable:/v1.31/rpm/repodata/repomd.xml.key [root@default-image /]# vi /etc/yum.repos.d/kubernetes.repo #modify the point version in the image ...modifications done... [root@default-image /]# grep v /etc/yum.repos.d/kubernetes.repo #will fetch this version now /etc/yum.repos.d/kubernetes.repo:baseurl=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/ /etc/yum.repos.d/kubernetes.repo:gpgkey=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/repodata/repomd.xml.key ``` The package versions available for installation in the image can be checked with: ```bash [root@default-image /]# yum info kubeadm kubelet kubectl --disableexcludes=kubernetes ``` In this case versions 1.32.6 are seen to be available. The `kubeadm`, `kubelet` and `kubectl` packages are upgraded in advance, and the chroot environment is then exited: ```bash [root@default-image /]# yum install -y kubeadm kubelet kubectl --disableexcludes=kubernetes ... [root@default-image /]# exit #leave the chroot ``` The following set of commands then carries out the update on `node001`: ```bash [root@basecm11 ~]# export host=node001 [root@basecm11 ~]# cmsh -c "device use $host; imageupdate -w --wait" #update running image on node [root@basecm11 ~]# ssh $host kubeadm version #what version to fetch now [root@basecm11 ~]# ssh $host kubeadm upgrade plan #get suggested upgrade plan [root@basecm11 ~]# ssh $host kubeadm upgrade apply v1.32.6 #apply the upgrade [root@basecm11 ~]# kubectl drain $host --ignore-daemonsets #drain the host [root@basecm11 ~]# ssh $host sudo systemctl daemon-reload #reload systemd manager config [root@basecm11 ~]# ssh $host sudo systemctl restart kubelet #restart kubelet [root@basecm11 ~]# kubectl uncordon $host #new pods may now run on node ``` ### 4.22.5 Updating Subsequent Control Plane Nodes If there is only one control plane node, then this section can be skipped. If there is more than one control plane node, and if the first control plane node is a head node that has been updated, then the remaining control plane nodes should be updated. The `kubectl` command can be used to orient the cluster administrator, by displaying the state of the upgrade, roles, and Kubernetes versions per node: **Example** ```bash [root@basecm11 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION node001 Ready control-plane,worker 26m v1.31.10 node002 Ready control-plane,worker 26m v1.31.10 node003 Ready worker 26m v1.31.10 node004 Ready worker 26m v1.31.10 node005 Ready worker 26m v1.31.10 node006 Ready worker 26m v1.31.10 basecm11 Ready control-plane 27m v1.32.6 ``` The remaining control plane nodes can now be updated. In the reference example case, these control plane nodes are `node001` and `node002` as seen in the preceding example. The commands in the section on **Upgrading The Control Plane Node Starting With A Compute Node** ([here](#42242-upgrading-the-control-plane-node-starting-with-a-compute-node)) should be followed, but with one small difference. Instead of running: `kubeadm upgrade apply v1.32.6` the command `kubeadm upgrade node` is run. The software image should also be prepared with the new packages before exiting the chroot jail. After exiting the chroot jail the remaining control plane nodes can then be updated with the following set of commands: **Example** ```bash [root@basecm11 ~]# export host=node001 [root@basecm11 ~]# cmsh -c "device use $host; imageupdate -w --wait" [root@basecm11 ~]# ssh $host kubeadm version [root@basecm11 ~]# ssh $host kubeadm upgrade plan [root@basecm11 ~]# ssh $host kubeadm upgrade node # I'm special! (apply not used here) [root@basecm11 ~]# kubectl drain $host --ignore-daemonsets [root@basecm11 ~]# ssh $host sudo systemctl daemon-reload [root@basecm11 ~]# ssh $host sudo systemctl restart kubelet [root@basecm11 ~]# kubectl uncordon $host ``` Output after this should now look similar to: ```bash [root@basecm11 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION node001 Ready control-plane,worker 30m v1.32.6 node002 Ready control-plane,worker 30m v1.31.10 node003 Ready worker 30m v1.31.10 node004 Ready worker 30m v1.31.10 node005 Ready worker 30m v1.31.10 node006 Ready worker 30m v1.31.10 basecm11 Ready control-plane 31m v1.32.6 ``` The procedure can be repeated for `node002` in the reference example system. After that, the control plane nodes are all in an updated state. The worker nodes still need updating. ### 4.22.6 Updating The Worker Nodes The commands for the worker nodes follow a similar pattern. They are updated one by one, and the software images must also be updated, if it has not already been done. The software image update follows the procedure described in the section on **Upgrading The Control Plane Node Starting With A Compute Node** ([here](#42242-upgrading-the-control-plane-node-starting-with-a-compute-node)), where a chroot jail is entered, the updates are carried out, and the chroot jail is left. Since nodes are drained as part of their update procedure, it is best to do them one by one, or at most in limited batches to avoid doing too many at once. The set of commands to update a single worker is: ```bash [root@basecm11 ~]# export host=node003 [root@basecm11 ~]# cmsh -c "device use $host; imageupdate -w --wait" [root@basecm11 ~]# ssh $host kubeadm upgrade node [root@basecm11 ~]# kubectl drain $host --ignore-daemonsets [root@basecm11 ~]# ssh $host sudo systemctl daemon-reload [root@basecm11 ~]# ssh $host sudo systemctl restart kubelet [root@basecm11 ~]# kubectl uncordon $host ``` The `kubectl drain` command might complain for other reasons. The administrator can decide on proceeding further by adding additional flags. For example: ```bash kubectl drain $host --ignore-daemonsets --delete-emptydir-data ``` forces the drain even if some pods have stateful data in `emptyDir` volumes. Multiple nodes can be updated as follows: **Example** ```bash [root@basecm11 ~]# export hosts='node00[4-6]' [root@basecm11 ~]# cmsh -c "device; imageupdate -n $hosts -w --wait" [root@basecm11 ~]# pdsh -w $hosts kubeadm upgrade node [root@basecm11 ~]# pdsh -N -w $hosts hostname | xargs -i -n 1 kubectl drain {} --ignore-daemonsets \ --delete-emptydir-data [root@basecm11 ~]# pdsh -w $hosts sudo systemctl daemon-reload [root@basecm11 ~]# pdsh -w $hosts sudo systemctl restart kubelet [root@basecm11 ~]# pdsh -N -w $hosts hostname | xargs -i -n 1 kubectl uncordon {} [root@basecm11 ~]# ``` Output after all the worker nodes are updated too should look similar to: ```bash [root@basecm11 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION node001 Ready control-plane,worker 35m v1.32.6 node002 Ready control-plane,worker 35m v1.32.6 node003 Ready worker 35m v1.32.6 node004 Ready worker 35m v1.32.6 node005 Ready worker 35m v1.32.6 node006 Ready worker 35m v1.32.6 basecm11 Ready control-plane 36m v1.32.6 ``` ### 4.22.7 Updating The Status In BCM For releases of BCM before BCM 11.32, setting the attribute `version` for the Kubernetes object in CMDaemon is not carried out automatically by BCM. Setting that attribute is however necessary in order to get the correct version of Kubernetes showing up in the available module files. A session to update the old version is as follows: ```bash [root@basecm11 ~]# module avail -l | grep kubernetes #what Kubernetes version is defined? kubernetes/default/1.31.10-150500.1.1 2025/06/30 18:01:08 [root@basecm11 ~]# cmsh [basecm11]% kubernetes [basecm11->kubernetes[default]]% get version 1.31.10-150500.1.1 [basecm11->kubernetes[default]]% !yum info kubeadm | grep -E '(^Source|^Version|^Release)' Version : 1.32.6 Release : 150500.1.1 Source : kubeadm-1.32.6-150500.1.1.src.rpm [basecm11->kubernetes[default]]% set version 1.32.6-150500.1.1 [basecm11->kubernetes*[default*]]% commit [basecm11->kubernetes[default]]% quit [root@basecm11 ~]# module avail -l | grep kubernetes #what Kubernetes version is defined now? kubernetes/default/1.32.6-150500.1.1 2025/06/30 21:40:57 ``` ### 4.22.8 Notes For Upgrading To Kubernetes v1.31.x The `kubeadm-init-`\* files generated by BCM for Kubernetes clusters older than v1.31, contain an incompatibility with v1.31 that cannot be solved automatically by `kubeadm config migrate` when carrying out a migration as described in [https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file). To prevent errors while executing the `kubeadm config migrate` command, any `JoinConfiguration` block needs to be removed from the `kubeadm-init-`\* files. The `kubeadm-init-`\* files are files that, for a Kubernetes cluster name, follow the pattern: ```bash /root/.kube/kubeadm-init-.yaml ``` Thus, for the default BCM Kubernetes cluster installation that creates a Kubernetes cluster called `default`, this would be the single YAML file: ```bash /root/.kube/kubeadm-init-default.yaml ``` The part to remove in such files looks like this: ```bash --- apiVersion: kubeadm.k8s.io/v1beta3 kind: JoinConfiguration # do not uncomment the following, kubelets will not be able to start # caCertPath: "/etc/kubernetes/pki/kc.name/ca.crt" ``` ### 4.22.9 Notes For Ubuntu The upstream documentation covers Ubuntu upgrades when it diverges from RHEL procedures. As an aid, the following commands, discussed in the procedure for RHEL within section [4.22](#422-kubernetes-upgrades) are followed by the corresponding commands for Ubuntu: ```bash # RHEL grep v /etc/yum.repos.d/kubernetes.repo vi /etc/yum.repos.d/kubernetes.repo #to change minor number yum list --showduplicates kubeadm --disableexcludes=kubernetes #see the versions sudo yum install -y kubeadm kubelet kubectl --disableexcludes=kubernetes #installs from repo # Ubuntu grep v /etc/apt/sources.list.d/kubernetes.list vi /etc/apt/sources.list.d/kubernetes.list #to change minor number apt-get update; apt-cache madison kubeadm # update for a fresh cache to see the versions # madison option lists versions and origin repos apt-mark unhold kubeadm kubelet kubectl && \ apt-get update && apt-get install -y kubeadm kubelet kubectl && \ apt-mark hold kubeadm kubelet kubectl #steps from upstream docs, installs from repo ``` ### 4.22.10 Notes For SLES For v1.32, for SLES, BCM can use the “tarball” approach. The installation of such non-package-manager packages is documented at [https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/), in the “Without a package manager” section. The commands in that section there can be used as a guide to find alternatives for the `yum` and `apt` commands used so far. ### 4.22.11 Other Approaches When carrying out the steps described in this chapter, some things may in practice end up being done differently from what is suggested. For example: control plane nodes often have different categories and software-images compared to the workers. For example, perhaps the control-plane nodes do not have GPUs, and need different packages. In that case, multiple software images have to be prepared with new packages. Earlier on, a process to update nodes one at a time was described. However, when a software image is updated, and multiple nodes are tied to that software image, then those nodes can all be provisioned at the same time. The binaries in the updated software image can therefore also be provisioned to all the nodes using that image. The problem with this is that it could result in nodes getting the new binaries prematurely if they happen to reboot during the update. If this is an unwanted risk, then it can be avoided in several ways, described next. #### 4.22.11.1 Additional Software Images A separate software image can be introduced. Nodes can be moved to the new software image one at a time. A software image can also be overruled at the level of a node. **Example** ```bash [root@basecm11 ~]# cmsh [basecm11]% device use node001 [basecm11->device[node001]]% get softwareimage default-image (category:default) [basecm11->device[node001]]% set softwareimage my-new-image [basecm11->device*[node001*]]% commit [basecm11->device[node001]]% get softwareimage my-new-image [basecm11->device[node001]]% quit ``` The cluster administrator should remember to undo the preceding settings, or should move the new software image to the appropriate category, and then clear the node-level override again. This is so that, after the upgrade, the system is organized as before. #### 4.22.11.2 Using NOSYNC The other option is to configure the nodes with `NOSYNC` for their `Next install mode`. This prevents them from syncing with their software image when rebooting (in the case that they still reboot): **Example** ```bash [root@basecm11 ~]# cmsh [basecm11]% device use node001 [basecm11->device[node001]]% set nextinstallmode nosync [basecm11->device*[node001*]]% commit ``` Both these approaches make updating slightly more tedious, but also more straightforward. During testing by BCM developers, nodes getting their binaries updated prematurely due to an unexpected reboot was not seen to be a significant issue. This is presumably because as long as the first control plane node is updated successfully, and the reboot of the extra node is by accident, there is an interruption anyway. The kubelet simply comes back up with the new version. However this is not the official recommended approach.