> This page is for version 10. > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Kubernetes Kubernetes is an open-source platform for automating deployment, scaling, and operations of application containers across clusters of hosts. With Kubernetes, it is possible to: * scale applications on the fly * seamlessly update running services * optimize hardware usage by using only the resources that are needed BCM provides the administrator with the required packages, allows Kubernetes to be set up on a cluster, and manages and monitors Kubernetes. More information about the design of Kubernetes, its command line interfaces, and other Kubernetes-specific details, can be found at the official online documentation at [https://kubernetes.io/docs/](https://kubernetes.io/docs/). To deploy most of Kubernetes, NVIDIA Base Command Manager from version 10.0 onwards uses `kubeadm` ([https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/)). BCM runs CoreDNS, the Kubelet component ([https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/)), and the entire control plane inside Kubernetes. The distributed key-value store used for Kubernetes, etcd, is typically run inside of Kubernetes on control planes. However, in BCM etcd is run outside of Kubernetes. Kubernetes integration with BCM is available for Kubernetes v1.28, 1.29, v1.30, v1.31, and v1.32 at the time of writing of this paragraph (April 2025). Kubernetes runs on the `x86_64` and `aarch64` architecture for all the BCM-supported Linux distributions. For a more up-to-date status, the features matrix at [https://service.bcm.nvidia.com/feature-matrix/](https://service.bcm.nvidia.com/feature-matrix/) can be checked. The version that is running on a cluster can be found with: **Example** ```bash root@basecm10:~# kubeadm version -o short v1.34.5 ``` The versions that are available with BCM integration can be found with the `cm-kubernetes-setup` utility (section [4.2](#42-kubernetes-setup)): **Example** ```bash root@basecm10:~# cm-kubernetes-setup --list-versions 1.35 1.34 1.33 (NVIDIA AI Enterprise certified) 1.32 (NVIDIA AI Enterprise certified) ``` ## 4.1 Reference Architecture A reference architecture for Kubernetes in BCM comprises: * *etcd nodes*: An etcd cluster—the Kubernetes distributed key-value storage—runs on an odd number (1, 3, 5 ...) of nodes. * *control plane nodes*: typically run on head nodes or on dedicated nodes. 2 or 3 are recommended. * *worker nodes*: typically run on regular nodes that are designated to run user workloads. To avoid single point of failure and to achieve high availability, a minimum of three nodes is recommended for etcd, and a minimum of two control plane nodes is recommended. **Load Balancing** An NGINX server is configured on the head node(s) and on all other nodes involved in the Kubernetes cluster (control plane or worker nodes). This NGINX server takes care of exposing the Kubernetes API server on a specific port, and load balances requests to the control plane nodes. Should one of those nodes go down it can detect this and stop sending requests until the node comes back up. Since BCM version 8.2, multiple clusters of Kubernetes can be deployed. In such a configuration, the same nodes cannot be shared across different Kubernetes clusters. Because of the NGINX server, a port is reserved on the head node(s) for every Kubernetes cluster. This is required for Kubernetes HA , and it also allows `kubectl` and other tools such as Helm to be used from the head node, to access each Kubernetes cluster. The same NGINX server is also used to similarly expose the NGINX Ingress Controller, if this has been chosen during setup. Alternatively, MetalLB ([https://metallb.universe.tf/](https://metallb.universe.tf/)) is also supported, and provides a different approach towards load balancing for Kubernetes that is more similar to managed cloud solutions. ### 4.1.1 Hardware Requirements To run properly, Kubernetes deployment requires a per-node minimum of * 16 GB RAM * two CPU cores Without these minimum requirements, deployment may succeed, but further configuration, such as starting up Jupyter kernels, may fail with problems that are hard to diagnose. For example, error reporting may not show a lack of resources because some components are automatically deleted. ## 4.2 Kubernetes Setup The usual, and recommended way, to install Kubernetes with BCM is to install it interactively from a TUI session (section [4.2.6](#426-kubernetes-setup-from-a-tui-session)). An alternative way, typically used for automated non-interactive setups, is to use a plain command line specification (section [4.2.5](#425-kubernetes-setup-from-the-command-line)), which usually results in a long command line. BCM deploys Kubernetes with the `cm-kubernetes-setup` utility, which is part of the `cm-setup` package. Several recent versions of Kubernetes are offered (figure [4.1](#fig-k8s-version-selection)): ![Kubernetes setup TUI session (section 4.2.6): version selection screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/563b981e717910b5fdc309bc7d3954215b2d3777d65552bc79ce3b2452c5f884/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-kubernetes-setup-versions.png) *Figure 4.1: Kubernetes setup TUI session (section [4.2.6](#426-kubernetes-setup-from-a-tui-session)): version selection screen* BCM provides or uses the following Kubernetes-related packages: * `conntrack` (`conntrack-tools` on RHEL-based distributions) and `nginx`: These distribution packages are always installed on the head node(s) and on the master and worker node(s). * `cm-etcd`: This BCM package is installed on the nodes selected for etcd. In a similar way to the case of Harbor or Docker Registry (section [3.2](/containerization/dockerregistries#32-docker-and-harbor-registries-setup-and-configuration)), the nodes that run etcd are protected by BCM with the `datanode` option ([here](/admin/provisioning#5444-full-install-confirmation-via-datanode-setting)). For etcd nodes, the option is added during the `cm-kubernetes-setup` installation. As in the case for the registries, the datanode option is set in order to help prevent the administrator from wiping out the existing state of etcd nodes. Wiping out the state of etcd nodes means that the Kubernetes cluster becomes incoherent and that it cannot be restored to where it was just before the etcd nodes were wiped. The etcd version installed by the BCM package is 3.5.8. * `cm-containerd`: This BCM package has the containerd runtime. * `cm-nvidia-container-toolkit`: This BCM package has the NVIDIA container toolkit (includes NVIDIA container runtime). * `cm-kube-diagnose`: This BCM package has Helper tools to diagnose malfunctioning Kubernetes clusters. Kubernetes .rpm and .deb packages themselves are installed from the Kubernetes community-owned software repositories ([https://kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/](https://kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/)). These repositories host Kubernetes versions starting from v1.24.0 at the time of writing of this paragraph (March 2024). ### 4.2.1 Kubernetes Networking Early on during the wizard session (figure [4.2](#fig-k8s-networking)), a name for the cluster is requested. The wizard pre-fills it with `default`, but this should not be confused with the Kubernetes `default` namespace. Here, the name is used instead, inside BCM, to identify the cluster, configuration files, and other resources such as module files. ![Kubernetes setup TUI session: networking selection screen](/base-command-manager/product-manuals/_fern-img/5f2b9ec9406639b10e77aca3a4ed36d9bcdebed07501aa17fa230c32a6a2c803.webp) *Figure 4.2: Kubernetes setup TUI session: networking selection screen* This screen also allows the following important choices: * `Kubernetes external FQDN`: This is the FQDN that is placed as one of the entries in the public-facing certificates generated for this Kubernetes cluster. Configuring the public-facing certificate of the NGINX Ingress Controller is discussed further in section [4.22.12](#42212-configuring-the-ingress-https-server-certificate). * `Service network base address` and `Service network netmastk bits`: These define the CIDR for the service network. The wizard pre-fills the fields. It also tries to avoid pre-filling them with overlapping network ranges, by taking any existing network known to BCM into account. * `Pod network base address` and `Pod network netmask bits`: These define the CIDR for the pod network. The wizard pre-fills these. It also tries to avoid pre-filling them with overlapping network ranges, by taking any existing network known to BCM into account. By default, entire /24 network ranges are assigned to individual Kubernetes nodes from the pod CIDR. The packages are installed automatically from the repository when the administrator runs `cm-kubernetes-setup` from the command line. The log file produced by the setup can be found in `/var/log/cm-kubernetes-setup.log`. ### 4.2.2 Kubernetes Core Add-ons During setup, some critical add-on components such as the Networking Component for Kubernetes are automatically deployed. Most components are in the `kube-system` namespace, but others have their own namespaces. In BCM some add-ons are treated as Kubernetes applications (Chapter [5](/containerization/kubernetesapps)), and belong to the default app group, `system`. In the future Helm is expected to manage most, if not all, components instead of BCM Kubernetes applications. The user is prompted for a Networking Component, Kubernetes operators (managed via Helm), Kubernetes add-ons (managed via Kubernetes applications), and more. They all result in either Helm charts being deployed, or in Kubernetes applications being managed from BCM. A `cmsh` treeview illustrating the hierarchy to access these applications is: ```bash [cmsh] |-- ... | |-- kubernetes[default] | `-- appgroups[system] | `-- applications |-- ... ``` Helm charts can be found using the `helm` command. For example using `helm list -A -a` on a cluster (after loading the correct module file). In the past the DNS component was also provided as an add-on. However since BCM version 10.0, `kubeadm` is used to bundle CoreDNS as part of the default control plane. Kubernetes apps are discussed further in Chapter [5](/containerization/kubernetesapps). Kubernetes operators are discussed further in Chapter [6](/containerization/kubernetesoperators). #### 4.2.2.1 Kubeadm Components Components deployed as part of the default Kubernetes control plane are: * Kubernetes API server * Kubernetes Scheduler * Kubernetes Controller Manager * Kubernetes Proxy * Core DNS These are deployed through the “static pod” mechanism ([https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/](https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/)). Kubernetes control plane components are described at: [https://kubernetes.io/docs/concepts/overview/components/](https://kubernetes.io/docs/concepts/overview/components/). **CoreDNS** CoreDNS is the DNS server add-on for internal service discovery. It reads the IP addresses of services and pods from etcd, and resolves domain names for them. If a domain name is not found because the domain is external to the Kubernetes cluster, then CoreDNS forwards the request to the main DNS server. BCM uses CoreDNS version 1.11.1 with Kubernetes version 1.31. #### 4.2.2.2 Networking Component An important component managed through BCM’s Kubernetes Apps is the Networking component. This adheres to the Container Networking Interface, or CNI interface. At the time of writing the available CNIs in the setup wizard are either Calico or Flannel. However BCM does support other CNIs, for example Cilium, as described in the knowledge base article at [https://enterprise-support.nvidia.com/s/article/installing-cilium-networking-for-kubernetes](https://enterprise-support.nvidia.com/s/article/installing-cilium-networking-for-kubernetes). **Flannel** Flannel is a simple and easy way to configure a layer 3 network fabric designed for Kubernetes. **Calico** Calico is a popular open source networking and network security solution for Kubernetes. It provides network connectivity between workloads and security policies features. It can be configured to use eBPF. BCM uses Calico version 3.26. Further details on Calico can be found at [https://docs.projectcalico.org/](https://docs.projectcalico.org/). If the Kubernetes cluster is composed of more than 50 nodes, then the Calico component Typha is also automatically deployed for better scalability. The number of Typha replicas is calculated by allocating one Typha replica per 150 nodes, with a minimum of 3 (above 50 nodes) and a maximum of 20. If an initial deployment of the Kubernetes cluster has fewer than 50 nodes, but nodes are then added to the Kubernetes cluster so that the 50 node threshold is exceeded, then Typha is not automatically enabled. In this case, Typha can be enabled manually via `cmsh` as follows: **Example** ```bash [basecm10->kubernetes[default]->appgroups[system]->applications[calico]]% environment [basecm10->kubernetes[default]->appgroups[system]->applications[calico]->environment]% list Name (key) Value Nodes environment ---------------------- ---------------- ------------------ calico_typha_replicas 0 no calico_typha_service none no cidr 10.141.0.0/16 no [basecm10->kubernetes...[calico]->environment]% set calico_typha_service value calico-typha [basecm10->kubernetes*...[calico*]->environment*]% set calico_typha_replicas value 3 [basecm10->kubernetes*[default*]->appgroups*[system*]->applications*[calico*]->environment*]% commit [basecm10->kubernetes[default]->appgroups[system]->applications[calico]->environment]% ``` **Cilium** Cilium ([https://cilium.io/](https://cilium.io/)) is a networking, observability, and security solution with an eBPF-based dataplane. ### 4.2.3 Kubernetes Optional Add-ons The following add-ons are installed by default unless otherwise noted. However, the user can choose to skip some or all of them during the setup. #### 4.2.3.1 NGINX Ingress Controller The official Kubernetes Ingress controller add-on is built around the Kubernetes Ingress resource, using a ConfigMap to store the NGINX configuration. Ingress provides HTTP and HTTPS routes from outside a Kubernetes cluster to services within the cluster. Traffic routing is controlled by rules defined in the Ingress resource. BCM uses NGINX Ingress Controller version 1.5.1. By default, BCM suggests the following ports for Ingress: 30080 is the default that is set for the HTTP, and port 30443 is the default that is set for HTTPS. These 2 ports are exposed on every Kubernetes node, both masters and workers. The Ingress Controller is deployed as a [`NodePort` service](https://kubernetes.io/docs/concepts/services-networking/service/#nodeport), which means that it comes with a default range of possible port values of `30000-32767`. #### 4.2.3.2 Kubernetes Dashboard Kubernetes Dashboard is the web user interface add-on for GUI cluster administration and metrics visualization. BCM uses Kubernetes Dashboard version 2.7.0. There are two ways to access the dashboard: * Using `kubectl proxy` and accessing [http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/](http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/) To use the proxy, `kubectl` must be set up locally ([section 9.3.2 of the *User Manual*](/user/kube#932-quickstart-using-kubectl-from-a-local-machine)). * Users on an external network can log in to `kubectl` or Kubernetes Dashboard by following the procedures described in section [4.16](#416-providing-access-to-external-users). If NGINX Ingress Controller is deployed, then a link pointing to the Kubernetes dashboard can also found on the BCM landing page. #### 4.2.3.3 Kubernetes Metrics Server The Kubernetes Metrics Server is an add-on that is a replacement for Heapster. It aggregates metrics, and provides container monitoring and performance analysis. It exposes metrics via an API. BCM runs Metrics server version 0.6.3. #### 4.2.3.4 Kubernetes State Metrics `kube-state-metrics` is an add-on agent to generate and expose cluster-level metrics for Kubernetes objects. The project is not focused on the health of the individual Kubernetes components, but rather on the health of the various objects inside, such as deployments, nodes and pods. ### 4.2.4 Helm Kubernetes Package Manager Helm is an add-on that manages *charts*, which are packages of pre-configured Kubernetes resources. The Helm component is installed and properly configured with BCM’s Kubernetes installation by default. It is initialized and ready for use by every Kubernetes user when the Kubernetes module is loaded. BCM uses Helm version 3. When the Helm binary is installed during the Kubernetes setup, an offer is made to deploy charts during the setup process. Most operators are documented in Chapter [6](/containerization/kubernetesoperators). Other parts of the wizard also determine which Helm charts are deployed as part of the setup for the following: * Kyverno and Kyverno Policies (section [4.10.1](#4101-kyverno)) * BCM Permissions Manager (section [4.15](#415-kubernetes-permission-manager)) * BCM Local Path Provisioner (section [4.19](#419-local-path-storage-class)) **Example** ```bash root@basecm10:~# helm list -A -a NAME NAMESPACE CHART APP VERSION cm-jupyter-kernel-operator cm ... cm-jupyter-kernel-operator-0.1.10 0.1.10 cm-kubernetes-mpi-operator cm ... mpi-operator-0.4.0 0.4.0 gpu-operator gpu-operator ... gpu-operator-v23.9.1 v23.9.1 kyverno kyverno ... kyverno-3.0.4 v1.10.2 kyverno-policies kyverno ... kyverno-policies-3.0.3 v1.10.2 local-path-provisioner cm ... cm-kubernetes-local-path-provisioner-0.0.26 0.0.26 metallb metallb-system ... metallb-0.14.3 v0.14.3 network-operator network-operator ... network-operator-23.10.0 v23.10.0 permissions-manager cm ... cm-kubernetes-permissions-manager-0.4.8 0.4.8 postgres-operator postgres-operator ... postgres-operator-1.10.1 1.10.1 prometheus-adapter prometheus ... prometheus-adapter-3.3.1 v0.9.1 prometheus-operator prometheus ... kube-prometheus-stack-35.5.1 0.56.3 spark-operator spark-operator ... spark-operator-1.1.27 v1beta2-1.3.8-3.1.1 ``` ### 4.2.5 Kubernetes Setup From The Command Line The `cm-kubernetes-setup` command line utility has the following usage synopsis: ```bash [root@basecm10 ~]# cm-kubernetes-setup -h usage: Kubernetes Setup cm-kubernetes-setup [-c ] [--list-versions] [--list-operators-versions] [--cluster CLUSTER_NAME] [--skip-docker] [--skip-reboot] [--skip-image-update] [--skip-dns-configuration-check] [--skip-package-manager-update-check] [--skip-install-repos] [--add-user USERNAME_ADD] [--list-users] [--get-user GET_USER] [--modify-user USERNAME_MODIFY] [--remove-user USERNAME_REMOVE] [--namespace NAMESPACE] [--add-to-namespace] [--remove-from-namespace] [--role edit,admin,view,cluster-admin] [--runas-uid RUNAS_UID] [--runas-gids RUNAS_GIDS] [--user-paths USER_PATHS] [--allow-all-uids] [--operators OPERATORS] [--backup-permissions FILE] [--restore-permissions FILE] [--list-operators] [--update-addons] [--remove] [--yes-i-really-mean-it] [--remove-ceph-storage] [--pull] [--images IMAGES] [--nodes NODES] [--node-selector NODE_SELECTOR] [--pull-registry-server PULL_REGISTRY_SERVER] [--pull-registry-username PULL_REGISTRY_USERNAME] [--pull-registry-email PULL_REGISTRY_EMAIL] [--pull-registry-password PULL_REGISTRY_PASSWORD] [--docker-io-username DOCKER_IO_USERNAME] [--docker-io-password DOCKER_IO_PASSWORD] [--kubeadm-image-repository KUBEADM_IMAGE_REPOSITORY] [--containerd-sandbox-image CONTAINERD_SANDBOX_IMAGE] [--skip-netq-prerequisites-checks] [-v] [--store-name-aliases] [--no-distro-checks] [--json] [--output-remote-execution-runner] [--on-error-action debug,remotedebug,undo,abort] [--skip-packages] [--min-reboot-timeout ] [--allow-running-from-secondary] [--dev] [-h] optional arguments: --cluster CLUSTER_NAME Name of the referred Kubernetes cluster -h, --help Print this screen common: Common arguments -c Load runtime configuration for plugins from a YAML config file Supported versions information: --list-versions Show supported Kubernetes versions --list-operators-versions List available for selection Kubernetes operators versions installing Kubernetes clusters: Flags for installing or managing Kubernetes clusters --skip-docker Skip the Docker installation steps. --skip-reboot Skip the reboot steps. --skip-image-update Skip the image update steps. --skip-dns-configuration-check Skip DNS configuration check before install (setup will not continue of it finds DNS malfunctioning). --skip-package-manager-update-check Skip Package Manager update/refresh check before install (setup will not continue if 'apt update' fails on Ubuntu for example). --skip-install-repos Skip installation of kubernetes and helm package repos.(assumes they have been manually configured, e.g., in the case of air-gapped). user management: Flags for adding a new user in a Kubernetes cluster --add-user USERNAME_ADD Create a new user in a Kubernetes cluster --list-users Get information about configured Kubernetes users --get-user GET_USER Get information about configured Kubernetes users --modify-user USERNAME_MODIFY Modify user in a Kubernetes cluster --remove-user USERNAME_REMOVE Remove existing user from a Kubernetes cluster --namespace NAMESPACE Specify namespace for user (--get-user, --modify-user) role binding --add-to-namespace Indicate if permissions to manage namespace needs to be granted for a given user (--modify-user) --remove-from-namespace Indicate if permissions to manage namespace needs to be revoked for a given user (--modify-user) --role edit,admin,view,cluster-admin Specify role for the new (--add-user) and existing (--modify-user) role binding Default: edit) For 'cluster-admin' namespace flag is ignored --runas-uid RUNAS_UID UID is allowed to be used in unprivileged pods (--add-user, --modify-user) --runas-gids RUNAS_GIDS Comma-separated list of GIDs allowed to be used in unprivileged pods (--add-user, --modify-user) --user-paths USER_PATHS Comma-separated list of paths user is able to mount in pods (--add-user, --modify-user) --allow-all-uids Allow user to run processes in pods as any user (--add-user, --modify-user) hostPath volumes will be disabled for such pods --operators OPERATORS Comma-separated list of operators user has access to (--add-user, --modify-user) backup or restore Permission Manager user configurations: Flag for managing permission manager user configuration --backup-permissions FILE Save permissions to file --restore-permissions FILE Restore permissions from file. Workload which is already run by users in their namespaces will be affected list available operators: Flag to list available Kubernetes operators --list-operators List available Kubernetes operators update kubernetes addons: Flags for updating Kubernetes addons --update-addons Update Addons removing Kubernetes clusters: Flags for removing a Kubernetes cluster --remove Remove a Kubernetes cluster --yes-i-really-mean-it Required for additional safety --remove-ceph-storage Remove Kubernetes osd pool from Ceph cluster pulling images to the nodes: Flags for pulling images to the nodes --pull Pull images to the nodes --images IMAGES Comma-separated list of images to pull (--pull) --nodes NODES Comma-separated list of nodes to pull images to (--pull) --node-selector NODE_SELECTOR Selector (label query) to filter on, supports '=', '==', and '!='.(e.g. key1=value1,key2=value2) (--pull) --pull-registry-server PULL_REGISTRY_SERVER Registry server to authenticate --pull-registry-username PULL_REGISTRY_USERNAME Registry username --pull-registry-email PULL_REGISTRY_EMAIL Email to authenticate --pull-registry-password PULL_REGISTRY_PASSWORD Registry password docker.io: Flags for authenticating to docker hub --docker-io-username DOCKER_IO_USERNAME Username --docker-io-password DOCKER_IO_PASSWORD Password kubeadm: Flags for configuring kubeadm --kubeadm-image-repository KUBEADM_IMAGE_REPOSITORY Custom imageRepository containerd: Flags for configuring containerd --containerd-sandbox-image CONTAINERD_SANDBOX_IMAGE Custom sandbox image (e.g. my-registry:5000/pause:3.8) netq: Flags for configuring NetQ --skip-netq-prerequisites-checks Provide this flag to skip precondition checks such as checking for disk size, Linux distro, etc. advanced: Various *advanced* configuration options flags. -v, --verbose Verbose output --store-name-aliases Store hostname aliases for head nodes (active and passive) and default category --no-distro-checks Disable distribution checks based on ds.json --json Use json formatting for log lines printed to stdout --output-remote-execution-runner Format output for CMDaemon --on-error-action debug,remotedebug,undo,abort Upon encountering a critical error, instead of asking the user for choice, setup will do selected action. --skip-packages Skip the any stages which install packages. Requires packages to be already installed. --min-reboot-timeout How long to wait for nodes to finish reboot (default and minimum allowed: 300 seconds). --allow-running-from-secondary Allow to run the wizard from the secondary when it is the active head node. --dev Enables additional command line arguments ``` The `cm-kubernetes-setup` utility should be executed on the console. #### 4.2.5.1 Dealing With A Pre-existing Docker Installation Docker (Chapter [2](/containerization/dockerengine)) is no longer a requirement for Kubernetes configured by BCM. This is because Kubernetes can directly interface with containerd through its Container Runtime Interface (CRI). Docker can co-exist with Kubernetes, and can be set up as discussed in section [2.1](/containerization/dockerengine#21-docker-setup). ### 4.2.6 Kubernetes Setup From A TUI Session When the Kubernetes installation is carried out using `cm-kubernetes-setup` without any options, a TUI wizard starts up. The administrator can answer several questions in the wizard. Questions that are asked include questions about which of the node categories or which of the individual nodes should be configured to run the Kubernetes services. There are also questions about the service and pod networks parameters, the port numbers that will be configured for the daemons, whether to install specific add-ons, and so on. After the wizard has been completed, a configuration file can be saved that can be used to set up Kubernetes. The configuration file can be deployed immediately from the wizard, or it can be deployed later by specifying it as an option to `cm-kubernetes-setup`, in the form `-c` \<*`file`*>. If no deployment has been carried out earlier, then the main operations screen of the wizard shows just two options, `Deploy` and `Exit`. If deployment has already been carried out, then the further options that are available are also displayed (figure [4.3](#fig-kube-urwid-wizard-operations)): **Example** ```bash [root@basecm10 ~]# cm-kubernetes-setup TUI session starts up: ``` ![Kubernetes setup TUI session: main operations screen after a deployment](/base-command-manager/product-manuals/_fern-img/ff22c6137ef385b67ffcdbcce42e9f55982e8f5e4a489fc1c529cb62e6b978ef.webp) *Figure 4.3: Kubernetes setup TUI session: main operations screen after a deployment* The deployment via CLI or via TUI assigns the appropriate roles, and adds the new Kubernetes cluster. The deployment adds health checks to the monitoring configuration, and it generates certificates for the Kubernetes daemons. Calico is set as the Kubernetes network plugin by default. Flannel is an option. The master, worker, and etcd nodes can be assigned to specific nodes or categories. The network configuration settings for the Kubernetes cluster can be specified. Ports have default assignments, but can be re-assigned as needed. The `etcd` spool file path can be set. The following options are also possible: * a registry mirror from DockerHub can be specified * the Kubernetes API server can be exposed to the external network * the internal network used by Kubernetes nodes can be selected Add-ons that are available are: * Ingress Controller (Nginx) * Kubernetes Dashboard * Kubernetes Metrics Server * Kubernetes State Metrics Operator packages are application managers, and are described further in Chapter [6](/containerization/kubernetesoperators). Operators that can be installed are: * `cm-jupyter-kernel-operator` * `cm-kubernetes-postgresql-operator` * `cm-kubernetes-spark-operator` The permission manager—an application for role-based access control—can also be configured. ### 4.2.7 Testing Kubernetes To test that Kubernetes works, the `cluster-info` command can be run. For example, on the head node, `basecm10`: **Example** ```bash [root@basecm10 ~]# module load kubernetes #not actually needed -- autoloaded these days [root@basecm10 ~]# kubectl cluster-info Kubernetes control plane is running at https://127.0.0.1:10443 CoreDNS is running at https://127.0.0.1:10443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. ``` After `cm-kubernetes-setup` finishes, and the regular nodes have been rebooted, the state of the nodes can be found by running the `get nodes` command: **Example** ```bash [root@basecm10 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION node001 Ready worker 18h v1.31 node002 Ready worker 18h v1.31 basecm10 Ready control-plane,master 18h v1.31 ``` A six node cluster should show the following Kubernetes installed add-ons, when using `kubectl` with the `get all -n kube-system` option (some lines truncated): **Example** ```bash [root@basecm10 ~]# kubectl get all -n kube-system NAME READY STATUS RESTARTS AGE pod/calico-kube-controllers-58497c65d5-skhgw 1/1 Running 0 26m pod/calico-node-27xj7 1/1 Running 0 26m pod/calico-node-6hmm5 0/1 Running 1 26m pod/calico-node-987qv 1/1 Running 0 26m pod/calico-node-gcbcm 1/1 Running 0 26m pod/calico-node-hlsrj 1/1 Running 0 26m pod/calico-node-q7k4v 1/1 Running 0 26m pod/calico-node-qdbq5 0/1 Running 0 26m pod/calico-node-v2dxj 1/1 Running 0 26m pod/coredns-6768db756-8l9fs 1/1 Running 0 26m pod/coredns-6768db756-cs58q 1/1 Running 0 26m pod/kube-state-metrics-758ccc75d6-75dsn 1/1 Running 0 26m pod/metrics-server-7b477dd7b9-2drkg 1/1 Running 0 26m pod/metrics-server-7b477dd7b9-z6nch 1/1 Running 0 26m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/calico-typha ClusterIP 10.150.121.25 5473/TCP 26m service/kube-dns ClusterIP 10.150.255.254 53/UDP,53/TCP,9153/TCP 26m service/kube-state-metrics ClusterIP None 8080/TCP,8081/TCP 26m service/metrics-server ClusterIP 10.150.99.149 443/TCP 26m NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/calico-node 8 8 6 8 6 kubernetes.io/os=linux 26m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/calico-kube-controllers 1/1 1 1 26m deployment.apps/calico-typha 0/0 0 0 26m deployment.apps/coredns 2/2 2 2 26m deployment.apps/kube-state-metrics 1/1 1 1 26m deployment.apps/metrics-server 2/2 2 2 26m NAME DESIRED CURRENT READY AGE replicaset.apps/calico-kube-controllers-58497c65d5 1 1 1 26m replicaset.apps/calico-typha-68857595fc 0 0 0 26m replicaset.apps/coredns-6768db756 2 2 2 26m replicaset.apps/kube-state-metrics-758ccc75d6 1 1 1 26m replicaset.apps/metrics-server-7b477dd7b9 2 2 2 26m ``` The administrator can now configure the cluster to suit the particular site requirements. ## 4.3 Using GPUs With Kubernetes: NVIDIA GPUs ### 4.3.1 Prerequisites The GPUs have to be recognized by the nodes, and have the appropriate drivers (such as `cuda-driver`) installed. Details on how to do this are given in [section 9 of the *Installation Manual*](/installation/nvidiagpus). To verify the GPUs are recognized and have drivers in place, the `nvidia-smi` command can be run. The response displayed for a GPU should look similar to the following: **Example** ```bash root@node001:~# nvidia-smi Tue Dec 7 11:25:21 2021 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 Tesla K40c On | 00000000:00:06.0 Off | Off | | 23% 37C P8 21W / 235W | 0MiB / 12206MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ ``` If a non-BCM Containerd has already been deployed before Kubernetes is deployed, then `cm-kubernetes-setup` may replace an existing Containerd configuration file in order to enable NVIDIA GPU integration via a Kubernetes CNI plugin. This is because Containerd is configured by `cm-kubernetes-setup`, overwriting any previous configuration. ### 4.3.2 New Kubernetes Installation As part of the setup, `cm-kubernetes-setup` assigns a new role to the Kubernetes worker nodes: `generic::containerd`. The role has a `Configurations` submode, in which the `containerd-cri` object can be configured. The entry for `Filename` specifies the path to the `cri.toml` file, which contains content used by the container runtime interface on the Kubernetes worker nodes that have been assigned the role. **Example** ```bash [basecm10->configurationoverlay[kube-default-worker]->roles]% use generic::containerd [...]->roles[generic::containerd]]% show Parameter Value -------------------------------- ------------------------------------------------ Name generic::containerd Type GenericRole Add services yes Services containerd Configurations <2 in submode> Environments <1 in submode> Exclude list snippets <1 in submode> Data node no [...]->roles[generic::containerd]]% configurations [...]->roles[generic::containerd]->configurations]% use containerd-cri [...]->roles[generic::containerd]->configurations[containerd-cri]]% show Parameter Value -------------------------------- -------------------------------------------------- Name containerd-cri Type static Create directory yes Filename /cm/local/apps/containerd/var/etc/conf.d/cri.toml Filemask directory 0644 User name Group name Disabled no Service action on write RESTART Service stop on failure yes Content <645B> Filemask 0644 ``` The file with the CRI (Container Runtime Interface) configuration is created in directory: ```bash /cm/local/apps/containerd/var/etc/conf.d ``` and included into the main Containerd configuration file: ```bash /cm/local/apps/containerd/var/etc/config.toml ``` with the `imports` statement: ```bash imports = ["/cm/local/apps/containerd/var/etc/conf.d/*.toml"] ``` Whatever the container runtime that is selected, if NVIDIA GPU integration is required then the NVIDIA container toolkit is taken care of by the installer. The configuration for Containerd is added as a separate configuration file in the `configurations` submode, as `nvidia-cri`. NVIDIA GPUs are integrated into Kubernetes using the NVIDIA GPU operator. This is discussed further in section [6.4](/containerization/kubernetesoperators#64-the-nvidia-gpu-operator). ### 4.3.3 Existing Kubernetes Installation The NVIDIA GPU operator can always be deployed through Helm. The official documentation for the NVIDIA GPU operator is at [https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/overview.html](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/overview.html) ### 4.3.4 An Example Of Running A Workload Using NVIDIA GPUS: LLM Via NIM NIM originally stood for NVIDIA Inference Microservices, but does not stand for anything now. This is because inference engines—the “I” in NIM—are not the only workloads that are intended to run on GPUs that use pod microservices. NIM is now a product name, for a platform that hosts multiple GPU-based microservices. With NIM as a product name, “NVIDIA NIM microservices” is now regarded as a valid phrase, instead of standing for the excessively redundant phrase “NVIDIA NVIDIA Inference Microservices microservices”. Once GPUs are configured and available for pods, then it is easy to deploy NVIDIA NIM microservices. #### 4.3.4.1 Requirements For NIM A minimal setup to run NIM has the following requirements: * A `git` CLI installed on the cluster node where the Kubernetes API and Kubernetes CLI are available. * Access to the NGC catalog. This is used to acquire a personal API key to download the container image and model. The key must have access to the following services (figure [4.4](#fig-nim-api-key)): * `AI Foundation Models and Endpoints` * `NGC Catalog` * `Private Registry` ![NIM personal key generation](/base-command-manager/product-manuals/_fern-img/e441454b1e61d8b8a9f7fe38b05a123cc8c4a0f9fa0496d927dffe8e6e322ef6.webp) *Figure 4.4: NIM personal key generation* * If the image repository does not provide a Helm chart with a container and model already, then `nvcr.io/nim/meta/llama3-8b-instruct` is used by default. Supported GPU configurations for Llama 3 8B are specified at: [https://docs.nvidia.com/nim/large-language-models/latest/support-matrix.html#llama-3-8b-instruct](https://docs.nvidia.com/nim/large-language-models/latest/support-matrix.html#llama-3-8b-instruct) GPU support configuration for the various LLMs is described at: ` https://docs.nvidia.com/nim/large-language-models/latest/support-matrix.html` #### 4.3.4.2 Setting Up And Running The LLM Via NIM A running LLM, `llama3-8b-instruct`, can be created and hosted as a service as follows: ```bash [root@basecm10 ~]# git clone https://github.com/NVIDIA/nim-deploy.git [root@basecm10 ~]# cd nim-deploy/helm [root@basecm10 ~]# export NGC_CLI_API_KEY= [root@basecm10 ~]# kubectl create namespace nim [root@basecm10 ~]# helm --namespace nim install my-nim nim-llm/ \ --set model.ngcAPIKey=$NGC_CLI_API_KEY \ --set persistence.enabled=true ``` The preceding sets up a namespace `nim` with the name `my-nim`, deployed with the personal API key from NGC. The existence of the service (sometimes called a microservice) can be checked with: ```bash [root@basecm10 ~]# kubectl get service -n nim NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE my-nim-nim-llm ClusterIP 10.150.226.159 8000/TCP 2m29s ``` The API can be tested with: ```bash [root@basecm10 ~]# curl -X POST http://10.150.226.159:8000/v1/completions \ -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -d '{"model": "meta-llama3-8b-instruct", "max_tokens": 64, "prompt": "Once upon a time"}' ``` A typical response is: ```bash { "id": "cmpl-9fed783f0c914e8fa82625946a78cd5e", "object": "text_completion", "created": 1716571188, "model": "meta-llama3-8b-instruct", "choices": [ { "index": 0, "text": ", there was a young man named Jack who lived in a small village at the", "logprobs": null, "finish_reason": null, "stop_reason": null } ], "usage": { "prompt_tokens": 5, "total_tokens": 69, "completion_tokens": 64 } } ``` #### 4.3.4.3 More About NIM More documentation on NIM can be found at: [https://docs.nvidia.com/nim/large-language-models/latest/getting-started.html](https://docs.nvidia.com/nim/large-language-models/latest/getting-started.html) and [https://github.com/NVIDIA/nim-deploy/tree/main](https://github.com/NVIDIA/nim-deploy/tree/main) ## 4.4 Using GPUs With Kubernetes: AMD GPUs ### 4.4.1 Prerequisites The GPUs have to be recognized by the node. One way to check this from within BCM is to run `sysinfo` for the node: **Example** ```bash [basecm10->device[basecm10]]% sysinfo | grep GPU Number of GPUs 1 GPU Driver Version 4.18.0-193.el8.x86_64 GPU0 Name Radeon Instinct MI25 ``` In order to make Kubernetes aware of nodes that have AMD GPUs, the AMD GPU device plugin has to be deployed as a DaemonSet inside Kubernetes. The official GitHub repository that hosts this plugin can be found at: [https://github.com/RadeonOpenCompute/k8s-device-plugin](https://github.com/RadeonOpenCompute/k8s-device-plugin) The device plugin requires Kubernetes v1.16+, which has been around since BCM version 9.0. With some extra instructions, the plugin can also be made a part of BCM version 8.2. The DaemonSet YAML file can be deployed with: **Example** ```bash kubectl create -f https://raw.githubusercontent.com/RadeonOpenCompute/k8s-device-plugin/v1.16/ k8s-ds-amdgpu-dp.yaml ``` ### 4.4.2 Managing The YAML File Through CMDaemon The plugin can be added by the user via the Kubernetes appgroups as an application. In the session that follows, it is given the arbitrary name `device-plugin`: **Example** ```bash [root@basecm10 ~]# wget https://raw.githubusercontent.com/RadeonOpenCompute/k8s-device-plugin/ v1.16/k8s-ds-amdgpu-dp.yaml -O /tmp/k8s-ds-amdgpu-dp.yaml [root@basecm10 ~]# cmsh [basecm10]% kubernetes [basecm10->kubernetes[default]]% appgroups [basecm10->kubernetes[default]->appgroups]% add amd [basecm10->kubernetes*[default*]->appgroups*[amd*]]% applications [basecm10->kubernetes*[default*]->appgroups*[amd*]->applications]% add device-plugin ``` The configuration of the plugin can be set to the YAML file, by setting the `config` parameter to take the value of the YAML file path. **Example** ```bash [basecm10->...[amd*]->applications*[device-plugin*]]% set config /tmp/k8s-ds-amdgpu-dp.yaml [basecm10->kubernetes*[default*]->appgroups*[amd*]->applications*[device-plugin*]]% show Parameter Value -------------------------------- ------------------------------------------------ Name device-plugin Revision Format Yaml Enabled yes Config <914B> Environment <0 in submode> Exclude list snippets <0 in submode> [basecm10->kubernetes*[default*]->appgroups*[amd*]->applications*[device-plugin*]]% commit ``` The YAML file can also be edited within `cmsh` after it has been set, by running `set config` without a value. There are older releases available, starting from Kubernetes v1.10, if needed. Saving this device-plugin YAML should result in pods being scheduled on all the non-tainted nodes, as seen by listing the pods (some columns elided): ```bash [root@basecm10 ~]# kubectl get pod -n kube-system -l name=amdgpu-dp-ds -o wide NAME READY STATUS ... IP NODE ... amdgpu-device-plugin-daemonset-66jl7 1/1 Running ... 172.29.112.135 gpu001 ... amdgpu-device-plugin-daemonset-8mh9w 1/1 Running ... 172.29.152.130 gpu002 ... ``` ### 4.4.3 Including Head Nodes as part of the DaemonSet: BCM taints head nodes, so that they do not run non-critical pods. The taint can be removed with the “`-`” operator to allow non-critical pods to run: **Example** ```bash kubectl taint nodes basecm10 node-role.kubernetes.io/control-plane- ``` However, a more specific exception can be configured in the DaemonSet itself. Within the YAML file, the following existing `tolerations` definition has to be modified, from: ```bash tolerations: - key: CriticalAddonsOnly operator: Exists ``` to: ```bash tolerations: - key: node-role.kubernetes.io/control-plane effect: NoSchedule operator: Exists ``` The modified toleration tolerates this taint, and therefore has the device plugin run on such tainted nodes. #### 4.4.3.1 Verifying That AMD GPUs Are Recognized By Kubernetes If Kubernetes is aware of the AMD GPUs for a node then several mentions of `amd.com/gpu` are displayed when running the `kubectl describe node` command for the node. The following session shows output for a node `gpu01`, ellipsized for clarity: **Example** ```bash [root@basecm10 ~]# kubectl describe node gpu01 Name: gpu01 ... Capacity: amd.com/gpu: 3 cpu: 64 ephemeral-storage: 1813510Mi hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 527954676Ki pods: 50 ... ``` ### 4.4.4 Running The DaemonSet Only On Specific Nodes The AMD GPU device plugin, unlike the NVIDIA GPU device plugin Daemonset, is scheduled to run on each Kubernetes host. This means that it runs even if the host has no GPU. This can be prevented with the following steps: A LabelSet can be created via `cmsh`, and the nodes or categories that have GPUs are assigned within the `labelsets` mode: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% kubernetes [basecm10->kubernetes[default]]% labelsets [basecm10->kubernetes[default]->labelsets]% use nvidia [basecm10->kubernetes[default]->labelsets[nvidia]]% .. #but, we're using AMD GPUs, so let's go back up: [basecm10->kubernetes[default]->labelsets]% add amd [basecm10->kubernetes*[default*]->labelsets*[amd*]]% set labels nvidia.com/amd-gpu-accelerator= [basecm10->kubernetes*[default*]->labelsets*[amd*]]% append categories gpu-nodes [basecm10->kubernetes*[default*]->labelsets*[amd*]]% commit ``` This assigns the labels to the nodes with GPUs. This can be verified with: **Example** ```bash kubectl get nodes -l nvidia.com/amd-gpu-accelerator= NAME STATUS ROLES AGE VERSION gpu001 Ready master 66m v1.18.8 gpu002 Ready master 66m v1.18.8 ... ``` The DaemonSet YAML can now be adjusted to only run the device plugin on nodes with this new label. This can be done by adding an affinity block after the tolerations block: **Example** ```bash tolerations: - key: CriticalAddonsOnly # toleration may be different, if changes were made to it operator: Exists affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: 'nvidia.com/amd-gpu-accelerator' operator: Exists ``` This results in the device plugin pods being removed immediately from all nodes that do not have the label. ### 4.4.5 Running An Example Workload An example workload can be run as described in the official AMD GPU Kubernetes device plugin documentation at: [https://github.com/RadeonOpenCompute/k8s-device-plugin/tree/v1.16#example-workload](https://github.com/RadeonOpenCompute/k8s-device-plugin/tree/v1.16#example-workload) Thus it should now be possible to run: ```bash [root@basecm10 ~]# kubectl create -f https://raw.githubusercontent.com/RadeonOpenCompute/ k8s-device-plugin/v1.16/example/pod/alexnet-gpu.yaml ``` The YAML requests only one GPU at the bottom of the YAML file: ```bash apiVersion: v1 kind: Pod metadata: name: alexnet-tf-gpu-pod labels: purpose: demo-tf-amdgpu spec: containers: - name: alexnet-tf-gpu-container image: rocm/tensorflow:latest workingDir: /root env: - name: HIP_VISIBLE_DEVICES value: "0" # # 0,1,2,...,n for running on GPU and select the GPUs, -1 for running on CPU command: ["/bin/bash", "-c", "--"] args: ["python3 benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py --model=alexnet;\ trap : TERM INT; sleep infinity & wait"] resources: limits: amd.com/gpu: 1 # requesting a GPU ``` Container creation might take a while due to the image size. Once scheduled, it prints out that it found exactly one GPU, and proceeds to run a TensorFlow workload. **Example** ```bash [root@basecm10 ~]# kubectl logs -f alexnet-tf-gpu-pod WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version. Instructions for updating: non-resource variables are not supported in the long term 2021-01-08 21:03:29.222293: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2495445000 Hz 2021-01-08 21:03:29.222398: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x39f62f0 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2021-01-08 21:03:29.222420: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2021-01-08 21:03:29.223754: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libamdhip64.so 2021-01-08 21:03:31.635339: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3a40bb0 initialized for platform ROCM (this does not guarantee that XLA will be used). Devices: 2021-01-08 21:03:31.635363: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Device 738c, AMDGPU ISA version: gfx908 2021-01-08 21:03:31.931125: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 0 with properties: pciBusID: 0000:27:00.0 name: Device 738c ROCm AMD GPU ISA: gfx908 coreClock: 1.502GHz coreCount: 120 deviceMemorySize: 31.98GiB deviceMemoryBandwidth: 1.12TiB/s ... TensorFlow: 2.3 Model: alexnet Dataset: imagenet (synthetic) Mode: training SingleSess: False Batch size: 512 global 512 per device Num batches: 100 Num epochs: 0.04 Devices: ['/gpu:0'] ... ``` Had more GPUs been requested, more would have been made available to the container. For comparison, a CPU version of the container is also available. The official instructions can be referred to for these, too. ## 4.5 Kubernetes Configuration Overlays A list of configuration overlays can be seen from within `configurationoverlay` mode: **Example** ```bash [basecm10->configurationoverlay]% list Name (key) Priority Nodes Categories Roles ------------------- -------- ----------------- ---------- -------------------------... kube-default-etcd 500 node001..node003 Etcd::Host kube-default-master 510 node001..node003 generic::containerd, kube... kube-default-worker 500 node004..node006 default generic::containerd, kube... kube-default-netq 501 node001 generic::netq ``` The NetQ configuration overlay is only normally displayed if NetQ is deployed. Configuration overlays can be used to manage the Kubernetes services used with a particular configuration. For example, when managing the Kubernetes services used for a Kubernetes engine within an Auto Scale tracker ([section 8.4.9 of the *Administrator Manual*](/admin/cm-scale#849-auto-scaling-a-job-on-premises-to-a-workload-manager-and-kubernetes)). ## 4.6 Removing A Kubernetes Cluster A Kubernetes cluster can be removed using `cm-kubernetes-setup` with the `--remove` and `--yes-i-really-mean-it` options. Also, if there more than one cluster present, then the cluster name must be specified using the `--cluster` parameter. A removal run looks as follows (some output ellipsized): **Example** ```bash [root@basecm10 ~]# cm-kubernetes-setup --remove --cluster default --yes-i-really-mean-it Connecting to CMDaemon Executing 20 stages ################### Starting execution for 'Kubernetes Setup' - kubernetes - docker ## Progress: 0 #### stage: kubernetes: Get Kube Cluster ## Progress: 5 #### stage: kubernetes: Check Kube Cluster Exists ## Progress: 10 #### stage: kubernetes: Find Installed Components ## Progress: 15 #### stage: kubernetes: Find Files On Headnodes ## Progress: 20 #### stage: kubernetes: Firewall Zone Close ## Progress: 25 #### stage: kubernetes: Firewall Interface Close ## Progress: 30 #### stage: kubernetes: Firewall Policy Close ## Progress: 35 #### stage: kubernetes: Nginx Reverse Proxy Close ## Progress: 40 #### stage: kubernetes: IP Ports Close ## Progress: 60 #### stage: kubernetes: Remove Installed Components ## Progress: 65 #### stage: kubernetes: Remove Files On Headnodes ## Progress: 70 #### stage: kubernetes: Remove Etcd Spool ## Progress: 80 #### stage: kubernetes: Set Reboot Required You need to reboot 2 nodes to cleanup the network configuration ## Progress: 85 #### stage: kubernetes: Collection Update Provisioners ## Progress: 100 Took: 00:08 min. Progress: 100/100 ################### Finished execution for 'Kubernetes Setup', status: completed Kubernetes Setup finished! ``` Using the `--remove` option removes the Kubernetes cluster configuration from BCM, unassigns Kubernetes-related roles—including the EtcdHost role—and removes Kubernetes health checks. The command does not remove packages that were installed with a `cm-kubernetes-setup` command before that. After the disabling procedure has finished, the cluster has no Kubernetes configured and running. ## 4.7 Kubernetes Cluster Configuration Options Kubernetes allows many Kubernetes clusters to be configured. These are separated sets of hosts with different certificates, users and other global settings. When carrying out the Kubernetes setup run, a Kubernetes cluster name is asked, and a new object with the cluster settings is then added into the CMDaemon configuration. The administrator can change the settings of the cluster from within the `kubernetes` mode of `cmsh`, or within the `Kubernetes Clusters` options window of Base View accessible via the navigation path `Containers` > `Kubernetes Clusters`. The `cmsh` equivalent looks like: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% kubernetes list Name (key) ------------------ default [basecm10]% kubernetes use default [basecm10->kubernetes[default]]% show Parameter Value ------------------------------------ ------------------------------------------------------ Name default Revision Etcd Cluster kube-default Pod Network kube-default-pod Pod Network Node Mask Internal Network internalnet KubeDNS IP 10.150.255.254 Kubernetes API server Kubernetes API server proxy port 10443 App Groups <1 in submode> Label Sets <3 in submode> Notes Version 1.27.10-150500.1.1 Trusted domains basecm10.openstacklocal,master,localhost,10.141.255.254 Module file template <1.01KiB> Kubeadm init file <1.19KiB> Service Network kube-default-service Kubeadm CERT Key ********* Kube CA Cert ********* Kube CA Key ********* Kubernetes users <0 in submode> External no External Kubernetes Ingress server External port 0 Capi template no Capi namespace default Kubernetes management cluster [basecm10->kubernetes[default]]% ``` The preceding `kubernetes` mode parameters are described in table [4.1](#kubernetes-mode-parameters): | **Parameter** | **Description** | | :----------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `App Groups` | Groups of Kubernetes add-ons managed by CMDaemon. | | `CAPI namespace` | Namespace where CAPI is deployed (optional). | | `CAPI template` | Is this Kubernetes cluster configuration a template for CAPI workload clusters? | | `Etcd Cluster` | The etcd cluster instance. | | `External` | Is this Kubernetes cluster configuration for an external Kubernetes instance (e.g., cloud)? | | `External Kubernetes Ingress server` | The Ingress endpoint for the external Kubernetes. | | `External port` | Additional port used by external Kubernetes cluster (unused). | | `Internal Network` | Network to back the internal communications. | | `Kube CA Cert` | Path to PEM-encoded RSA or ECDSA certificate used for the CA | | `Kube CA Key` | Path to PEM-encoded RSA or ECDSA private key used for the CA | | `KubeDNS IP` | CoreDNS IP Address. | | `Kubeadm Cert Key` | Key used to encrypt the control plane certificates in the kubeadm-certs Secret. | | `Kubeadm Init file` | Contents of the init configuration file provided to Kubeadm to initialize or join nodes. | | `Kubernetes API Server` | Kubernetes API server address (format: `https://` \<*host*>`:<`*port number*>). | | `Kubernetes API Server Proxy Port` | Kubernetes API server proxy (NGINX LoadBalancer) port (default: 10443). | | `Kubernetes Management cluster` | Relevant to CAPI only: the Kubernetes cluster managing this CAPI workload cluster (optional). | | `Kubernetes Users` | Submode to manage users for this Kubernetes cluster (e.g.: whether or not to manage their Kube config files) | | `Label Sets` | Submode to manage assignment of labels to Kubernetes nodes | | `Module file template` | Template used for writing the Kubernetes TCL module file. | | `Name` | The name, identifier, or label for the Kubernetes cluster. | | `Pod Network` | Network where pod IP addresses are assigned from | | `Pod Network Node Mask` | Corresponding subnet mask used to define the size of the address space allocated to each node for its pods | | `Service Network` | Network from which the service cluster IP addresses are assigned, in IPv4 CIDR format. Must not overlap with any IP address ranges assigned to nodes for pods. Default: `172.29.0.0/16` | | `Trusted Domains` | Trusted domains to be included in Kubernetes-related certificates as Alternative Subject Names. | | `Version` | Version of the Kubernetes cluster. | *Table 4.1: `kubernetes` mode parameters* ## 4.8 EtcdCluster The EtcdCluster mode sets the global etcd cluster settings. It can be accessed via the top level `etcd` mode of `cmsh`.
**Parameter** **Description** **Option to `etcd`**
`Name` etcd cluster name. `--initial-cluster-token`
`Election Timeout` Election timeout, in milliseconds. `--election-timeout`
`Heart Beat Interval` Heart beat interval, in milliseconds. `--heartbeat-interval`
`CA` The certificate authority (CA) Certificate path for etcd, used to generate certificates for etcd. `--peer-trusted-ca-file`
`CA Key` The CA Key path for etcd, used to generate certificates for etcd.
`Member Certificate` The certificate path to use for etcd cluster members, signed with the etcd CA. The EtcdHost role can specify a member CA as well, and in that case it overwrites any value set here. `--peer-cert-file`
`Member Certificate Key` The key path to use for etcd cluster members, signed with the etcd CA. The EtcdHost role can specify a member CA as well, and in that case it overwrites any value set here. `--peer-key-file`
`Client CA` The CA used for client certificates. When set it is assumed client certificate and key are generated and signed with this CA by another party. etcd still expects the path to be correct for the client certificate and key. `--trusted-ca-file`
`Client Certificate` The client certificate, used by etcdctl, for example. `--cert-file`
`Client Certificate Key` The client certificate key, used by etcdctl for example. `--key-file`
\* Boolean (takes `yes` or `no` as a value) *Table 4.2: EtcdCluster role parameters and etcd options* ## 4.9 Kubernetes Roles Kubernetes roles include the following roles: * `EtcdHost` ([here](#491-etcdhost-role)) * `KubernetesApiServerProxy` ([here](#492-the-kubernetesapiserverproxy-role)) * `Kubelet` ([here](#493-the-kubelet-role)) * `generic::containerd` ([here](#494-containerd-role)) * `generic::netq` ([here](#45-kubernetes-configuration-overlays)) When nodes are configured using Kubernetes roles, then settings in these roles may sometimes use the same values (pointer variables). **Example** ```bash [basecm10->configurationoverlay[kube-default-etcd]->roles[Etcd::Host]]% get etcdcluster kube-default ``` and ```bash [basecm10->kubernetes[default]]% get etcdcluster kube-default ``` Pointer variables such as these have definitions that are shared across the roles, as indicated by the parameter description tables for the roles, and which are described in the following pages. In `cmsh`, the roles can be assigned: * for individual nodes via the `roles` submode of `device` mode * for a category via the `roles` submode of a category * for a configuration overlay via the `roles` submode of `configurationoverlay` mode ### 4.9.1 EtcdHost Role The EtcdHost role is used to configure and manage the `etcd` service for a node. The `etcd` service manages the `etcd` database, which is a hierarchical distributed key-value database. The database is used by Kubernetes to store its configurations. The EtcdHost role parameters are described in table [4.3](#etcdhost-role-parameters-table):
**Parameter** **Description** **Option to `etcd`**
`Member Name` The human-readable name for this `etcd` member ( `$hostname` is replaced by the node hostname) `--name`
`Spool` Path to the data directory (default: `/var/lib/etcd` ) `--data-dir`
`Advertise Client` List of client URLs for this member to `--advertise-client-urls`
`URLs` advertise publicly (default: `http://$hostname:5001` )
`Advertise Peers` List of peer URLs for this member to `--initial-advertise-peer-urls`
`URLs` advertise to the rest of the cluster (default: `http://$hostname:5002` )
`Listen Client URLs` List of URLs to listen on for client traffic (default: `http://$hostname:5001` , [http://127.0.0.1:2379](http://127.0.0.1:2379) ) `--listen-client-urls`
`Listen Peer URLs` List of URLs to listen on for peer traffic (default: `http://$hostname:5002` ) `--listen-peer-urls`
`Snapshot Count` Number of committed transactions that trigger a snapshot to disk (default: `5000` ) `--snapshot-count`
`Debug` \* Drop the default log level to DEBUG for all subpackages? (default: `no` ) `--debug`
`Member Certificate` etcd member certificate, signed with CA specified in the etcd cluster. Setting it overrules the value set in the EtcdCluster object ( `etcd` mode, section [4.8](#48-etcdcluster) ). Default empty. `--peer-cert-file`
`Member Certificate Key` etcd member certificate key, signed with CA specified in the etcd cluster. Setting it overrules the value set in the EtcdCluster object ( `etcd` mode, section [4.8](#48-etcdcluster) ). Default empty. `--peer-key-file`
`Options` Additional parameters for the `etcd` daemon (empty by default)
\* Boolean (takes `yes` or `no` as a value) *Table 4.3: EtcdHost role parameters and `etcd` options* The `etcd` settings are updated by BCM in `/cm/local/apps/etcd/current/etc/cm-etcd.conf`. ### 4.9.2 The KubernetesAPIServerProxy Role The KubernetesApiServerProxy role sets up a proxy that provides the entry point for one or more instances of the Kubernetes API server. The proxy runs on every node of a Kubernetes cluster instance, including the head node. If multiple Kubernetes master nodes are present, then it enables HA for the Kubernetes master components (section [4.1](#41-reference-architecture)). This means that local port 10443 on all these nodes load balances over all control-plane node API servers. Port 10443 can also be opened up for external access. The `cm-kubernetes-setup` wizard (section [4.2](#42-kubernetes-setup)) in one of its screens (figure [4.5](#fig-kube-external-network)) prompts for this. ![Wizard question](/base-command-manager/product-manuals/_fern-img/d2841972517dfb446ffe3a869c7e541e65873e91812ae2b8be88ac1ee20ce07d.webp) *Figure 4.5: Wizard question* If the prompt is answered with `no`, then the firewall role on the head node can still be customized with `cmsh` later on: **Example** ```bash root@basecm10:~# cmsh [basecm10]% device use master [basecm10->device[basecm10]]% roles [basecm10->device[basecm10]->roles]% use firewall [basecm10->device[basecm10]->roles[firewall]]% openports [basecm10->device[basecm10]->roles[firewall]->openports]% list Index Action Network Port Destination Protocol Address Description ------ --------- ------------ ------------ ------------ --------- ------------ ------------------------ 0 ACCEPT net 30080 fw TCP 0.0.0.0/0 Kubernetes 'default' In+ 1 ACCEPT net 30443 fw TCP 0.0.0.0/0 Kubernetes 'default' In+ [basecm10->device[basecm10]->roles[firewall]->openports]% add ACCEPT net 10443 tcp fw ok. [basecm10->device*[basecm10*]->roles*[firewall*]->openports[2]]% commit [basecm10->device[basecm10]->roles[firewall]->openports[2]]% list Index Action Network Port Destination Protocol Address Description ------ --------- ------------ ------------ ------------ --------- ------------ ------------------------ 0 ACCEPT net 30080 fw TCP 0.0.0.0/0 Kubernetes 'default' In+ 1 ACCEPT net 30443 fw TCP 0.0.0.0/0 Kubernetes 'default' In+ 2 ACCEPT net 10443 fw TCP 0.0.0.0/0 [basecm10->device[basecm10]->roles[firewall]->openports[2]]% Thu Oct 31 15:58:56 2024 [notice] basecm10: Service shorewall was restarted ``` #### 4.9.2.1 Ingress Server Proxying Using The Default HTTPS Port 443 In BCM 11 a new `KubernetesIngressServerProxy` role is introduced to allow Ingress Server Proxying. In BCM 10, to maintain backward compatibility, new roles cannot be introduced. The proxying functionality of the Ingress server is therefore partly dealt with using the KubernetesApiServerProxy role. The proxy provides the entry point for one or more instances of the Kubernetes Ingress server, and runs on every node of a Kubernetes cluster instance, including the head node. If multiple Kubernetes master nodes are present, then the role enables HA for the Kubernetes master components (section [4.1](#41-reference-architecture)). Before BCM 10.24.11, the Kubernetes Ingress Server bundled NGINX Ingress server as an option, and used non-standard HTTPS ports. The defaults have always been 30080 for HTTP, and 30443 for HTTPS as NodePort services ([https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport](https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport)). These ports are customizable, but a limitation of NodePort services is that they may only use the port range 30000–-32767. Starting with BCM 10.24.11, during the `cm-kubernetes-setup` session to install Kubernetes, there is a screen which asks if the Ingress Server should be available on the default HTTPS port 443 (figure [4.6](#fig-kube-external-ingress)). It is recommended to respond with `yes`, because nowadays services running inside Kubernetes are commonly run on the default HTTPS port. This in turn results in user redirects from the non-standard port (for example, 30443) to port 443. Using the default port is also more in line with how Ingress is offered by cloud-based solutions that provide Kubernetes. ![Wizard question](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/6c1becda3aa761753fb2ff7a0e5e3ecc91ffec7064ed83302bd39a2f0680ef97/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/kube-external-ingress.png) *Figure 4.6: Wizard question* Setting 443 for the proxy ensures that local port 443 on all the concerned nodes carries out loadbalancing over all the control-plane node Kubernetes Ingress Servers. It also ensures that the port is opened in the firewall. Setting `no` means that an open port can still be configured later on by carrying the 4 steps that follow shortly. The discerning cluster administrator should be able to see from these steps how the Ingress server proxying functionality is implemented without the additional role. The extra parameters are set per KubeCluster entity inside BCM. This is because Ingress ports can differ between multiple installations of Kubernetes. The steps: 1. **Ensuring HTTPS is not already in use:** If port 443 it is already in use by BCM (for example, if the BCM landing page is served on port 443 via `apache2` or `httpd`), then that port must be changed or disabled. For Ubuntu-based systems, this means commenting out `Listen 443` in `/etc/apache2/ports.conf`. For RHEL-based systems, that line is located in `/etc/httpd/conf.d/ssl.conf`. The web service should then be restarted (`apache2` for Ubuntu, `httpd` for RHEL and Rocky). 2. **Assigning the configuration to the KubeCluster entities:** This can be done as follows in `cmsh`: **Example** This is the KubeCluster entity that the parameters are added to: ```bash root@basecm10:~# cmsh [basecm10]% kubernetes [basecm10->kubernetes[default]]% show Parameter Value ------------------------------------ -------------------------------------------------- Name default Revision Etcd Cluster kube-default Pod Network kube-default-pod Pod Network Node Mask Internal Network internalnet KubeDNS IP 10.150.255.254 ... ``` The following parameters are set next: * `ingress_proxy_backend_port`: a fallback port in case the NGINX NodePort cannot be read from the applications * `ingress_proxy_listen_port`: the port the proxy should listen on (default HTTPS port is 443) * `ingress_proxy_enable`: a flag to enable the proxying functionality In `cmsh` this can be carried out with special `-e|--extra` options as follows, which set up new parameters in the kubernetes instance: **Example** ```bash [basecm10->kubernetes[default]]% set -e ingress_proxy_backend_port 30443 [basecm10->kubernetes*[default*]]% set -e ingress_proxy_listen_port 443 [basecm10->kubernetes*[default*]]% set -e ingress_proxy_enable true [basecm10->kubernetes*[default*]]% commit [basecm10->kubernetes[default]]% show Parameter Value ------------------------------------ ------------------------------------------------------------------- ingress_proxy_backend_port 30443 ingress_proxy_enable true ingress_proxy_listen_port 443 Name default Revision Etcd Cluster kube-default Pod Network kube-default-pod Pod Network Node Mask Internal Network internalnet KubeDNS IP 10.150.255.254 ... ``` In BCM 10, back in the KubernetesApiServerProxy role, a commit then triggers the NGINX configuration reload: ```bash [basecm10->kubernetes[default]]% device use master [basecm10->device[basecm10]]% roles [basecm10->device[basecm10]->roles]% use kubernetes::apiserverproxy [basecm10->device[basecm10]->roles[Kubernetes::ApiServerProxy]]% commit [basecm10->device[basecm10]->roles[Kubernetes::ApiServerProxy]]% Fri Nov 22 14:49:15 2024 [notice] basecm10: Service nginx was reloaded ``` As the preceding session implies, the `/etc/nginx/nginx.conf` file and `nginx` service are affected. Local port 443 should now load balance the Kubernetes Ingress service to all control-plane nodes. 3. **Opening the HTTPS port in the firewall:** The port in the default firewall, Shorewall ([s](/installation/thirdparty#72-shorewall)ection [7.2](/installation/thirdparty#72-shorewall) of the *Installation Manual*), can be modified using the `firewall` role in `cmsh`. For historic reasons, ports 30080 and 30443 are still opened up by default in BCM 10. These ports can also be removed from the entries for the Ingress controller, as shown in the following session: ```bash root@basecm10:~# cmsh [basecm10]% device use master [basecm10->device[basecm10]]% roles [basecm10->device[basecm10]->roles]% use firewall [basecm10->device[basecm10]->roles[firewall]]% openports [basecm10->device[basecm10]->roles[firewall]->openports]% list -v Index Action Network Port Destination Protocol Address Description ------ --------- -------- -------- ------------ --------- ------------ ------------------------ 0 ACCEPT net 30080 fw TCP 0.0.0.0/0 Kubernetes 'default' Ingress HTTP 1 ACCEPT net 30443 fw TCP 0.0.0.0/0 Kubernetes 'default' Ingress HTTPS 2 ACCEPT net 10443 fw TCP 0.0.0.0/0 [basecm10->device[basecm10]->roles[firewall]->openports]% remove 0 [basecm10->device*[basecm10*]->roles*[firewall*]->openports*]% remove 1 [basecm10->device*[basecm10*]->roles*[firewall*]->openports*]% add ACCEPT net 443 tcp fw ok. [basecm10->device*[basecm10*]->roles*[firewall*]->openports*[3]]% list Index Action Network Port Destination Protocol Address Description ------ --------- -------- -------- ------------ --------- ------------ ------------ 2 ACCEPT net 10443 fw TCP 0.0.0.0/0 3 ACCEPT net 443 fw TCP 0.0.0.0/0 [basecm10->device*[basecm10*]->roles*[firewall*]->openports*[3]]% .. [basecm10->device*[basecm10*]->roles*[firewall*]->openports*]% commit [basecm10->device[basecm10]->roles[firewall]->openports]% Thu Oct 31 16:35:19 2024 [notice] basecm10: Service shorewall was restarted [basecm10->device[basecm10]->roles[firewall]->openports]% list Index Action Network Port Destination Protocol Address Description ------ --------- -------- -------- ------------ --------- ------------ ------------ 0 ACCEPT net 10443 fw TCP 0.0.0.0/0 1 ACCEPT net 443 fw TCP 0.0.0.0/0 ``` As seen in the preceding session, this immediately affects the `/etc/shorewall/rules` file and `shorewall` service. 4. **Fixing the BCM landing page:** Finally the landig page YAML file is applied. The file should be available on the head nodes for BCM version 10.24.11 and beyond. ```bash root@basecm10# kubectl apply -f \ /cm/local/apps/cm-setup/lib/python3.12/site-packages/cmsetup/plugins/kubernetes/addons/1.27/landingpage.yaml configmap/nginx-proxy-config created deployment.apps/nginx-proxy created service/nginx-proxy-service created Warning: annotation "kubernetes.io/ingress.class" is deprecated, please use 'spec.ingressClassName' instead ingress.networking.k8s.io/default-ingress created ``` In the preceding session, a simple Deployment ([https://kubernetes.io/docs/concepts/workloads/controllers/deployment/](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/)) is deployed. The Deployment manages a single pod that runs an NGINX container with the configuration provided as a ConfigMap ([https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/)). This configuration sets a redirect to the single `http://master` backend, where the BCM landing page ([section 2.4.1 of the *Administrator Manual*](/admin/clustermanagement#241-installing-the-cluster-management-gui-service)) is running, and running with the HTTP rather than the HTTPS protocol. The service exposes the pod, and an Ingress rule is created and used as a fallback/default handler for the Kubernetes NGINX Ingress server. This way, if no other Ingress rule matches, then the BCM landing page is served via the HTTPS port. In other words, the page is served when there is no other matching domain name or other path-based Ingress rule to take priority over the landing page. This means that the landing page is still served when the head node IP address is entered directly in the browser. ### 4.9.3 The Kubelet Role The Kubelet role is used to configure and manage the `kubelet` service. BCM takes care of joining these nodes with `kubeadm join` when needed. **Control Plane** The Kubelet role has a parameter `Control plane` that is set to the value `yes` for control plane nodes. In that case it also runs Kubernetes control plane services, such as: * [Kubernetes API server (`kube-apiserver`)](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/), * [Kubernetes scheduler (`kube-scheduler`)](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/), * [Kubernetes controller manager (`kube-controller-manager`)](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/), * [Kubernetes network proxy (`kube-proxy`)](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/), * [CoreDNS (`coredns`)](https://coredns.io/manual/toc/#what-is-coredns). **Workers** The Kubelet role has a parameter `worker` that is set to `yes` for worker nodes. In that case, the control plane pods will not be running on the node. The Kubelet role parameters are described in table [4.4](#kubelet-role-parameters-table):
**Parameter** **Description**
`Kubernetes Cluster` The Kubernetes cluster instance (a pointer)
`Control plane` Is Kubelet running services on this node, making it a control plane node?
`Worker` Kubelet is a worker flag
`Max Pods` Configuration to change max number of pods on Kubelet
`Options` Submode that allows configuration of additional flags to containers specified in manifest `/etc/kubernetes/manifests`
\* Boolean (takes `yes` or `no` as a value) *Table 4.4: Kubelet role parameters and `kubelet` options* Further details on the Kubelet service can be found at [https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/). #### 4.9.3.1 The `options` Submode ```bash [basecm10->configurationoverlay[kube-default-master]->roles[kubelet]]% help options Name: options - Manage kubelet container command options Usage: options [kubelet] options [kubelet] options [kubelet] set [name] key=value options [kubelet] set [name] key options [kubelet] clear [name] key Examples: options kube-apiserver.yaml set advertise-address=192.168.200.148 [basecm10->configurationoverlay[kube-default-master]->roles[kubelet]]% options Filename Container Key Value ------------------- --------------- --------------------- ------------------------------------ kube-apiserver.yaml kube-apiserver cors-allowed-origins https://ui.example.com kube-apiserver.yaml kube-apiserver oidc-client-id kubernetes kube-apiserver.yaml kube-apiserver oidc-issuer-url https://sso.example.com/realms/bcm kube-apiserver.yaml kube-apiserver oidc-username-prefix - ``` The full path to the files should be specified for the `options` command. For example, `kube-apiserver.yaml` may be specified as `/etc/kubernetes/manifests/kube-apiserver.yaml`. ### 4.9.4 Containerd Role The Containerd role is used to configure and manage the `containerd` daemon. This is done through a generic role in BCM. Generic roles under Kubernetes are found under the `cmsh` path indicated by: `cmsh > configurationoverlay[`*`kube-object`*`] > roles[`*`generic::role`*`]` **Example** ```bash [basecm10->configurationoverlay[kube-default-master]->roles[generic::containerd]]% show Parameter Value -------------------------------- ------------------------------------------------ Name generic::containerd Revision Type GenericRole Add services yes Services containerd Configurations <2 in submode> Environments <1 in submode> Exclude list snippets <2 in submode> Data node no ``` The role has several submodes: * The `configurations` submode: This contains various configuration drop-ins. Configuration drop-ins that BCM may manage in the `generic::containerd` role are: * NVIDIA Container Toolkit configuration. * Docker registry certs directory configuration. * Docker Hub credentials configuration. * Harbor Registry Mirror configuration. * CRI cgroup configuration. * CNI bin dir configuration. * CRI registry configuration. * CRI custom sandbox image configuration. The drop-ins that are actually available depend on choices made by the cluster administrator during deployment. * The `environment` submode: This has optional environment variables that are made available to the configuration files and that are to be used as templates. * The `excludelistsnippets` submode: This has files related to containerd that need to be excluded, such as the most obvious directory `/var/lib/containerd`. ## 4.10 Security Model The Kubernetes security model allows authentication using a certificate authority (CA), with the user and daemon certificates signed by a Kubernetes CA. The Kubernetes CA should not be confused with the BCM CA. BCM lets `kubeadm` create a CA specifically for issuing all Kubernetes-related certificates. The certificates are put into `/etc/kubernetes/pki//` by default. In Kubernetes terminology a user is a unique identity accessing the Kubernetes API server. The user may be a human or an automated process. For example an admin or a developer are human users, but kubelet represents an infrastructure user. Both types of users are authorized and authenticated in the same way against the API server. Kubernetes uses client certificates, tokens, or HTTP basic authentication methods to authenticate users for API calls. BCM configures client certificate usage by default. The authentication is performed by the API server which validates the user certificate using the common name part of the certificate subject. In Kubernetes, authorization happens as a separate step from authentication. Authorization applies to all HTTP accesses on the main (secure) API server port. BCM by default enables RBAC (Role-Based Access Control) combined with Node Authorization. The authorization check for any request thus takes the common name and/or organization part of the certificate subject to determine which roles the user or service has associated. Roles carry a certain set of privileges for resources within Kubernetes. ### 4.10.1 Kyverno BCM has support for the Kyverno policy engine ([https://kyverno.io/](https://kyverno.io/)). If Kyverno is installed, then Kubernetes Permissions Manager (section [4.15](#415-kubernetes-permission-manager)) creates policy manifests packed as a Helm chart for every user added to Kubernetes via `cm-kubernetes-setup`. In addition, a kyverno-policy chart is installed in enforce mode to implement Pod Security Standards ([https://kyverno.io/policies/](https://kyverno.io/policies/)). During installation, some exclusions are added to the policies automatically to make chosen features of Kubernetes cluster work. For every created user the following defaults are applied: * The user has an associated service account with the same name * A \<*username*>-restricted namespace is created. So, for a user `john` the namespace is `john-restricted`. * An `edit` cluster role is bound to the service account in \<*username*>-restricted namespace. The user is allowed to create pods, services, configmaps, etc. in the namespace * The user is allowed to list nodes in the cluster * Kyverno policies are applied to the resources in \<*username*>-restricted namespace or to pod created or updated by the associated user * If hostPath is not the home directory of the user (of the format `/home/` \<*username*>) then the creation of the resource is denied * The UID and GID of the running process are set to the same value as the UID and GID of the PAM user Modifications from the defaults are: * If the `Allow any UID process in pods` checkbox is ticked, or if the `--allow-all-uids` argument is specified, then the UID and GID of the running process becomes the user’s UID and GID only if the hostPath volume is specified. Otherwise it can be set to any UID and GID. * Cluster roles can be set not only to * `edit` but also to * `view` * `admin` * `cluster-admin` More details on these roles can be found at:\ [https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles). * In addition, the user can be given access to custom CRDs, such as Postgres Operator, Jupyter Operator or Google Spark Operator ### 4.10.2 PodSecurityPolicy PodSecurityPolicy (PSP) was available in older versions of BCM. However, PSP support was completely removed in Kubernetes v1.25. BCM therefore now uses Kyverno (section [4.10.1](#4101-kyverno)) for an equivalent functionality. ## 4.11 Addition Of New Kubernetes Users BCM users can use Kubernetes by making them Kubernetes users. This means having Kubernetes configuration and access set up for them. This can be carried out a via the `cm-kubernetes-setup` TUI utility, and choosing the `Add user` option (figure [4.3](#fig-kube-urwid-wizard-operations)). The utility then prompts for * a Kubernetes cluster * a user name * a namespace that the privileges are to be assigned to * a role for the user, with choices provided from: * `cluster-admin`: cluster-wide administrator * `admin`: administrator * `edit`: regular user * `view`: read-only user * a switch if the user is allowed to run as any user, including root, inside pods * a comma-separated list of paths that the user is able to mount to pods * the UIDs and GIDs for user processes in pods * a list of the Kubernetes operators that a user can use Based on the input, a YAML for the Kubernetes Permission Manager is generated. This in turn, creates a Helm chart with all the required roles, role bindings, and Kyverno rules. Creation of the user also triggers CMDaemon to create certificate and configuration files in the `~/.kube` directory ### 4.11.1 Adding Users Non-Interactively With `cm-kubernetes-setup` The `cm-kubernetes-setup` CLI wizard provides the following options: ```bash cm-kubernetes-setup -h usage: Kubernetes Setup cm-kubernetes-setup [-c ] [--cluster CLUSTER_NAME] [--skip-docker] [--skip-reboot] [--skip-image-update] [--add-user USERNAME_ADD] [--list-users] [--get-user GET_USER] [--modify-user USERNAME_MODIFY] [--remove-user USERNAME_REMOVE] [--namespace NAMESPACE] [--add-to-namespace] [--remove-from-namespace] [--role edit,admin,view,cluster-admin] [--runas-uid RUNAS_UID] [--runas-gids RUNAS_GIDS] [--user-paths USER_PATHS] [--allow-all-uids] [--operators OPERATORS] [--backup-permissions FILE] [--restore-permissions FILE] [--list-operators] [--update-addons] [--remove] [--yes-i-really-mean-it] [--remove-ceph-storage] [--pull] [--images IMAGES] [--nodes NODES] [--node-selector NODE_SELECTOR] [--pull-registry-server PULL_REGISTRY_SERVER] [--pull-registry-username PULL_REGISTRY_USERNAME] [--pull-registry-email PULL_REGISTRY_EMAIL] [--pull-registry-password PULL_REGISTRY_PASSWORD] [-v] [--no-distro-checks] [--json] [--output-remote-execution-runner] [--on-error-action debug,remotedebug,undo,abort] [--skip-packages] [--min-reboot-timeout ] [--allow-running-from-secondary] [--dev] [-h] ... ``` The user has to be a user that exists on the cluster already and available via PAM. If `--add-to-namespace` is specified, then the namespace has to exist on the Kubernetes cluster already. **Example** ```bash cm-kubernetes-setup --add-user john ``` The preceding example creates a user `john` for the default `john-restricted` namespace. It also assigns the `edit` role, and gives permission to run processes in the pod with the current UID/GIDs of the user. The ability to mount `~/john` as a hostPath is also provided. A way to assign any of the default Kubernetes user-facing roles is also provided by using `--role` key, as documented at [https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) The possible roles are: `view`, `edit`, `admin`, and `cluster-admin`. **Example** ```bash cm-kubernetes-setup --add-user john --role view ``` The preceding example creates a user `john` with `view` privileges only, for the default `john-restricted` namespace. **Example** ```bash cm-kubernetes-setup --add-user john --user-paths /home/john,/scratch --allow-all-uids \ --operators cm-jupyter-kernel-operator ``` The preceding example creates a user `john` with the following privileges: * `edit` privileges * able to mount `/home/john` and `/scratch` as hostPath volume, when the process runs with UID/GIDs taken from the PAM subsystem on the moment of creation * able to run as any user, including root (attempts to mount any hostPath volume will be rejected) * access to the Jupyter Kernel Operator, i.e. with access to the resource `kind: CmKubernetesOperatorPermissionsJupyterKernel` ## 4.12 Getting Information And Modifying Existing Kubernetes Users It is possible to edit user properties and permissions. `cm-kubernetes-setup` provides 2 ways of doing it: interactively or via CLI options. Modifying users can be done interactively by choosing `Modify User` in the `cm-kubernetes-setup` main menu. Guidance is then given on choosing the cluster, users, and on modifying permissions. Modifying users can also be done via CLI options, by specifying the `--modify-user` argument: **Example** ```bash cm-kubernetes-setup --add-user john --user-paths /home/john \ --allow-all-uids --operators cm-jupyter-kernel-operator ``` The default `john-restricted` namespace is created as user `john` is added, along with the settings specified by the other options. The following example then adds permission to mount the `/scratch` hostPath into pods, and gives access to the Postgres Operator: ```bash cm-kubernetes-setup --modify-user john --user-paths /home/john,/scratch --namespace john-restricted\ --allow-all-uids --operators cm-jupyter-kernel-operator,cm-kubernetes-postgresql-operator ``` Information about existing users can be found with: **Example** ```bash cm-kubernetes-setup --list-users ``` Permission for user `john` to operate in the `dev` namespace can be added with: **Example** ```bash kubectl create namespace dev cm-kubernetes-setup --modify-user john --namespace dev --add-to-namespace ``` Permission for the user `john` to operate in the `dev` namespace can be revoked with: **Example** ```bash kubectl create namespace dev cm-kubernetes-setup --modify-user john --namespace dev --remove-from-namespace ``` ## 4.13 List Of Resources Defined For Users These resources are rendered by the Permission Manager Operator, and can therefore be found inside Kubernetes. #### 4.13.0.1 The Role Bindings Deployed For Every User By Default By default, the role bindings deployed for the user `john` created in the preceding section are: * `ClusterRole/john-nodes` (in namespace `john-restricted`) * `ClusterRoleBinding/john-nodes` (in namespace `john-restricted`) User `john` is given read-only rights for the Nodes resource (for `kubectl get nodes`). #### 4.13.0.2 The Secure Namespace Related Resources The secure namespace for user `john` is: * `Namespace/john-restricted` The service account used by `john`: * `ServiceAccount/john` (in namespace `john-restricted`) This is found referenced, for example, in `john`’s `$HOME/.kube/config`. The PodSecurityPolicy that defines the user can run non-privileged pods, and use only ports above `1024`, and so on: * `PodSecurityPolicy/john-restricted` (in namespace `john-restricted`) More details on this can be found in section [4.10](#410-security-model), [here](#4102-podsecuritypolicy). This policy will only do something as soon as the `PodSecurityPolicy` Admission Controller is enabled in the API server. A PodSecurityPolicy that defines the user can run as root as well, but *without* hostPath volumes: * `PodSecurityPolicy/john-restricted-root` (in namespace `john-restricted`) To give the aforementioned privileges to `john`’s secure namespace, so that `john` can run workloads, execute `kubectl get all`, and more: * `Role/john-restricted` (in namespace `john-restricted`) * `RoleBinding/john-restricted` (in namespace `john-restricted`) The RoleBinding assigns it to the user `john` and ServiceAccount account for `john`. The upstream documentation at [https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin](https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin) has more details on this. The user `john` can be given the ability to use the PodSecurityPolicy defined earlier in his secure namespace, but also in other namespaces: * `ClusterRole/john-psp` (in namespace `john-restricted`) * `ClusterRoleBinding/john-psp` (in namespace `john-restricted`) The same ability can be given for the second `root but no hostPath` PodSecurityPolicy: * `ClusterRole/john-psp-root` (in namespace `john-restricted`) * `ClusterRoleBinding/john-psp-root` (in namespace `john-restricted`) If the Kyverno engine is installed then several policies are added: * `clusterpolicies.kyverno.io/john-*-drop-privs-w-hostpath*`: Policy to modify pod manifests to run process with specified UID/GID * `clusterpolicies.kyverno.io/john-*-limit-hostpath-vols`: Policy to deny pods if hostPath volumes does not match specified paths The full content of all the documents created for the user can be viewed by checking the generated Helm manifest: **Example** ```bash helm get manifest -n cm-permissions john-XXXXXX ``` ## 4.14 Kyverno Kyverno ([https://kyverno.io/](https://kyverno.io/)) is a policy engine designed for Kubernetes. With Kyverno, policies are managed as Kubernetes resources, and no new language is required to write policies. This allows the use of familiar tools such as kubectl, git, and kustomize to manage policies. Kyverno policies can validate, mutate, and generate Kubernetes resources, as well as ensure OCI image supply chain security. ### 4.14.1 Kyverno Installation Kyverno engine and Kyverno policy Helm charts can be installed as a part of `cm-kubernetes-setup`: ![Choosing Kyverno installation](/base-command-manager/product-manuals/_fern-img/8ab83ad4a5b44803d133858ed4dc583132048b1d8827c279fd8746a6958d3f7f.webp) *Figure 4.7: Choosing Kyverno installation* ![Kyverno high availability setup](/base-command-manager/product-manuals/_fern-img/bf88b0d45956bd6c464db73c66bcd28aa699d7bc7c3f081cd4d84f15f630ac1b.webp) *Figure 4.8: Kyverno high availability setup* ![Kyverno policy setup](/base-command-manager/product-manuals/_fern-img/eadee4bc145c6e9da4cd17ba2b76ce28c5942f7293ac587d9fe09d92d6dc1f17.webp) *Figure 4.9: Kyverno policy setup* The installation adds 2 Helm charts in the namespace ’kyverno’: ```bash [root@basecm10 ~]# module load kubernetes/ [root@basecm10 ~]# helm list -n kyverno NAME NAMESPACE STATUS ... CHART APP VERSION kyverno kyverno deployed ... kyverno-v2.5.2 v1.7.2 kyverno-policies kyverno deployed ... kyverno-policies-v2.5.2 v1.7.2 ``` If the HA option is chosen, then the replica count value is set to 3. ```bash [root@basecm10 ~]# helm get values -n kyverno kyverno USER-SUPPLIED VALUES: replicaCount: 3 ``` This means that at any given time Kubernetes scheduler tries to run 3 pods at the same time: ```bash [root@basecm10 ~]# kubectl get pods -n kyverno NAME READY STATUS RESTARTS AGE kyverno-5bfb99b9c9-ddmmw 1/1 Running 0 1h kyverno-5bfb99b9c9-hgfsc 1/1 Running 0 1h kyverno-5bfb99b9c9-n67rv 1/1 Running 0 1h ``` ### 4.14.2 Kyverno Policies It is also recommended to install Kyverno policies in order to enforce Pod Security Standards [https://kyverno.io/policies/](https://kyverno.io/policies/). BCM configures Kyverno policies in ’enforce’ mode, adding service namespaces as exclusions. The list of namespaces to be excluded from particular policies depend on the selected features during install: ```bash [root@basecm10 ~]# helm get values -n kyverno kyverno-policies USER-SUPPLIED VALUES: validationFailureAction: enforce policyExclude: disallow-host-namespaces: any: - resources: kinds: - Pod namespaces: - default - prometheus disallow-host-path: any: - resources: kinds: - Pod namespaces: - default - local-path-storage - '*-restricted' - prometheus - kube-system - gpu-operator disallow-host-ports: any: - resources: kinds: - Pod namespaces: - default - prometheus ``` In the preceding output, all namespaces that match the wildcard `*-restricted` are excluded from the policy named ’disallow-host-path’ ([https://kyverno.io/policies/pod-security/baseline/disallow-host-path/disallow-host-path/](https://kyverno.io/policies/pod-security/baseline/disallow-host-path/disallow-host-path/)). This means that, without additional restrictions, all pods in the user namespaces can mount any host path from an underlying node. To prevent that Kubernetes Permission Manager creates a Kyverno Cluster Policy for every newly-created user, and restricts the hostPath to only the home directory of the user: ```bash [root@basecm10 ~]# kubectl get clusterpolicies.kyverno.io | grep john john-n730sr0-drop-privs-w-hostpath false enforce true john-n730sr0-drop-privs-w-hostpath-containers false enforce true john-n730sr0-drop-privs-w-hostpath-initcontainers false enforce true john-n730sr0-limit-hostpath-vols false enforce true ``` ## 4.15 Kubernetes Permission Manager The Kubernetes permission manager is a custom operator based on Helm. It helps to manage user and system account permissions, roles, role bindings and pod security policies. The operator itself is packed and distributed as a Helm chart, so it can be installed during Kubernetes cluster creation via the `cm-kubernetes-setup` TUI. The Helm chart for the operator is located in `/cm/shared/apps/kubernetes-permissions-manager/current/helm`. The output to the following command shows if it is installed: ```bash [root@basecm10 ~]# module load kubernetes/ [root@basecm10 ~]# helm list -n cm NAME NAMESPACE STATUS ... CHART APP VERSION local-path-provisioner cm deployed ... cm-kubernetes-local-path-provisioner-0.0.20 0.0.20 permissions-manager cm deployed ... cm-kubernetes-permissions-manager-0.0.1 0.0.1 ``` The Helm chart of the operator includes custom resource definitions (CRD), and makes it possible for the administrator to manage resources using the `kubectl` tool: **Example** ```bash [root@basecm10 ~]# cat > permissions.yaml<kubernetes[default]]% get trusteddomains basecm10.example.com kubernetes kubernetes.default kubernetes.default.svc master localhost ``` In the preceding example, the FQDN of the cluster is `basecm10.example.com`. The cluster administrator managing their own cluster will have another FQDN, and not this FQDN. For `kubectl`, the Kubernetes API server proxy port should be open to the external network. The proxy port can be retrieved from the Kubernetes cluster entity as follows: ```bash [basecm10->kubernetes[default]]% get kubernetesapiserverproxyport 10443 ``` For the Kubernetes Dashboard, the Ingress Controller HTTPS port should be open to the external network. This port, by default with a value of `30443`, can be retrieved from the `ingress_controller` add-on environment: **Example** ```bash [basecm10->kubernetes[default]]% appgroups [basecm10->kubernetes[default]->appgroups]% applications system [basecm10->kubernetes[default]->appgroups[system]->applications]% environment ingress_controller [basecm10->...applications[ingress_controller]->environment]% list Name (key) Value Nodes environment --------------------------- --------------------------------------- ------------------ CM_KUBE_EXTERNAL_FQDN basecm10.example.com yes CM_KUBE_INGRESS_HTTPS_PORT 30443 yes CM_KUBE_INGRESS_HTTP_PORT 30080 yes ingress_controller_label brightcomputing.com/ingress-controller no replicas 1 no ``` If exposing the Kubernetes API server to the external network is selected during setup with `cm-kubernetes-setup`, then the HTTPS and HTTP ports in the preceding example are opened on the Shorewall service that runs on the head node. Exposure to the external network is enabled by default. #### 4.16.0.1 Convention Of Using A Domain Name As A Prefix Label In the preceding example, the `brightcomputing.com` prefix that is part of the value for `ingress_controller_label` is just a label rather than a domain. The reason that prefix is used is that it simply follows the convention of using domain names as labels, such as is done by the Kubernetes community (domain: `kubernetes.io`) and RHEL OpenShift (domain: `openshift.io`). The prefix `brightcomputing.com` could equally well have been the prefix `brightaccess` instead. However it is probably less confusing now to follow the established convention. So that is what is done here for the label. #### 4.16.0.2 Users Can Access The Kubernetes Dashboard Users can access the Kubernetes Dashboard using `dashboard`. By default, the URL takes the FQDN and the port value along with the `dashboard` subdomain, and has the form: `https://dashboard.` \<*CM_KUBE_EXTERNAL_FQDN*>`:<`*CM_KUBE_INGRESS_HTTPS_PORT*> So, for example, it could be something like: **Example** ```bash https://dashboard.basecm10.example.com:30443 ``` #### 4.16.0.3 Ingress Configuration For Dashboard In `cmsh` The default Ingress rule described earlier can be found as an object within `cmsh`: ```bash [basecm10->kubernetes[default]->appgroups[system]->applications[dashboard_ingress]]% get config apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: kubernetes-dashboard namespace: kubernetes-dashboard annotations: kubernetes.io/ingress.class: "nginx" nginx.ingress.kubernetes.io/secure-backends: "true" nginx.ingress.kubernetes.io/ssl-passthrough: "true" nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" spec: rules: - host: "dashboard.$CM_KUBE_EXTERNAL_FQDN" http: paths: - path: / backend: serviceName: kubernetes-dashboard servicePort: 443 ``` Using `kubectl`, the Ingress resource can be found with: ```bash bash$ kubectl get ingress -n kubernetes-dashboard NAME HOSTS ADDRESS PORTS AGE kubernetes-dashboard dashboard.cluster1.local 10.150.153.251 80 45h ``` The official documentation for Ingress, at [https://v1-16.docs.kubernetes.io/docs/concepts/services-networking/ingress/](https://v1-16.docs.kubernetes.io/docs/concepts/services-networking/ingress/), explains it well. Path rewrites without domain names can also be used to set up Ingress with multiple backends (`serviceName` and servicePort pairs), without having to deal with setting up a DNS. #### 4.16.0.4 Ingress Controller Running On Compute Nodes For scenarios where the head node is not involved in a Kubernetes setup, BCM does not currently set up any forwarding for the Ingress Controller. BCM does set up an NGINX proxy to expose the Kubernetes API Server in such cases, and accessing the Dashboard can then be done with the `kubectl proxy` approach. For now a workaround to forward Ingress to a compute node can be achieved with port-forwarding, for example by adding the following line to `/etc/shorewall/rules` in Shorewall [(](/installation/thirdparty#72-shorewall)section [7.2](/installation/thirdparty#72-shorewall) of the *Installation Manual*): **Example** ```bash DNAT net nat:10.141.0.1:30443 tcp 30443 ``` #### 4.16.0.5 Using One Ingress Controller For Multiple Kubernetes Clusters BCM does not offer an out-of-the-box solution for one Ingress Controller with multiple Kubernetes clusters. This configuration can be achieved by configuring software such as NGINX to proxy, based on the domain name to the appropriate backend(s). ## 4.17 Networking Model Kubernetes expects all pods to have unique IP addresses, which are reachable from within the cluster. This can be implemented in several ways, including adding pod network interfaces to a network bridge created on each host, or by using 3rd party tools to manage pod virtual networks. With BCM the default pod network provider is Calico ([https://www.projectcalico.org/](https://www.projectcalico.org/)). Calico uses the Border Gateway Protocol (BGP) to distribute routes for every Kubernetes pod. This allows the Kubernetes cluster to be integrated without the need for overlays (IP-in-IP). Calico is particularly suitable for large Kubernetes deployments on bare metal, or in private clouds. This is because for larger deployments the performance and complexity costs of overlay networks can become significant. ## 4.18 Kubernetes Monitoring When `cm-kubernetes-setup` is run, it configures the following Kubernetes-related health checks: 1. `KubernetesChildNode`: checks if all the expected agents and services are up and running for active nodes 2. `KubernetesComponentsStatus`: checks if all the daemons running on a node are healthy 3. `KubernetesNodesStatus`: checks if Kubernetes nodes have a status of `Ready` 4. `KubernetesPodsStatus`: checks if all the pods are in one of these states: `Running`, `Succeeded`, or `Pending` ## 4.19 Local Path Storage Class For storage, instead of creating Kubernetes PersistentVolumes every time, a modern and practical way is to use the StorageClass feature. Further documentation on StorageClass is available at: * [http://blog.kubernetes.io/2016/10/dynamic-provisioning-and-storage-in-kubernetes.html](http://blog.kubernetes.io/2016/10/dynamic-provisioning-and-storage-in-kubernetes.html) * [https://kubernetes.io/docs/concepts/storage/persistent-volumes/#storageclasses](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#storageclasses) As a part of initial installation it is possible to choose a Local Path Storage class to utilize the shared storage mounted on every node of the Kubernetes cluster. Possible options include any POSIX shared filesystems, such as NFS, BeeGFS, LustreFS, etc. ![Kubernetes setup TUI session: local storage configuration](/base-command-manager/product-manuals/_fern-img/df90ae853421bba2638a11fccd7e80fdbb4eab52484137d7a45242d9da7d299f.webp) *Figure 4.10: Kubernetes setup TUI session: local storage configuration* During setup, the installation wizard asks for a path for where Kubernetes physical volumes (PV) will be physically located. This path should be located on a shared filesystem accessible from all nodes. After installation, the storage class can be seen to be available with: ```bash # kubectl get storageclasses.storage.k8s.io NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE local-path (default) rancher.io/local-path Delete Immediate false 1h ``` Users of the cluster can then freely create persistent volume claims (PVC) resources and use them in running pods. ## 4.20 Setup Of A Storage Class For Ceph Pods running on Kubernetes can use Ceph as a distributed storage system to store data in a persistent way. This section assumes a working Ceph cluster. Ceph installation for BCM is covered in [Chapter 9 of the *Administrator Manual*](/admin/cephinstall). A new pool `kube` can be created with a replication factor of 3: **Example** ```bash [root@basecm10 ~]# ceph osd pool create kube 100 100 pool 'kube' created [root@basecm10 ~]# ceph osd pool set kube size 3 set ppol 1 size to 3 [root@basecm10 ~]# ceph osd pool set kube min_size 1 set pool 1 min_size to 1 ``` The parameters settings in the preceding example are documented at the Ceph website, at * [http://docs.ceph.com/docs/master/rados/operations/pools/](http://docs.ceph.com/docs/master/rados/operations/pools/) for documentation on Ceph operations * [http://docs.ceph.com/docs/master/rados/configuration/pool-pg-config-ref/](http://docs.ceph.com/docs/master/rados/configuration/pool-pg-config-ref/) for documentation on Ceph pool and PG (placement group) configuration The pods of a given namespace have to have access to the Ceph RBD images created to back the volumes. A `kube` client can be created with: **Example** ```bash [root@basecm10 ~]# ceph auth get-or-create client.kube mon 'allow r' osd 'allow rwx pool=kube' [client.kube] key = AQCnOvdZpYewBBAAWv1d7c7/XbEvj7QO7N0THg== ``` A list of the current users, and their access control can be viewed with (some output elided): **Example** ```bash [root@basecm10 ~]# ceph auth list installed auth entries: osd.0 key: AQD9M/dZw8HPNRAAT+X8mGSgRUkjLnQo38j4EA== caps: [mon] allow rwx caps: [osd] allow * osd.1 ... client.admin key: AQCnM/dZONOPMxAAwqY9ADbJV+6i2Uq/ZNqh5A== auid: 0 caps: [mds] allow * caps: [mgr] allow * caps: [mon] allow * caps: [osd] allow * ... client.kube key: AQCnOvdZpYewBBAAWv1d7c7/XbEvj7QO7N0THg== caps: [mon] allow r caps: [osd] allow rwx pool=kube ``` The `kube` user must be able to map images. The `kube` configuration must therefore look similar to the section for `client.kube` in the preceding example. A Kubernetes secret must be created in the `default` namespace, and in every Kubernetes namespace that needs storage, using the Ceph user key: **Example** ```bash [root@basecm10 ~]# kubectl create secret generic ceph-secret-user \ --from-literal=userKey=$(ceph auth get-key client.kube) --from-literal=userID=kube \ --namespace=default secret "ceph-secret-user" created ``` The Ceph cluster ID can be retrieved by running Ceph’s `fsid` command. The `fsid` command was given its name because it originally retrieved a **f**ile **s**ystem **ID**. However, Ceph has evolved since that time, and a file system is not required for the ID to be retrieved for Ceph storage. The ID is now a general ID for the Ceph storage system: **Example** ```bash [root@basecm10 ~]# ceph fsid fe513405-53f6-40a0-8ccb-651e1935c5e4 ``` Ceph monitor \<*IP address*>:\<*port*> values can be found by running `ceph mon stat`: **Example** ```bash [root@basecm10 ~]# ceph mon stat e1: 3 mons at {node001=10.141.0.1:6789/0,node002=10.141.0.2:6789/0,node003=10.141.0.3:6789/0},\ election epoch 38, quorum 0,1,2 node001,node002,node003 ``` The Helm repository `ceph-csi` has the container storage interface drivers for Ceph. It can be added with: **Example** ```bash [root@basecm10 ~]# helm repo add ceph-csi https://ceph.github.io/csi-charts "ceph-csi" has been added to your repositories ``` A request can be made to add a chart `ceph-csi-rbd` for a RADOS block device image that provides default information for the pods to come up: **Example** ```bash [root@basecm10 ~]# helm install --namespace "ceph-csi-rbd" "ceph-csi-rbd" ceph-csi/ceph-csi-rbd \ --create-namespace --set nodeplugin.httpMetrics.containerPort=8082 \ --set provisioner.httpMetrics.containerPort=8082 ``` The configmap has to be adjusted to add the Ceph cluster that is used for the storageclass in `config.json`. The cluster ID, as well as the Ceph monitor IP addresses and ports match the Ceph information from earlier on. **Example** ```bash # kubectl describe configmap -n ceph-csi-rbd ceph-csi-config Name: ceph-csi-config Namespace: ceph-csi-rbd Labels: app=ceph-csi-rbd app.kubernetes.io/managed-by=Helm chart=ceph-csi-rbd-3.9.0 component=nodeplugin heritage=Helm release=ceph-csi-rbd Annotations: meta.helm.sh/release-name: ceph-csi-rbd meta.helm.sh/release-namespace: ceph-csi-rbd Data ==== cluster-mapping.json: ---- [] config.json: ---- [ { "clusterID": "fe513405-53f6-40a0-8ccb-651e1935c5e4", "monitors":["10.141.0.1:6789", "10.141.0.2:6789", "10.141.0.3:6789"] } ] BinaryData ==== Events: ``` At this point, a `storage-class.yml` file can then be created, similar to: **Example** ```bash apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: fast parameters: clusterID: fe513405-53f6-40a0-8ccb-651e1935c5e4 imageFeatures: layering imageFormat: "2" pool: kube csi.storage.k8s.io/provisioner-secret-name: ceph-secret-user csi.storage.k8s.io/provisioner-secret-namespace: default provisioner: rbd.csi.ceph.com reclaimPolicy: Delete volumeBindingMode: Immediate ``` In the preceding YAML file `clusterID` and `pool` must be set to the appropriate values. Details about the `StorageClass` parameters can be found at: [https://github.com/ceph/ceph-csi/blob/devel/examples/rbd/storageclass.yaml](https://github.com/ceph/ceph-csi/blob/devel/examples/rbd/storageclass.yaml) The Kubernetes storage class for Ceph RBD can now be created: **Example** ```bash [root@basecm10 ~]# kubectl apply -f storage-class.yml storageclass.storage.k8s.io/fast created ``` To verify that it has been created, the new `StorageClass` can be listed with: **Example** ```bash [root@basecm10 ~]# kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE fast rbd.csi.ceph.com Delete Immediate false 33s ``` ## 4.21 Integration With Harbor In order to spawn pods that use images from the Harbor registry, a secret must first be created with the credentials: ```bash [root@basecm10 ~]# kubectl create secret docker-registry myregistrykey \ --docker-server=node001:9443 --docker-username=admin --docker-password=Harbor12345 ``` The secret must then be referenced from the pod: ```bash apiVersion: v1 kind: Pod metadata: name: foo spec: containers: - name: foo image: node001:9443/library/nginx imagePullSecrets: - name: myregistrykey ``` Further information on this is available at [https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod](https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod). ## 4.22 Kubernetes Upgrades This section assumes that an upgrade to a BCM Kubernetes is being considered. This means that it is assumed to have been set up with `cm-kubernetes-setup`. Upgrades to Kubernetes can be done by following the principles behind the official upstream documentation at [https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/). The following points must be kept in mind when upgrading a BCM Kubernetes cluster: * Nodes that are part of the cluster may be provisioned through their software image. * The Kubernetes version in BCM must manually be updated to match the version of the upgrade. * Only some Kubernetes versions are supported. A list of supported available versions can be retrieved with: **Example** ```bash [root@basecm10 ~]# cm-kubernetes-setup --list-versions 1.30 1.29 (NVIDIA AI Enterprise certified) 1.28 (NVIDIA AI Enterprise certified) 1.27 (NVIDIA AI Enterprise certified) ``` In the example that follows, a BCM cluster is upgraded from Kubernetes 1.27.13 to Kubernetes 1.28.9. [A](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/)t the time of writing (May 2024) the URL for the upstream documentation redirects the readers to the upgrade instructions for, among others, 1.27 to 1.28. ### 4.22.1 Upgrade Prerequisites Kubernetes is not a single entity for upgrade purposes. There is a single driver, the `kubeadm` program, which is used to plan and execute the upgrade, but it does it only for itself and a handful of components central to Kubernetes’ functionality. Kubernetes does not have ways to manage its third-party components with `kubeadm`’s upgrade planning and execution. This means that multiple Kubernetes components that are essential to Kubernetes are not involved in the upgrade process. If these components are neglected during an upgrade, then they can either prevent successful completion of the upgrade, or can themselves fail as a result of the upgrade. Some noteworthy components that are excluded by upgrades are the etcd database, CNIs, CSIs, container runtimes, as well as individual pods not managed by higher-level entities such as `Deployment`, `ReplicaSet` and so on. The Kubernetes documentation at [https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/#pod-disruption-budgets) describes the workings of the eviction process in greater detail. Kubernetes upgrades assume that various entities, in particular, pods, can migrate away from the node being upgraded. This assumption may not be true because of various constraints on pod scheduling. The constraints need to be satisfied or worked around in order for migration to work. Particular care has to be taken when dealing with persistent volumes or physical resources such as GPUs that cannot migrate, but are needed for a pod to be deployed. #### 4.22.1.1 Hot And Cold Upgrades * A hot upgrade is one that allows the system to stay online at all times. * A cold upgrade is one that requires at least a brief service interruption. Since Kubernetes is aimed at providing 100% system uptime, the upgrade process is designed to be hot. The cluster administrator carrying out the hot upgrade does however need to make an effort for the instances where Kubernetes cannot ensure service uptime. **Upgrading components that cannot be stopped in a regular manner:** One common problem with hot upgrades is the need to migrate or to stop `DaemonSet` pods. These are intended to run on all Kubernetes nodes, even those being drained. Kubernetes can be told to ignore `DaemonSet` pods by using the `--ignore-daemonsets` option. The problem with doing that is that it avoids upgrading those pods, and also typically avoids upgrading the third-party component that created them. To ensure upgrade completion, the administrator has to carry out a separate upgrade action, appropriate for each component ignored in this way. **Upgrading gradually:** Kubernetes upgrades come without any tools to perform upgrades at scale. The available tools can only upgrade a single node at a time. This means that, especially for larger clusters, the administrator needs to carry out the upgrade gradually, and has to plan and allocate extra capacity to accommodate pods migrated from nodes being upgraded. Special precautions need to be taken when dealing with deployments which require larger replica counts. Administrators may need to perform special procedures. Third-party components in particular may require the simultaneous presence of multiple entities across multiple nodes, in order to be able to upgrade those components without a service interruption. Finally, it is possible that some components are simply not designed for hot upgrades. In that case, administrators have to remove those components from the system, and install the newer version, with the downtime depending on each individual case. ### 4.22.2 Example RHEL9 Cluster A cluster with 3 nodes that run the control plane, and based on RHEL9, is considered as a reference example. One control plane node is the single head node, and the other two control plane nodes are two regular (compute) nodes. There are four additional worker-only nodes, and all the non-head nodes share the same category and software image. ```bash [root@basecm10 ~]# module load kubernetes [root@basecm10 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION node001 Ready control-plane,master,worker 8m25s v1.27.13 node002 Ready control-plane,master,worker 8m25s v1.27.13 node003 Ready worker 8m43s v1.27.13 node004 Ready worker 8m41s v1.27.13 node005 Ready worker 8m42s v1.27.13 node006 Ready worker 8m41s v1.27.13 basecm10 Ready control-plane,master 9m22s v1.27.13 ``` ### 4.22.3 Before Starting The Upgrade The upstream instructions at [https://v1-28.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/#before-you-begin](https://v1-28.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/#before-you-begin), on preparing for the upgrade should be read. A node that is to be upgraded must be drained. ### 4.22.4 Updating The First Control Plane Node The first control plane node is upgraded in this section. Instead of going into much detail on upgrades—the official documentation at [https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/) does a good job—an example covering an upgrade to a Rocky cluster running Kubernetes version 1.27 is shown that uses the official documentation for guidance. Instructions for Ubuntu differ slightly but can be carried out in a similar way. Typically the control plane is on the head node, so that the cluster administrator starts with updating the control plane being run by the head node. However, for completeness, the two subsections that follow describe updating the control plane if starting with: * either the head node * or a compute node #### 4.22.4.1 Upgrading A Control Plane Node Starting With The Head Node To upgrade the control plane when the head node is the first node to be upgraded, the repository entry for picking up packages from the Kubernetes repository should have the new version set. **Example** ```bash [root@basecm10 ~]# grep -ir kubernetes /etc/yum.repos.d/* #is anything already there? /etc/yum.repos.d/kubernetes.repo:[kubernetes] /etc/yum.repos.d/kubernetes.repo:name=Kubernetes /etc/yum.repos.d/kubernetes.repo:exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni [root@basecm10 ~]# grep v /etc/yum.repos.d/kubernetes.repo #what version is fetched? baseurl=https://pkgs.k8s.io/core:/stable:/v1.27/rpm/ gpgkey=https://pkgs.k8s.io/core:/stable:/v1.27/rpm/repodata/repomd.xml.key ``` The `grep` outputs confirm there is a repository file, and that it is set to pick up Kubernetes version 1.27. The .27 indicates what is called the minor version in Kubernetes. To instead pick up version 1.28, the upstream documentation instructs changing the minor version in the repository file to the next minor version upgrade number, so here from `v1.27` to `v1.28`. Skipping minor versions for upgrades is unsupported. An additional check to confirm the running version is: ```bash [root@basecm10 ~]# kubeadm version kubeadm version: &version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.13", GitCommit: ... ``` The minor version in `kubernetes.repo` is changed from `v1.27` to `v1.28` using a text editor. The installation for a new `kubeadm` can then be carried out with: ```bash [root@basecm10 ~]# yum install -y kubeadm --disableexcludes=kubernetes ... [root@basecm10 ~]# kubeadm version #should now show we are at version 1.28 kubeadm version: &version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.9", ... ``` The upgrade plan can be inspected: ```bash [root@basecm10 ~]# kubeadm upgrade plan ... You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.28.9 ... ``` In this case it gives a recommended upgrade command to run on the head node: The upgrade is now made to version `1.28.9`, as suggested. The curious cluster administrator can view the list of available versions can be viewed with the `--showduplicates` option to `yum`: **Example** ```bash [root@basecm10 ~]# yum list --showduplicates kubeadm --disableexcludes=kubernetes ... Installed Packages kubeadm.x86_64 1.28.9-150500.2.1 @kubernetes Available Packages kubeadm.aarch64 1.28.0-150500.1.1 kubernetes kubeadm.ppc64le 1.28.0-150500.1.1 kubernetes kubeadm.s390x 1.28.0-150500.1.1 kubernetes kubeadm.src 1.28.0-150500.1.1 kubernetes kubeadm.x86_64 1.28.0-150500.1.1 kubernetes ... ``` Most cluster administrators stick to the recommendation. ```bash [root@basecm10 ~]# kubeadm upgrade apply v1.28.9 ... [upgrade/version] You have chosen to change the cluster version to "v1.28.9" [upgrade/versions] Cluster version: v1.27.13 [upgrade/versions] kubeadm version: v1.28.9 [upgrade] Are you sure you want to proceed? [y/N]: y ... [upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.28.9". Enjoy! [upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so. ``` The host should now be drained, if it has not already been drained: ```bash [root@basecm10 ~]# kubectl drain $(hostname) --ignore-daemonsets ``` As suggested, the kubelets are now upgraded. Updating the `kubectl` version at the same time is also a good idea: ```bash [root@basecm10 ~]# yum install -y kubelet kubectl --disableexcludes=kubernetes ``` The kubelets can be restarted so that the new versions run: ```bash [root@basecm10 ~]# kubelet --version Kubernetes v1.28.9 [root@basecm10 ~]# systemctl status kubelet ... Active: active (running) since ...; 21h ago #need to reload kubelets to new versions ... [root@basecm10 ~]# systemctl daemon-reload [root@basecm10 ~]# systemctl restart kubelet && systemctl status kubelet | grep Active Active: active (running) since ... 14ms ago ``` This ends the procedure of upgrading the control plane by first starting on a head node. Upgrading the control plane by first starting with a regular node instead is described next. #### 4.22.4.2 Upgrading The Control Plane Node Starting With A Compute Node This section describes the list of commands for upgrading the control plane when it is the first control plane node to be upgraded. For example `node001`, rather than a head node. The appropriate software image and related information for the node must first be found. For `node001` this can be done with: ```bash [root@basecm10 ~]# cmsh [basecm10]% device use node001 [basecm10->device[node001]]% get softwareimage default-image (category:default) [basecm10->device[node001]]% softwareimage [basecm10->softwareimage]% use default-image [basecm10->softwareimage[default-image]]% get path /cm/images/default-image ``` The software image is then entered and modified via a chroot jail. The repository file of the image, with an absolute path outside the chroot jail of `/cm/images/default-image/etc/yum.repos.d/kubernetes.repo`, has its minor version changed appropriately: ```bash [root@basecm10 ~]# cm-chroot-sw-img /cm/images/default-image [root@default-image /]# grep v /etc/yum.repos.d/kubernetes.repo #what version would be fetched? /etc/yum.repos.d/kubernetes.repo:baseurl=https://pkgs.k8s.io/core:/stable:/v1.27/rpm/ /etc/yum.repos.d/kubernetes.repo:gpgkey=https://pkgs.k8s.io/core:/stable:/v1.27/rpm/repodata/repomd.xml.key [root@default-image /]# vi /etc/yum.repos.d/kuberenetes.repo #modify the point version in the image ...modifications done... [root@default-image /]# grep v /etc/yum.repos.d/kubernetes.repo #will fetch this version now /etc/yum.repos.d/kubernetes.repo:baseurl=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/ /etc/yum.repos.d/kubernetes.repo:gpgkey=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/repodata/repomd.xml.key ``` The package versions available for installation in the image can be checked with: ```bash [root@default-image /]# yum info kubeadm kubelet kubectl --disableexcludes=kubernetes ``` In this case versions 1.28.9 are seen to be available. The `kubeadm`, `kubelet` and `kubectl` packages are upgraded in advance, and the chroot environment is then exited: ```bash [root@default-image /]# yum install -y kubeadm kubelet kubectl --disableexcludes=kubernetes ... [root@default-image /]# exit #leave the chroot ``` The following set of commands then carries out the update on `node001`: ```bash [root@basecm10 ~]# export host=node001 [root@basecm10 ~]# cmsh -c "device use $host; imageupdate -w --wait" #update running image on node [root@basecm10 ~]# ssh $host kubeadm version #what version to fetch now [root@basecm10 ~]# ssh $host kubeadm upgrade plan #get suggested upgrade plan [root@basecm10 ~]# ssh $host kubeadm upgrade apply v1.28.9 #apply the upgrade [root@basecm10 ~]# kubectl drain $host --ignore-daemonsets #drain the host [root@basecm10 ~]# ssh $host sudo systemctl daemon-reload #reload systemd manager config [root@basecm10 ~]# ssh $host sudo systemctl restart kubelet #restart kubelet [root@basecm10 ~]# kubectl uncordon $host #new pods may now run on node ``` ### 4.22.5 Updating Subsequent Control Plane Nodes If there is only one control plane node, then this section can be skipped. If there is more than one control plane node, and if the first control plane node is a head node that has been updated, then the remaining control plane nodes should be updated. The `kubectl` command can be used to orient the cluster administrator, by displaying the state of the upgrade, roles, and Kubernetes versions per node: **Example** ```bash [root@basecm10 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION node001 Ready control-plane,master,worker 26m v1.27.13 node002 Ready control-plane,master,worker 26m v1.27.13 node003 Ready worker 26m v1.27.13 node004 Ready worker 26m v1.27.13 node005 Ready worker 26m v1.27.13 node006 Ready worker 26m v1.27.13 basecm10 Ready control-plane,master 27m v1.28.9 ``` The remaining control plane nodes can now be updated. In the reference example case, these control plane nodes are `node001` and `node002` as seen in the preceding example. The commands in the section on **Upgrading The Control Plane Node Starting With A Compute Node** ([here](#42242-upgrading-the-control-plane-node-starting-with-a-compute-node)) should be followed, but with one small difference. Instead of running: `kubeadm upgrade apply v1.28.9` the command `kubeadm upgrade node` is run. The software image should also be prepared with the new packages before exiting the chroot jail. After exiting the chroot jail the remaining control plane nodes can then be updated with the following set of commands: **Example** ```bash [root@basecm10 ~]# export host=node001 [root@basecm10 ~]# cmsh -c "device use $host; imageupdate -w --wait" [root@basecm10 ~]# ssh $host kubeadm version [root@basecm10 ~]# ssh $host kubeadm upgrade plan [root@basecm10 ~]# ssh $host kubeadm upgrade node # I'm special! (apply not used here) [root@basecm10 ~]# kubectl drain $host --ignore-daemonsets [root@basecm10 ~]# ssh $host sudo systemctl daemon-reload [root@basecm10 ~]# ssh $host sudo systemctl restart kubelet [root@basecm10 ~]# kubectl uncordon $host ``` Output after this should now look similar to: ```bash [root@basecm10 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION node001 Ready control-plane,master,worker 30m v1.28.9 node002 Ready control-plane,master,worker 30m v1.27.13 node003 Ready worker 30m v1.27.13 node004 Ready worker 30m v1.27.13 node005 Ready worker 30m v1.27.13 node006 Ready worker 30m v1.27.13 basecm10 Ready control-plane,master 31m v1.28.9 ``` The procedure can be repeated for `node002` in the reference example system. After that, the control plane nodes are all in an updated state. The worker nodes still need updating. ### 4.22.6 Updating The Worker Nodes The commands for the worker nodes follow a similar pattern. They are updated one by one, and the software images must also be updated, if it has not already been done. The software image update follows the procedure described in the section on **Upgrading The Control Plane Node Starting With A Compute Node** ([here](#42242-upgrading-the-control-plane-node-starting-with-a-compute-node)), where a chroot jail is entered, the updates are carried out, and the chroot jail is left. Since nodes are drained as part of their update procedure, it is best to do them one by one, or at most in limited batches to avoid doing too many at once. The set of commands to update a single worker is: ```bash [root@basecm10 ~]# export host=node003 [root@basecm10 ~]# cmsh -c "device use $host; imageupdate -w --wait" [root@basecm10 ~]# ssh $host kubeadm upgrade node [root@basecm10 ~]# kubectl drain $host --ignore-daemonsets [root@basecm10 ~]# ssh $host sudo systemctl daemon-reload [root@basecm10 ~]# ssh $host sudo systemctl restart kubelet [root@basecm10 ~]# kubectl uncordon $host ``` The `kubectl drain` command might complain for other reasons. The administrator can decide on proceeding further by adding additional flags. For example: `kubectl drain $host --ignore-daemonsets --delete-emptydir-data` forces the drain even if some pods have stateful data in `emptyDir` volumes. Multiple nodes can be updated as follows: **Example** ```bash [root@basecm10 ~]# export hosts='node00[4-6]' [root@basecm10 ~]# cmsh -c "device; imageupdate -n $hosts -w --wait" [root@basecm10 ~]# pdsh -w $hosts kubeadm upgrade node [root@basecm10 ~]# pdsh -N -w $hosts hostname | xargs -i -n 1 kubectl drain {} --ignore-daemonsets \ --delete-emptydir-data [root@basecm10 ~]# pdsh -w $hosts sudo systemctl daemon-reload [root@basecm10 ~]# pdsh -w $hosts sudo systemctl restart kubelet [root@basecm10 ~]# pdsh -N -w $hosts hostname | xargs -i -n 1 kubectl uncordon {} [root@basecm10 ~]# ``` Output after all the worker nodes are updated too should look similar to: ```bash [root@basecm10 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION node001 Ready control-plane,master,worker 35m v1.28.9 node002 Ready control-plane,master,worker 35m v1.28.9 node003 Ready worker 35m v1.28.9 node004 Ready worker 35m v1.28.9 node005 Ready worker 35m v1.28.9 node006 Ready worker 35m v1.28.9 basecm10 Ready control-plane,master 36m v1.28.9 ``` ### 4.22.7 Updating The Status In BCM At the time of writing of this section (May 2024), the following change is not yet carried out automatically by BCM. It is however necessary in order to get the correct version of Kubernetes reflected in the module file. The procedure to update the old version can be as follows: ```bash [root@basecm10 ~]# module load kubernetes/ #what do we have? module load kubernetes/default/1.27.13-150500.2.1 [root@basecm10 ~]# cmsh [basecm10]% kubernetes [basecm10->kubernetes[default]]% get version 1.27.13-150500.2.1 [basecm10->kubernetes[default]]% !yum info kubeadm | grep -E '(^Source|^Version|^Release)' Version : 1.28.9 Release : 150500.2.1 Source : kubeadm-1.28.9-150500.2.1.src.rpm [basecm10->kubernetes[default]]% set version 1.28.9-150500.2.1 [basecm10->kubernetes*[default*]]% commit [basecm10->kubernetes[default]]% quit [root@basecm10 ~]# module load kubernetes/ #what do we have now? module load kubernetes/default/1.28.9-150500.2.1 ... ``` ### 4.22.8 Notes For Upgrading To Kubernetes v1.31.x The `kubeadm-init` files generated by BCM for Kubernetes clusters older than v1.31, contain an incompatibility with v1.31 that cannot be solved automatically by `kubeadm config migrate`. To prevent errors while executing the `kubeadm config migrate` command, the whole `JoinConfiguration` block needs to be removed from the kubeadm init file. The part to remove looks like this: ```bash --- apiVersion: kubeadm.k8s.io/v1beta3 kind: JoinConfiguration # do not uncomment the following, kubelets will not be able to start # caCertPath: "/etc/kubernetes/pki/kc.name/ca.crt" ``` ### 4.22.9 Notes For Ubuntu The upstream documentation covers Ubuntu upgrades when it diverges from RHEL procedures. As an aid, the following commands, discussed in the procedure for RHEL within section [4.22](#422-kubernetes-upgrades) are followed by the corresponding commands for Ubuntu: ```bash # RHEL grep v /etc/yum.repos.d/kubernetes.repo vi /etc/yum.repos.d/kubernetes.repo #to change minor number yum list --showduplicates kubeadm --disableexcludes=kubernetes #see the versions sudo yum install -y kubeadm kubelet kubectl --disableexcludes=kubernetes #installs from repo # Ubuntu grep v /etc/apt/sources.list.d/kubernetes.list vi /etc/apt/sources.list.d//kubernetes.list #to change minor number apt-get update; apt-cache madison kubeadm # update for a fresh cache to see the versions # madison option lists versions and origin repos apt-mark unhold kubeadm kubelet kubectl && \ apt-get update && apt-get install -y kubeadm kubelet kubectl && \ apt-mark hold kubeadm kubelet kubectl #steps from upstream docs, installs from repo ``` ### 4.22.10 Notes For SLES The upstream documentation does not provide explicit instructions for distributions other than RHEL-based and Ubuntu-based for v1.28. However, starting with v1.29, SLES-based distributions are included. For v1.28, for SLES, BCM uses the “tarball” approach. The installation of such non-package-manager packages is documented at [https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/), in the “Without a package manager” section. The commands in that section there can be used as a guide to find alternatives for the `yum` and `apt` commands used so far. For example, the following command from the installation instructions documentation can be used to get the binaries for the same Kubernetes version used so far in the reference example: This is the “Without a package manager” equivalent of the previous `yum install` and `apt install` commands: ```bash RELEASE="v1.28.9" ARCH="amd64" pushd /usr/bin/ sudo curl -L --remote-name-all \ https://dl.k8s.io/release/${RELEASE}/bin/linux/${ARCH}/{kubeadm,kubelet,kubectl} sudo chmod +x {kubeadm,kubelet,kubectl} popd ``` ### 4.22.11 Other Approaches When carrying out the steps described in this chapter, some things may in practice end up being done differently from what is suggested. For example: control plane nodes often have different categories and software-images compared to the workers. For example, perhaps the master nodes do not have GPUs, and need different packages. In that case, multiple software images have to be prepared with new packages. Earlier on, a process to update nodes one at a time was described. However, when a software image is updated, and multiple nodes are tied to that software image, then those nodes can all be provisioned at the same time. The binaries in the updated software image can therefore also be provisioned to all the nodes using that image. The problem with this is that it could result in nodes getting the new binaries prematurely if they happen to reboot during the update. If this is an unwanted risk, then it can be avoided in several ways, described next. #### 4.22.11.1 Additional Software Images A separate software image can be introduced. Nodes can be moved to the new software image one at a time. A software image can also be overruled at the level of a node. **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% device use node001 [basecm10->device[node001]]% get softwareimage default-image (category:default) [basecm10->device[node001]]% set softwareimage my-new-image [basecm10->device*[node001*]]% commit [basecm10->device[node001]]% get softwareimage my-new-image [basecm10->device[node001]]% quit ``` The cluster administrator should remember to undo the preceding settings, or should move the new software image to the appropriate category, and then clear the node-level override again. This is so that, after the upgrade, the system is organized as before. #### 4.22.11.2 Using NOSYNC The other option is to configure the nodes with `NOSYNC` for their `Next install mode`. This prevents them from syncing with their software image when rebooting (in the case that they still reboot): **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% device use node001 [basecm10->device[node001]]% set nextinstallmode nosync [basecm10->device*[node001*]]% commit ``` Both these approaches make updating slightly more tedious, but also more straighforward. During testing by BCM developers, nodes getting their binaries updated prematurely due to an unexpected reboot was not seen to be a significant issue. This is presumably because as long as the first control plane node is updated successfully, and the reboot of the extra node is by accident, there is an interruption anyway. The kubelet simply comes back up with the new version. However this is not the official recommended approach. ### 4.22.12 Configuring The Ingress HTTPS Server Certificate Kubernetes applications that are exposed via the Kubernetes Ingress server on BCM use HTTPS on port 30443 by default. This port number is not the default value of 443 that is typically used by Kubernetes Ingress. Some Kubernetes applications may have issues with this, due to hard-coding of the value of 443. An example is an HTML page with a hyperlink that points to port 443 instead of 30443, which in turn leads to a non-existent page. Keeping an eye out for this and related issues is a good idea. For Ingress, it is assumed that * the cluster administrator has provided the cluster with a domain name, for example `my-cluster.nvidia.local` * there is a DNS entry present that makes this domain name resolve to the cluster IP address * a matching `server.key` and `server.crt` server certificate key pair file has been provided #### 4.22.12.1 Ingress Self-signed Certificate For Kubernetes Ingress testing purposes it is possible to create a self-signed certificate with a private certificate authority (CA) owned by the cluster administrator for its HTTPS protocol. That is, using that instead of a trusted (third-party) CA. However the effort needed to configure this means that the procedure is not generally recommended. Reasons to avoid using a self-signed certificate are: * each user needs the self-signed CA certificate pairs working on their system between the appropriate applications * each user also needs to override a warning about the untrusted certificate by accepting the self-signed certificate on browsers such as Chrome. * it is very easy to run into subtle issues later on that are hard to uncover That said, appendix [A](/containerization/selfsigned) has instructions on setting up a self-signed certificate for testing purposes. The steps that are needed to also make the certificates trusted on any given subsystem or application such as the web browser, are outside the scope of the manuals and BCM support. It is assumed that this has already been set up by the organization. #### 4.22.12.2 Ingress Trusted CA Certificate In case no self-signed certificate is to be used—for example if there is already a trusted CA certificate to use for this purpose—then another Kubernetes wizard session can be run from the active head node to help make the necessary changes to the Ingress controller. The cluster administrator carries out the following procedure: * uses SSH to get into the active head node * copies certificate pair files, `server.crt` and `server.key`, over to the head node (e.g., to the `/root` directory) * invokes `cm-kubernetes-setup` and chooses (figure [4.11](#fig-setup-ingress-https)):\ `Configure Ingress (Configure Ingress Server Certificate)` * answers `yes` when the wizard prompts: `Do you want to configure an existing, properly signed Server certificate pair?` * sets paths to the certificate pair files, so that the Ingress controller then uses the trusted CA certificate pair, instead of generating a self-signed pair * goes along with other steps such as Kubernetes secret creation, patching the appropriate YAML, and so on * waits for the configuration change (a minute or so) to restart the Ingress controller * confirms that the HTTPS certificate is working correctly, via the SAN check (described shortly) ![Option to configure Server Certificate for Ingress Controller](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/0776e7ac83d3381ddb0c8aa522aeee64176f44c2c567576c0ba3ac24552d8a63/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/setup-ingress-https.png) *Figure 4.11: Option to configure Server Certificate for Ingress Controller* **The SAN check:** The following command can be used to inspect the SAN (Subject Alternative Name) part of the certificate currently running on the Ingress port: **Example** ```bash [root@basecm10 ~]# openssl s_client -connect localhost:30443 < /dev/null 2>/dev/null | \ openssl x509 -noout -text | grep -A1 'Subject Alternative Name' X509v3 Subject Alternative Name: DNS:my-cluster.nvidia.local, DNS:*.apps.my-cluster.nvidia.local, DNS:master.cm.cluster ``` ## 4.23 Air-gapped Kubernetes Upgrade An air-gapped cluster environment is one where the cluster is not directly connected to the internet. A BCM cluster that runs Kubernetes that is air-gapped can be upgraded with the procedure outlined in this section. ### 4.23.1 Prerequisites Before beginning the upgrade, the following general prerequisites and version considerations must be carried out: * The cluster version must be at a sufficiently high point release number ([section 11.1.2 of the *Administrator Manual*](/admin/softwareimagemanagement#1112-bcm-package-point-release-versions-and-the-cm-package-release-info-command)): * for BCM version 10 the point release version number must be 10.31.0 or higher * The upgrade path should take into account the Kubernetes version skew policy, as defined at [https://kubernetes.io/releases/version-skew-policy/](https://kubernetes.io/releases/version-skew-policy/). The principles to take into account are: * Only two adjacent minor releases may exist in the cluster at any time. So, v1.34 and v1.35 are acceptable; but v1.33, v1.34, and v1.35 are not acceptable. * The target version must not violate the skew values. So, if the cluster nodes are currently on two different adjacent minor releases, then the only valid upgrade path is to the newest of the two minor releases. For example if some nodes are running v1.34 and some are running v1.35, then the only valid upgrade path is to upgrade the v1.34 nodes to v1.35. This ensures that the cluster remains functional and compliant with the Kubernetes version skew policies during the rolling upgrade process. * The upgrade path should also take into account the NVIDIA BCM Release Notes and Compatibility Matrix corresponding to the BCM version that is running. This must be consulted, to see if there is support for the desired target Kubernetes version (for example v1.35.x). * An upgrade path preference, for minor or patch, must be decided on for the case where all the nodes are on the same minor version: * Minor release upgrade: the latest release of the next minor release (for example: v1.34 to v1.35) * Patch release upgrade: the latest patch release of the current minor release (for example: v1.35.3 to v1.35.4) * The current Kubernetes cluster must already be operational, with healthy and ready nodes. * A local registry must already be deployed, and its FQDN must be retrievable (for example: master.cm.cluster:5000) * The upstream instructions on preparing for the upgrade ([https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/#before-you-begin](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/#before-you-begin)) should be read. * A node that is to be upgraded must be drained. ### 4.23.2 Airgap Preparation Steps The initial steps must be performed on a host with Internet access. This machine requires: * Internet access * An OS that matches the air-gapped cluster: Ubuntu 22.04, Ubuntu 24.04, Rocky Linux 9u3, or RHEL 9u3. * An architecture that matches that of the cluster * Currently, the tarball supports one architecture at a time. So if the cluster has a mixed architecture, for example head node being x86, but kubernetes control plane nodes being arm64 then more manual effort is needed. This is not covered by this guide at this time. * The BCM package repositories must be configured on the machine * The BCM `airgap-scripts` directory from the matching BCM version must be on the machine #### 4.23.2.1 Machine Has `cm-setup` Installed If the `cm-setup` is already installed on the machine, then for BCM version 10, the airgap scripts are at: `/cm/local/apps/cm-setup/lib/python3.9/site-packages/cmsetup/plugins/kubernetes/airgap-scripts/` If the `cm-setup` module is loaded (enabled by default) then the environment variable `K8S_AG_SCRIPTS` has the path already defined. #### 4.23.2.2 Machine Does Not Have `cm-setup` Installed If the `cm-setup` package is not already installed on the machine, then * The administrator must add the airgap scripts to the PATH environment: **Example** ```bash # export PATH=$PATH:$K8S_AG_SCRIPTS ``` * `helm 3.x` (not 4.x at the time of writing, March 2026) must be added to `/usr/local/bin/helm` as per the instructions at [https://helm.sh/docs/intro/install/](https://helm.sh/docs/intro/install/). Alternatively, the following helper script, that BCM ships with `cm-setup`, can be used: **Example** ```bash root@internet-host:~# download_helm_binary.sh /tmp ~ ... ‘helm-v3.17.0-linux-amd64.tar.gz’ saved [18014603/18014603] linux-amd64/ linux-amd64/LICENSE linux-amd64/helm linux-amd64/README.md ~ ``` Optionally, a Helm check can be carried out with: **Example** ```bash root@internet-host:~# helm version version.BuildInfo{Version:"v3.17.0", GitCommit:"301108edc7ac2a8ba79e4ebf5701b0b6ce6a31e4", GitTreeState:"clean", GoVersion:"go1.23.4"} ``` * Skopeo, an image manager, is installed: **Example** ```bash root@internet-host:~# apt install skopeo ``` * To work around a potential Docker Hub pull rate limit issue ([https://docs.docker.com/docker-hub/usage/pulls/](https://docs.docker.com/docker-hub/usage/pulls/)) the administrator can now sign up for an account at [https://app.docker.com/signup](https://app.docker.com/signup), and authenticate with `skopeo login docker.io`. This is to help with running the airgapped installation scripts in the next bullet point. Any credentials are fine—the credentials will not end up in the air-gapped tarball or environment. They are only needed for authentication to conveniently pull images from Docker Hub. This is because unauthenticated pulls are limited to a rate of 100 per 6 hours. Authenticated pulls, on the other hand, are allowed at a rate of 200 per 6 hours, which should be sufficient for preparing the airgapped tarball. * A working directory, for example `/root/airgapped`, can then be created, and installation scripts within the airgapped directory can then be run: **Example** ```bash root@internet-host:~# pwd /root root@internet-host:~# mkdir airgapped; cd airgapped root@internet-host:~/airgapped# download_ubuntu_packages.sh --kube-version=1.32 # for Ubuntu Linux ... [root@internet-host airgapped]# download_r9u3_packages.sh --kube-version=1.32 # for RHEL variants ... root@internet-host:~/airgapped# download_container_images.sh --kube-version=1.32 ... root@internet-host:~/airgapped# download_helm_charts.sh ... ``` The preceding scripted downloads specify the `--kube-version`, and should be adjusted to match the required upgrade version. For Kubernetes v1.35, the `--kube-version` flags are optional. To ensure the commands all succeed, it is best to execute them in sequence and let them complete. The download scripts all write their output to the current working directory. * The `airgapped` directory can now be tarred: **Example** ```bash root@internet-host:~/airgapped# ls helm-charts packages packages-dgx packages-non-dgx root@internet-host:~/airgapped# cd .. root@internet-host:~# tar -czf airgapped.tar airgapped ``` The `airgapped.tar` file should be saved to some transfer media (for example: a USB stick), and should be copied over to the active head node of the air-gapped cluster. ### 4.23.3 Head Node Preparation Steps On the active head node of the air-gapped cluster * The `airgapped.tar` file is extracted: ```bash root@basecm10:~# tar -xvf airgapped.tar ``` The remaining steps are carried out inside the `airgapped` directory that is untarred. For Ubuntu systems, the `airgapped` directory should be moved to `/tmp/airgapped`. This is necessary because `apt` runs as the `apt` user, which lacks permissions to access packages in `/root`. If the move is not carried out, then `File not found` errors show up during package installation. A future update is planned to automate this, but for now, on Ubuntu head nodes, `/tmp` is used: **Example** ```bash root@basecm10:~# mv airgapped /tmp/airgapped ``` * The `airgapped` directory is entered: **Example** ```bash [root@basecm10 ~]# cd airgapped # for non-Ubuntu head node root@basecm10:~# cd /tmp/airgapped # for Ubuntu head node ``` * The `airgapped` helper scripts are added to `PATH`: **Example** ```bash root@basecm10:~/tmp/airgapped# export PATH=$PATH:$K8S_AG_SCRIPTS ``` * The packages are installed on the head node and the required software images. For BCM HA clusters, these steps must be repeated on the secondary head node. **Example** ```bash root@basecm10:~/tmp/airgapped# install_ubuntu_packages.sh # For Ubuntu Linux [root@basecm10 airgapped]# install_r9u3_packages.sh # For Rocky or RHEL 9u3 Linux ``` Examples: Ubuntu Linux with two software images for Kubernetes installation: **Example** ```bash root@basecm10:~/tmp/airgapped# install_ubuntu_packages.sh /cm/images/k8s-control-plane-image \ /cm/images/k8s-worker-image ``` Rocky or RHEL 9u3 Linux with one software image: **Example** ```bash [root@basecm10 airgapped]# install_r9u3_packages.sh /cm/images/default-image ``` For HA BCM clusters, these examples must be carried out on the secondary head node too. ### 4.23.4 Pushing All Container Images And Helm Charts * All Docker images are now pushed to the Docker registry: **Example** ```bash [root@basecm10 airgapped]# push_container_images.sh --registry master.cm.cluster:5000 --kube-version=1.35 ``` The same `--kube-version` flag must be chosen as in the download procedure earlier (1.35, 1.34 or 1.33). If the default value of 1.35 is used, then the flag can be left out completely. * The Helm charts can now be pushed too: **Example** ```bash [root@basecm10 airgapped]# push_helm_charts.sh --registry master.cm.cluster:5000 ``` Other options can be viewed with the `-h|--help` option. ### 4.23.5 Upgrading The Kubernetes Cluster The upgrade steps start with the control plane nodes. #### 4.23.5.1 Updating The First Control Plane Node The node that is to be upgraded should first be updated to the latest binaries. This can be done with an `imageupdate` run from `cmsh` on the head node ([section 5.6.2 of the *Administrator Manual*](/admin/provisioning#562-updating-running-nodes-with-cmsh-using-imageupdate)). For example for a node with the arbitrary name of `controlplane01`: **Example** ```bash root@headnode:~# cmsh -c 'device; imageupdate -w controlplane01 --wait' ... ...[notice] headnode: Provisioning started: sending headnode:/cm/images/default-image to controlplane01:/, mode UPDATE, dry run = no ... ...[notice] headnode: Provisioning completed: sent headnode:/cm/images/default-image to controlplane01:/, mode UPDATE, dry run = no ``` If the `-w|--write` flag is not used, then a dry-run is carried out. After the image update is done, the `kubeadm upgrade plan` command is run from the node itself: **Example** ```bash root@headnode:~# ssh controlplane01 ... root@controlplane01:~# kubeadm upgrade plan --kubeconfig=/root/.kube/config-default ``` Explicitly providing the kubeconfig for the correct Kubernetes cluster is a good practice. This is important if the BCM cluster has multiple Kubernetes clusters. The following session shows output from an upgrade plan from 1.32.10 to 1.33.11. The plan shows that there is indeed a newer 1.32.11 available already: **Example** ```bash ... [upgrade/versions] Target version: v1.32.10 [upgrade/versions] Latest version in the v1.32 series: v1.32.11 Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply': COMPONENT NODE CURRENT TARGET kubelet node001 v1.32.10 v1.32.11 kubelet node002 v1.32.10 v1.32.11 kubelet node003 v1.32.10 v1.32.11 kubelet node004 v1.32.10 v1.32.11 kubelet node005 v1.32.10 v1.32.11 kubelet node006 v1.32.10 v1.32.11 kubelet node007 v1.32.10 v1.32.11 kubelet node008 v1.32.10 v1.32.11 Upgrade to the latest version in the v1.32 series: COMPONENT NODE CURRENT TARGET kube-apiserver node001 v1.32.10 v1.32.11 kube-apiserver node002 v1.32.10 v1.32.11 kube-apiserver node003 v1.32.10 v1.32.11 kube-controller-manager node001 v1.32.10 v1.32.11 kube-controller-manager node002 v1.32.10 v1.32.11 kube-controller-manager node003 v1.32.10 v1.32.11 kube-scheduler node001 v1.32.10 v1.32.11 kube-scheduler node002 v1.32.10 v1.32.11 kube-scheduler node003 v1.32.10 v1.32.11 kube-proxy 1.32.10 v1.32.11 CoreDNS v1.11.3 v1.12.0 You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.32.11 ``` The upgrade plan suggests that an upgrade to v1.32.11 must be done first. However, the skew versions policies ([here](#4231-prerequisites)) allow jumping to v1.33.6. A check of the changelogs through the appropriate versions should be done. The changelogs are located under: [https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/) and are well-organized. After studying these, if the cluster administrator decides it makes sense to just jump ahead to v1.33.6, then that can be carried out as follows: Instead using `v1.32.11` as suggested in the preceding session plan, the value `v1.33.6` is used. The `--kubeconfig` flag is also set as a matter of best practice: **Example** ```bash root@controlplane01:~# kubeadm upgrade apply v1.33.6 --kubeconfig=/root/.kube/config-default ``` The following should then show in the output: **Example** ```bash [upgrade] SUCCESS! A control plane node of your cluster was upgraded to "v1.33.6". ``` **Validation can be done by:** * checking that the first control plane node is in the `Ready` state, and running the target Kubernetes version **Example** ```bash root@controlplane01:~# kubectl get nodes NAME STATUS ROLES AGE VERSION basecm10 Ready control-plane,master 6h7m v1.33.6 controlplane01 Ready control-plane,master,worker 6h5m v1.33.6 ... ``` * checking that all critical pods are running and in the `Ready` state, especially those in `kube-system` namespace: **Example** ```bash root@controlplane01:~# kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE calico-system calico-apiserver-5699d94dbc-hmj76 1/1 Running 0 6h11m calico-system calico-apiserver-5699d94dbc-hmtmw 1/1 Running 0 6h11m calico-system calico-kube-controllers-67c964bc98-5957z 1/1 Running 0 6h11m calico-system calico-node-2jrxv 1/1 Running 0 6h11m ... kube-system coredns-66bc5c9577-lsvdt 1/1 Running 0 6h13m kube-system coredns-66bc5c9577-w7jhm 1/1 Running 0 6h13m kube-system kube-apiserver-basecm10 1/1 Running 0 6h12m kube-system kube-apiserver-controlplane01 1/1 Running 0 6h8m kube-system kube-controller-manager-basecm10 1/1 Running 1 6 ... ``` An extra check is to see if the appropriate kube-apiserver version image, v1.33.6, is being used by the upgraded `controlplane01` node, by executing: **Example** ```bash root@controlplane01:~# kubectl describe pod -n kube-system kube-apiserver-controlplane01 | grep Image: Image: master.cm.cluster:5000/kube-apiserver:v1.33.6 ``` #### 4.23.5.2 Updating Subsequent Control Plane Nodes The following commands can be repeated on the other control plane nodes. From the head node: **Example** ```bash root@basecm10:~# cmsh -c 'device; imageupdate -w --wait' ``` From the node itself: **Example** ```bash root@node001:~# kubeadm upgrade node --kubeconfig /root/.kube/config-default ``` In the preceding, an `upgrade apply` with version does not need to be provided. The validation steps from before ([here](#airgapped-controlplane-validation)) should be repeated. If all control-plane nodes are running without issues, then the worker nodes can also be updated. #### 4.23.5.3 Updating The Worker Nodes In The Airgapped Update Updating worker nodes is done in a similar way to updating control plane nodes, but usually with some additional care, since worker nodes can be running user workloads. Section [4.22.6](#4226-updating-the-worker-nodes), about updating the worker nodes in a non-airgapped cluster, has more on being careful during worker node updates. In particular, it explains how to drain nodes beforehand, so that end users do not get upset by lost jobs. The upgrade step itself is carried out with: **Example** ```bash root@workernode001:~# kubeadm upgrade node --kubeconfig /root/.kube/config-default ``` The `--kubeconfig` flag should be provided in the preceding command. To summarize, a node that needs to be drained requires the following commands: **On the head node:** **Example** ```bash root@basecm10:~# kubectl drain $host --ignore-daemonsets # check if drain succeeds # update its software image if needed root@basecm10:~# cmsh -c "device use $host; imageupdate -w --wait" ``` **On the non-head node:** **Example** ```bash root@:~# kubeadm upgrade node --kubeconfig=/root/.kube/config- root@:~# systemctl daemon-reload root@:~# systemctl restart kubelet root@:~# kubectl uncordon $host # check if uncordon succeeds ``` Validation can be done, as before ([here](#airgapped-controlplane-validation)), by: * checking that all critical pods are running and in the `Ready` state, especially those in `kube-system` namespace, by running: `kubectl get nodes` * checking that all critical pods are running and in the Ready state (especially those in `kube-system` namespace, but not only those), by running: `kubectl get pod -A`