Kubernetes

Kubernetes is an open-source platform for automating deployment, scaling, and operations of application containers across clusters of hosts. With Kubernetes, it is possible to:

• scale applications on the fly

• seamlessly update running services

• optimize hardware usage by using only the resources that are needed

BCM provides the administrator with the required packages, allows Kubernetes to be set up on a cluster, and manages and monitors Kubernetes. More information about the design of Kubernetes, its command line interfaces, and other Kubernetes-specific details, can be found at the official online documentation at https://kubernetes.io/docs/.

To deploy most of Kubernetes, NVIDIA Base Command Manager from version 10.0 onwards uses kubeadm (https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/).

BCM runs CoreDNS, the Kubelet component (https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/), and the entire control plane inside Kubernetes.

The distributed key-value store used for Kubernetes, etcd, is typically run inside of Kubernetes on control planes. However, in BCM etcd is run outside of Kubernetes.

Kubernetes integration with BCM is available for Kubernetes v1.28, 1.29, v1.30, v1.31, and v1.32 at the time of writing of this paragraph (April 2025). Kubernetes runs on the x86_64 and aarch64 architecture for all the BCM-supported Linux distributions. For a more up-to-date status, the features matrix at https://service.bcm.nvidia.com/feature-matrix/ can be checked.

The version that is running on a cluster can be found with:

Example

root@basecm10:~# kubeadm version -o short
v1.34.5

The versions that are available with BCM integration can be found with the cm-kubernetes-setup utility (section 4.2):

Example

root@basecm10:~# cm-kubernetes-setup --list-versions
1.35
1.34
1.33 (NVIDIA AI Enterprise certified)
1.32 (NVIDIA AI Enterprise certified)

4.1 Reference Architecture

A reference architecture for Kubernetes in BCM comprises:

• etcd nodes: An etcd cluster—the Kubernetes distributed key-value storage—runs on an odd number (1, 3, 5 …) of nodes.

• control plane nodes: typically run on head nodes or on dedicated nodes. 2 or 3 are recommended.

• worker nodes: typically run on regular nodes that are designated to run user workloads.

To avoid single point of failure and to achieve high availability, a minimum of three nodes is recommended for etcd, and a minimum of two control plane nodes is recommended.

Load Balancing An NGINX server is configured on the head node(s) and on all other nodes involved in the Kubernetes cluster (control plane or worker nodes). This NGINX server takes care of exposing the Kubernetes API server on a specific port, and load balances requests to the control plane nodes. Should one of those nodes go down it can detect this and stop sending requests until the node comes back up.

Since BCM version 8.2, multiple clusters of Kubernetes can be deployed. In such a configuration, the same nodes cannot be shared across different Kubernetes clusters. Because of the NGINX server, a port is reserved on the head node(s) for every Kubernetes cluster. This is required for Kubernetes HA , and it also allows kubectl and other tools such as Helm to be used from the head node, to access each Kubernetes cluster.

The same NGINX server is also used to similarly expose the NGINX Ingress Controller, if this has been chosen during setup. Alternatively, MetalLB (https://metallb.universe.tf/) is also supported, and provides a different approach towards load balancing for Kubernetes that is more similar to managed cloud solutions.

4.1.1 Hardware Requirements

To run properly, Kubernetes deployment requires a per-node minimum of

• 16 GB RAM

• two CPU cores

Without these minimum requirements, deployment may succeed, but further configuration, such as starting up Jupyter kernels, may fail with problems that are hard to diagnose. For example, error reporting may not show a lack of resources because some components are automatically deleted.

4.2 Kubernetes Setup

The usual, and recommended way, to install Kubernetes with BCM is to install it interactively from a TUI session (section 4.2.6).

An alternative way, typically used for automated non-interactive setups, is to use a plain command line specification (section 4.2.5), which usually results in a long command line.

BCM deploys Kubernetes with the cm-kubernetes-setup utility, which is part of the cm-setup package. Several recent versions of Kubernetes are offered (figure 4.1):

Figure 4.1: Kubernetes setup TUI session (section 4.2.6): version selection screen

BCM provides or uses the following Kubernetes-related packages:

• conntrack (conntrack-tools on RHEL-based distributions) and nginx: These distribution packages are always installed on the head node(s) and on the master and worker node(s).

• cm-etcd: This BCM package is installed on the nodes selected for etcd. In a similar way to the case of Harbor or Docker Registry (section 3.2), the nodes that run etcd are protected by BCM with the datanode option (here). For etcd nodes, the option is added during the cm-kubernetes-setup installation. As in the case for the registries, the datanode option is set in order to help prevent the administrator from wiping out the existing state of etcd nodes. Wiping out the state of etcd nodes means that the Kubernetes cluster becomes incoherent and that it cannot be restored to where it was just before the etcd nodes were wiped. The etcd version installed by the BCM package is 3.5.8.

• cm-containerd: This BCM package has the containerd runtime.

• cm-nvidia-container-toolkit: This BCM package has the NVIDIA container toolkit (includes NVIDIA container runtime).

• cm-kube-diagnose: This BCM package has Helper tools to diagnose malfunctioning Kubernetes clusters.

Kubernetes .rpm and .deb packages themselves are installed from the Kubernetes community-owned software repositories (https://kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/). These repositories host Kubernetes versions starting from v1.24.0 at the time of writing of this paragraph (March 2024).

4.2.1 Kubernetes Networking

Early on during the wizard session (figure 4.2), a name for the cluster is requested. The wizard pre-fills it with default, but this should not be confused with the Kubernetes default namespace. Here, the name is used instead, inside BCM, to identify the cluster, configuration files, and other resources such as module files.

Figure 4.2: Kubernetes setup TUI session: networking selection screen

This screen also allows the following important choices:

• Kubernetes external FQDN: This is the FQDN that is placed as one of the entries in the public-facing certificates generated for this Kubernetes cluster.

  Configuring the public-facing certificate of the NGINX Ingress Controller is discussed further in section 4.22.12.

• Service network base address and Service network netmastk bits: These define the CIDR for the service network. The wizard pre-fills the fields. It also tries to avoid pre-filling them with overlapping network ranges, by taking any existing network known to BCM into account.

• Pod network base address and Pod network netmask bits: These define the CIDR for the pod network. The wizard pre-fills these. It also tries to avoid pre-filling them with overlapping network ranges, by taking any existing network known to BCM into account. By default, entire /24 network ranges are assigned to individual Kubernetes nodes from the pod CIDR.

The packages are installed automatically from the repository when the administrator runs cm-kubernetes-setup from the command line.

The log file produced by the setup can be found in /var/log/cm-kubernetes-setup.log.

4.2.2 Kubernetes Core Add-ons

During setup, some critical add-on components such as the Networking Component for Kubernetes are automatically deployed. Most components are in the kube-system namespace, but others have their own namespaces. In BCM some add-ons are treated as Kubernetes applications (Chapter 5), and belong to the default app group, system. In the future Helm is expected to manage most, if not all, components instead of BCM Kubernetes applications.

The user is prompted for a Networking Component, Kubernetes operators (managed via Helm), Kubernetes add-ons (managed via Kubernetes applications), and more. They all result in either Helm charts being deployed, or in Kubernetes applications being managed from BCM.

A cmsh treeview illustrating the hierarchy to access these applications is:

[cmsh]
  |-- ...
  |
  |-- kubernetes[default]
  |              `-- appgroups[system]
  |                            `-- applications
  |-- ...

Helm charts can be found using the helm command. For example using helm list -A -a on a cluster (after loading the correct module file).

In the past the DNS component was also provided as an add-on. However since BCM version 10.0, kubeadm is used to bundle CoreDNS as part of the default control plane.

Kubernetes apps are discussed further in Chapter 5. Kubernetes operators are discussed further in Chapter 6.

4.2.2.1 Kubeadm Components

Components deployed as part of the default Kubernetes control plane are:

• Kubernetes API server

• Kubernetes Scheduler

• Kubernetes Controller Manager

• Kubernetes Proxy

• Core DNS

These are deployed through the “static pod” mechanism (https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/). Kubernetes control plane components are described at: https://kubernetes.io/docs/concepts/overview/components/.

CoreDNS CoreDNS is the DNS server add-on for internal service discovery. It reads the IP addresses of services and pods from etcd, and resolves domain names for them. If a domain name is not found because the domain is external to the Kubernetes cluster, then CoreDNS forwards the request to the main DNS server. BCM uses CoreDNS version 1.11.1 with Kubernetes version 1.31.

4.2.2.2 Networking Component

An important component managed through BCM’s Kubernetes Apps is the Networking component. This adheres to the Container Networking Interface, or CNI interface. At the time of writing the available CNIs in the setup wizard are either Calico or Flannel. However BCM does support other CNIs, for example Cilium, as described in the knowledge base article at https://enterprise-support.nvidia.com/s/article/installing-cilium-networking-for-kubernetes.

Flannel

Flannel is a simple and easy way to configure a layer 3 network fabric designed for Kubernetes.

Calico

Calico is a popular open source networking and network security solution for Kubernetes. It provides network connectivity between workloads and security policies features. It can be configured to use eBPF. BCM uses Calico version 3.26.

Further details on Calico can be found at https://docs.projectcalico.org/.

If the Kubernetes cluster is composed of more than 50 nodes, then the Calico component Typha is also automatically deployed for better scalability. The number of Typha replicas is calculated by allocating one Typha replica per 150 nodes, with a minimum of 3 (above 50 nodes) and a maximum of 20.

If an initial deployment of the Kubernetes cluster has fewer than 50 nodes, but nodes are then added to the Kubernetes cluster so that the 50 node threshold is exceeded, then Typha is not automatically enabled. In this case, Typha can be enabled manually via cmsh as follows:

Example

    [basecm10->kubernetes[default]->appgroups[system]->applications[calico]]% environment
    [basecm10->kubernetes[default]->appgroups[system]->applications[calico]->environment]% list
    Name (key)             Value            Nodes environment
    ---------------------- ---------------- ------------------
    calico_typha_replicas  0                no
    calico_typha_service   none             no
    cidr                   10.141.0.0/16    no
    [basecm10->kubernetes...[calico]->environment]% set calico_typha_service value calico-typha
    [basecm10->kubernetes*...[calico*]->environment*]% set calico_typha_replicas value 3
    [basecm10->kubernetes*[default*]->appgroups*[system*]->applications*[calico*]->environment*]% commit
    [basecm10->kubernetes[default]->appgroups[system]->applications[calico]->environment]%

Cilium

Cilium (https://cilium.io/) is a networking, observability, and security solution with an eBPF-based dataplane.

4.2.3 Kubernetes Optional Add-ons

The following add-ons are installed by default unless otherwise noted. However, the user can choose to skip some or all of them during the setup.

4.2.3.1 NGINX Ingress Controller

The official Kubernetes Ingress controller add-on is built around the Kubernetes Ingress resource, using a ConfigMap to store the NGINX configuration. Ingress provides HTTP and HTTPS routes from outside a Kubernetes cluster to services within the cluster. Traffic routing is controlled by rules defined in the Ingress resource. BCM uses NGINX Ingress Controller version 1.5.1.

By default, BCM suggests the following ports for Ingress: 30080 is the default that is set for the HTTP, and port 30443 is the default that is set for HTTPS.

These 2 ports are exposed on every Kubernetes node, both masters and workers.

The Ingress Controller is deployed as a NodePort service, which means that it comes with a default range of possible port values of 30000-32767.

4.2.3.2 Kubernetes Dashboard

Kubernetes Dashboard is the web user interface add-on for GUI cluster administration and metrics visualization. BCM uses Kubernetes Dashboard version 2.7.0.

There are two ways to access the dashboard:

• Using kubectl proxy and accessing http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/ To use the proxy, kubectl must be set up locally (section 9.3.2 of the User Manual).

• Users on an external network can log in to kubectl or Kubernetes Dashboard by following the procedures described in section 4.16.

If NGINX Ingress Controller is deployed, then a link pointing to the Kubernetes dashboard can also found on the BCM landing page.

4.2.3.3 Kubernetes Metrics Server

The Kubernetes Metrics Server is an add-on that is a replacement for Heapster. It aggregates metrics, and provides container monitoring and performance analysis. It exposes metrics via an API. BCM runs Metrics server version 0.6.3.

4.2.3.4 Kubernetes State Metrics

kube-state-metrics is an add-on agent to generate and expose cluster-level metrics for Kubernetes objects. The project is not focused on the health of the individual Kubernetes components, but rather on the health of the various objects inside, such as deployments, nodes and pods.

4.2.4 Helm Kubernetes Package Manager

Helm is an add-on that manages charts, which are packages of pre-configured Kubernetes resources. The Helm component is installed and properly configured with BCM’s Kubernetes installation by default. It is initialized and ready for use by every Kubernetes user when the Kubernetes module is loaded. BCM uses Helm version 3.

When the Helm binary is installed during the Kubernetes setup, an offer is made to deploy charts during the setup process. Most operators are documented in Chapter 6.

Other parts of the wizard also determine which Helm charts are deployed as part of the setup for the following:

• Kyverno and Kyverno Policies (section 4.10.1)

• BCM Permissions Manager (section 4.15)

• BCM Local Path Provisioner (section 4.19)

Example

root@basecm10:~# helm list -A -a
NAME                        NAMESPACE             CHART                                         APP VERSION
cm-jupyter-kernel-operator  cm                ... cm-jupyter-kernel-operator-0.1.10             0.1.10
cm-kubernetes-mpi-operator  cm                ... mpi-operator-0.4.0                            0.4.0
gpu-operator                gpu-operator      ... gpu-operator-v23.9.1                          v23.9.1
kyverno                     kyverno           ... kyverno-3.0.4                                 v1.10.2
kyverno-policies            kyverno           ... kyverno-policies-3.0.3                        v1.10.2
local-path-provisioner      cm                ... cm-kubernetes-local-path-provisioner-0.0.26   0.0.26
metallb                     metallb-system    ... metallb-0.14.3                                v0.14.3
network-operator            network-operator  ... network-operator-23.10.0                      v23.10.0
permissions-manager         cm                ... cm-kubernetes-permissions-manager-0.4.8       0.4.8
postgres-operator           postgres-operator ... postgres-operator-1.10.1                      1.10.1
prometheus-adapter          prometheus        ... prometheus-adapter-3.3.1                      v0.9.1
prometheus-operator         prometheus        ... kube-prometheus-stack-35.5.1                  0.56.3
spark-operator              spark-operator    ... spark-operator-1.1.27                         v1beta2-1.3.8-3.1.1

4.2.5 Kubernetes Setup From The Command Line

The cm-kubernetes-setup command line utility has the following usage synopsis:

[root@basecm10 ~]# cm-kubernetes-setup -h
usage: Kubernetes Setup cm-kubernetes-setup [-c <config_file>]
[--list-versions]
[--list-operators-versions]
[--cluster CLUSTER_NAME]
[--skip-docker]
[--skip-reboot]
[--skip-image-update]
[--skip-dns-configuration-check]
[--skip-package-manager-update-check]
[--skip-install-repos]
[--add-user USERNAME_ADD]
[--list-users]
[--get-user GET_USER]
[--modify-user USERNAME_MODIFY]
[--remove-user USERNAME_REMOVE]
[--namespace NAMESPACE]
[--add-to-namespace]
[--remove-from-namespace]
[--role edit,admin,view,cluster-admin]
[--runas-uid RUNAS_UID]
[--runas-gids RUNAS_GIDS]
[--user-paths USER_PATHS]
[--allow-all-uids]
[--operators OPERATORS]
[--backup-permissions FILE]
[--restore-permissions FILE]
[--list-operators]
[--update-addons]
[--remove]
[--yes-i-really-mean-it]
[--remove-ceph-storage]
[--pull]
[--images IMAGES]
[--nodes NODES]
[--node-selector NODE_SELECTOR]
[--pull-registry-server PULL_REGISTRY_SERVER]
[--pull-registry-username PULL_REGISTRY_USERNAME]
[--pull-registry-email PULL_REGISTRY_EMAIL]
[--pull-registry-password PULL_REGISTRY_PASSWORD]
[--docker-io-username DOCKER_IO_USERNAME]
[--docker-io-password DOCKER_IO_PASSWORD]
[--kubeadm-image-repository KUBEADM_IMAGE_REPOSITORY]
[--containerd-sandbox-image CONTAINERD_SANDBOX_IMAGE]
[--skip-netq-prerequisites-checks]
[-v]
[--store-name-aliases]
[--no-distro-checks]
[--json]
[--output-remote-execution-runner]
[--on-error-action debug,remotedebug,undo,abort]
[--skip-packages]
[--min-reboot-timeout <reboot_timeout_seconds>]
[--allow-running-from-secondary]
[--dev]
[-h]
optional arguments:
  --cluster CLUSTER_NAME
                        Name of the referred Kubernetes cluster
  -h, --help            Print this screen
common:
  Common arguments
  -c <config_file>      Load runtime configuration for plugins from a YAML config file
Supported versions information:
  --list-versions       Show supported Kubernetes versions
  --list-operators-versions
                        List available for selection Kubernetes operators versions
installing Kubernetes clusters:
  Flags for installing or managing Kubernetes clusters
  --skip-docker         Skip the Docker installation steps.
  --skip-reboot         Skip the reboot steps.
  --skip-image-update   Skip the image update steps.
  --skip-dns-configuration-check
                        Skip DNS configuration check before install (setup will not continue of it finds DNS
                        malfunctioning).
  --skip-package-manager-update-check
                        Skip Package Manager update/refresh check before install (setup will not continue if
                        'apt update' fails on Ubuntu for example).
  --skip-install-repos  Skip installation of kubernetes and helm package repos.(assumes they have been manually
                        configured, e.g., in the case of air-gapped).
user management:
  Flags for adding a new user in a Kubernetes cluster
  --add-user USERNAME_ADD
                        Create a new user in a Kubernetes cluster
  --list-users          Get information about configured Kubernetes users
  --get-user GET_USER   Get information about configured Kubernetes users
  --modify-user USERNAME_MODIFY
                        Modify user in a Kubernetes cluster
  --remove-user USERNAME_REMOVE
                        Remove existing user from a Kubernetes cluster
  --namespace NAMESPACE
                        Specify namespace for user (--get-user, --modify-user) role binding
  --add-to-namespace    Indicate if permissions to manage namespace needs to be granted for a given user
                        (--modify-user)
  --remove-from-namespace
                        Indicate if permissions to manage namespace needs to be revoked for a given user
                       (--modify-user)
  --role edit,admin,view,cluster-admin
                        Specify role for the new (--add-user) and existing (--modify-user) role binding
                        Default: edit)
                        For 'cluster-admin' namespace flag is ignored
  --runas-uid RUNAS_UID
                        UID is allowed to be used in unprivileged pods (--add-user, --modify-user)
  --runas-gids RUNAS_GIDS
                        Comma-separated list of GIDs allowed to be used in unprivileged pods (--add-user,
                        --modify-user)
  --user-paths USER_PATHS
                        Comma-separated list of paths user is able to mount in pods (--add-user, --modify-user)
  --allow-all-uids      Allow user to run processes in pods as any user (--add-user, --modify-user)
                        hostPath volumes will be disabled for such pods
  --operators OPERATORS
                        Comma-separated list of operators user has access to (--add-user, --modify-user)
backup or restore Permission Manager user configurations:
  Flag for managing permission manager user configuration
  --backup-permissions FILE
                        Save permissions to file
  --restore-permissions FILE
                        Restore permissions from file. Workload which is already run by users in their
                        namespaces will be affected
list available operators:
  Flag to list available Kubernetes operators
  --list-operators      List available Kubernetes operators
update kubernetes addons:
  Flags for updating Kubernetes addons
  --update-addons       Update Addons
removing Kubernetes clusters:
  Flags for removing a Kubernetes cluster
  --remove              Remove a Kubernetes cluster
  --yes-i-really-mean-it
                        Required for additional safety
  --remove-ceph-storage
                        Remove Kubernetes osd pool from Ceph cluster
pulling images to the nodes:
  Flags for pulling images to the nodes
  --pull                Pull images to the nodes
  --images IMAGES       Comma-separated list of images to pull (--pull)
  --nodes NODES         Comma-separated list of nodes to pull images to (--pull)
  --node-selector NODE_SELECTOR
                        Selector (label query) to filter on, supports '=', '==', and '!='.(e.g.
                        key1=value1,key2=value2) (--pull)
  --pull-registry-server PULL_REGISTRY_SERVER
                        Registry server to authenticate
  --pull-registry-username PULL_REGISTRY_USERNAME
                        Registry username
  --pull-registry-email PULL_REGISTRY_EMAIL
                        Email to authenticate
  --pull-registry-password PULL_REGISTRY_PASSWORD
                        Registry password
docker.io:
  Flags for authenticating to docker hub
  --docker-io-username DOCKER_IO_USERNAME
                        Username
  --docker-io-password DOCKER_IO_PASSWORD
                        Password
kubeadm:
  Flags for configuring kubeadm
  --kubeadm-image-repository KUBEADM_IMAGE_REPOSITORY
                        Custom imageRepository
containerd:
  Flags for configuring containerd
  --containerd-sandbox-image CONTAINERD_SANDBOX_IMAGE
                        Custom sandbox image (e.g. my-registry:5000/pause:3.8)
netq:
  Flags for configuring NetQ
  --skip-netq-prerequisites-checks
                        Provide this flag to skip precondition checks such as checking for disk size, Linux
                        distro, etc.
advanced:
  Various *advanced* configuration options flags.
  -v, --verbose         Verbose output
  --store-name-aliases  Store hostname aliases for head nodes (active and passive) and default category
  --no-distro-checks    Disable distribution checks based on ds.json
  --json                Use json formatting for log lines printed to stdout
  --output-remote-execution-runner
                        Format output for CMDaemon
  --on-error-action debug,remotedebug,undo,abort
                        Upon encountering a critical error, instead of asking the user for choice, setup will
            do selected action.
  --skip-packages       Skip the any stages which install packages. Requires packages to be already installed.
  --min-reboot-timeout <reboot_timeout_seconds>
                        How long to wait for nodes to finish reboot (default and minimum allowed: 300 seconds).
  --allow-running-from-secondary
                        Allow to run the wizard from the secondary when it is the active head node.
  --dev                 Enables additional command line arguments

The cm-kubernetes-setup utility should be executed on the console.

4.2.5.1 Dealing With A Pre-existing Docker Installation

Docker (Chapter 2) is no longer a requirement for Kubernetes configured by BCM. This is because Kubernetes can directly interface with containerd through its Container Runtime Interface (CRI). Docker can co-exist with Kubernetes, and can be set up as discussed in section 2.1.

4.2.6 Kubernetes Setup From A TUI Session

When the Kubernetes installation is carried out using cm-kubernetes-setup without any options, a TUI wizard starts up. The administrator can answer several questions in the wizard. Questions that are asked include questions about which of the node categories or which of the individual nodes should be configured to run the Kubernetes services. There are also questions about the service and pod networks parameters, the port numbers that will be configured for the daemons, whether to install specific add-ons, and so on. After the wizard has been completed, a configuration file can be saved that can be used to set up Kubernetes.

The configuration file can be deployed immediately from the wizard, or it can be deployed later by specifying it as an option to cm-kubernetes-setup, in the form -c <file>.

If no deployment has been carried out earlier, then the main operations screen of the wizard shows just two options, Deploy and Exit.

If deployment has already been carried out, then the further options that are available are also displayed (figure 4.3):

Example

[root@basecm10 ~]# cm-kubernetes-setup
           TUI session starts up:

Figure 4.3: Kubernetes setup TUI session: main operations screen after a deployment

The deployment via CLI or via TUI assigns the appropriate roles, and adds the new Kubernetes cluster. The deployment adds health checks to the monitoring configuration, and it generates certificates for the Kubernetes daemons.

Calico is set as the Kubernetes network plugin by default. Flannel is an option.

The master, worker, and etcd nodes can be assigned to specific nodes or categories.

The network configuration settings for the Kubernetes cluster can be specified. Ports have default assignments, but can be re-assigned as needed. The etcd spool file path can be set.

The following options are also possible:

• a registry mirror from DockerHub can be specified

• the Kubernetes API server can be exposed to the external network

• the internal network used by Kubernetes nodes can be selected

Add-ons that are available are:

• Ingress Controller (Nginx)

• Kubernetes Dashboard

• Kubernetes Metrics Server

• Kubernetes State Metrics

Operator packages are application managers, and are described further in Chapter 6. Operators that can be installed are:

• cm-jupyter-kernel-operator

• cm-kubernetes-postgresql-operator

• cm-kubernetes-spark-operator

The permission manager—an application for role-based access control—can also be configured.

4.2.7 Testing Kubernetes

To test that Kubernetes works, the cluster-info command can be run. For example, on the head node, basecm10:

Example

[root@basecm10 ~]# module load kubernetes      #not actually needed -- autoloaded these days
[root@basecm10 ~]# kubectl cluster-info
Kubernetes control plane is running at https://127.0.0.1:10443
CoreDNS is running at https://127.0.0.1:10443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

After cm-kubernetes-setup finishes, and the regular nodes have been rebooted, the state of the nodes can be found by running the get nodes command:

Example

[root@basecm10 ~]# kubectl get nodes
NAME      STATUS   ROLES                  AGE   VERSION
node001   Ready    worker                 18h   v1.31
node002   Ready    worker                 18h   v1.31
basecm10  Ready    control-plane,master   18h   v1.31

A six node cluster should show the following Kubernetes installed add-ons, when using kubectl with the get all -n kube-system option (some lines truncated):

Example

[root@basecm10 ~]# kubectl get all -n kube-system
NAME                                           READY   STATUS    RESTARTS   AGE
pod/calico-kube-controllers-58497c65d5-skhgw   1/1     Running   0          26m
pod/calico-node-27xj7                          1/1     Running   0          26m
pod/calico-node-6hmm5                          0/1     Running   1          26m
pod/calico-node-987qv                          1/1     Running   0          26m
pod/calico-node-gcbcm                          1/1     Running   0          26m
pod/calico-node-hlsrj                          1/1     Running   0          26m
pod/calico-node-q7k4v                          1/1     Running   0          26m
pod/calico-node-qdbq5                          0/1     Running   0          26m
pod/calico-node-v2dxj                          1/1     Running   0          26m
pod/coredns-6768db756-8l9fs                    1/1     Running   0          26m
pod/coredns-6768db756-cs58q                    1/1     Running   0          26m
pod/kube-state-metrics-758ccc75d6-75dsn        1/1     Running   0          26m
pod/metrics-server-7b477dd7b9-2drkg            1/1     Running   0          26m
pod/metrics-server-7b477dd7b9-z6nch            1/1     Running   0          26m
NAME                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
service/calico-typha         ClusterIP   10.150.121.25    <none>        5473/TCP                 26m
service/kube-dns             ClusterIP   10.150.255.254   <none>        53/UDP,53/TCP,9153/TCP   26m
service/kube-state-metrics   ClusterIP   None             <none>        8080/TCP,8081/TCP        26m
service/metrics-server       ClusterIP   10.150.99.149    <none>        443/TCP                  26m
NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/calico-node   8         8         6       8            6           kubernetes.io/os=linux   26m
NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/calico-kube-controllers   1/1     1            1           26m
deployment.apps/calico-typha              0/0     0            0           26m
deployment.apps/coredns                   2/2     2            2           26m
deployment.apps/kube-state-metrics        1/1     1            1           26m
deployment.apps/metrics-server            2/2     2            2           26m
NAME                                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/calico-kube-controllers-58497c65d5   1         1         1       26m
replicaset.apps/calico-typha-68857595fc              0         0         0       26m
replicaset.apps/coredns-6768db756                    2         2         2       26m
replicaset.apps/kube-state-metrics-758ccc75d6        1         1         1       26m
replicaset.apps/metrics-server-7b477dd7b9            2         2         2       26m

The administrator can now configure the cluster to suit the particular site requirements.

4.3 Using GPUs With Kubernetes: NVIDIA GPUs

4.3.1 Prerequisites

The GPUs have to be recognized by the nodes, and have the appropriate drivers (such as cuda-driver) installed. Details on how to do this are given in section 9 of the Installation Manual.

To verify the GPUs are recognized and have drivers in place, the nvidia-smi command can be run. The response displayed for a GPU should look similar to the following:

Example

root@node001:~# nvidia-smi
Tue Dec  7 11:25:21 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla K40c          On   | 00000000:00:06.0 Off |                  Off |
| 23%   37C    P8    21W / 235W |      0MiB / 12206MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

If a non-BCM Containerd has already been deployed before Kubernetes is deployed, then cm-kubernetes-setup may replace an existing Containerd configuration file in order to enable NVIDIA GPU integration via a Kubernetes CNI plugin. This is because Containerd is configured by cm-kubernetes-setup, overwriting any previous configuration.

4.3.2 New Kubernetes Installation

As part of the setup, cm-kubernetes-setup assigns a new role to the Kubernetes worker nodes: generic::containerd.

The role has a Configurations submode, in which the containerd-cri object can be configured. The entry for Filename specifies the path to the cri.toml file, which contains content used by the container runtime interface on the Kubernetes worker nodes that have been assigned the role.

Example

[basecm10->configurationoverlay[kube-default-worker]->roles]% use generic::containerd
[...]->roles[generic::containerd]]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Name                             generic::containerd
Type                             GenericRole
Add services                     yes
Services                         containerd
Configurations                   <2 in submode>
Environments                     <1 in submode>
Exclude list snippets            <1 in submode>
Data node                        no
[...]->roles[generic::containerd]]% configurations
[...]->roles[generic::containerd]->configurations]% use containerd-cri
[...]->roles[generic::containerd]->configurations[containerd-cri]]% show
Parameter                        Value
-------------------------------- --------------------------------------------------
Name                             containerd-cri
Type                             static
Create directory                 yes
Filename                         /cm/local/apps/containerd/var/etc/conf.d/cri.toml
Filemask directory               0644
User name
Group name
Disabled                         no
Service action on write          RESTART
Service stop on failure          yes
Content                          <645B>
Filemask                         0644

The file with the CRI (Container Runtime Interface) configuration is created in directory:

     /cm/local/apps/containerd/var/etc/conf.d

and included into the main Containerd configuration file:

     /cm/local/apps/containerd/var/etc/config.toml

with the imports statement:

     imports = ["/cm/local/apps/containerd/var/etc/conf.d/*.toml"]

Whatever the container runtime that is selected, if NVIDIA GPU integration is required then the NVIDIA container toolkit is taken care of by the installer. The configuration for Containerd is added as a separate configuration file in the configurations submode, as nvidia-cri.

NVIDIA GPUs are integrated into Kubernetes using the NVIDIA GPU operator. This is discussed further in section 6.4.

4.3.3 Existing Kubernetes Installation

The NVIDIA GPU operator can always be deployed through Helm. The official documentation for the NVIDIA GPU operator is at

https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/overview.html

4.3.4 An Example Of Running A Workload Using NVIDIA GPUS: LLM Via NIM

NIM originally stood for NVIDIA Inference Microservices, but does not stand for anything now. This is because inference engines—the “I” in NIM—are not the only workloads that are intended to run on GPUs that use pod microservices.

NIM is now a product name, for a platform that hosts multiple GPU-based microservices. With NIM as a product name, “NVIDIA NIM microservices” is now regarded as a valid phrase, instead of standing for the excessively redundant phrase “NVIDIA NVIDIA Inference Microservices microservices”.

Once GPUs are configured and available for pods, then it is easy to deploy NVIDIA NIM microservices.

4.3.4.1 Requirements For NIM

A minimal setup to run NIM has the following requirements:

• A git CLI installed on the cluster node where the Kubernetes API and Kubernetes CLI are available.

• Access to the NGC catalog. This is used to acquire a personal API key to download the container image and model. The key must have access to the following services (figure 4.4):

  • AI Foundation Models and Endpoints

  • NGC Catalog

  • Private Registry

Figure 4.4: NIM personal key generation

• If the image repository does not provide a Helm chart with a container and model already, then

  nvcr.io/nim/meta/llama3-8b-instruct

  is used by default. Supported GPU configurations for Llama 3 8B are specified at:

  https://docs.nvidia.com/nim/large-language-models/latest/support-matrix.html#llama-3-8b-instruct

  GPU support configuration for the various LLMs is described at:

  https://docs.nvidia.com/nim/large-language-models/latest/support-matrix.html

4.3.4.2 Setting Up And Running The LLM Via NIM

A running LLM, llama3-8b-instruct, can be created and hosted as a service as follows:

[root@basecm10 ~]# git clone https://github.com/NVIDIA/nim-deploy.git
[root@basecm10 ~]# cd nim-deploy/helm
[root@basecm10 ~]# export NGC_CLI_API_KEY=<personal API key from NGC>
[root@basecm10 ~]# kubectl create namespace nim
[root@basecm10 ~]# helm --namespace nim install my-nim nim-llm/ \
    --set model.ngcAPIKey=$NGC_CLI_API_KEY \
    --set persistence.enabled=true

The preceding sets up a namespace nim with the name my-nim, deployed with the personal API key from NGC.

The existence of the service (sometimes called a microservice) can be checked with:

[root@basecm10 ~]# kubectl get service -n nim
NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
my-nim-nim-llm   ClusterIP   10.150.226.159   <none>        8000/TCP   2m29s

The API can be tested with:

[root@basecm10 ~]# curl -X POST http://10.150.226.159:8000/v1/completions \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -d '{"model": "meta-llama3-8b-instruct", "max_tokens": 64, "prompt": "Once upon a time"}'

A typical response is:

{
    "id": "cmpl-9fed783f0c914e8fa82625946a78cd5e",
    "object": "text_completion",
    "created": 1716571188,
    "model": "meta-llama3-8b-instruct",
    "choices": [
        {
            "index": 0,
            "text": ", there was a young man named Jack who lived in a small village at the",
            "logprobs": null,
            "finish_reason": null,
            "stop_reason": null
        }
    ],
    "usage": {
        "prompt_tokens": 5,
        "total_tokens": 69,
        "completion_tokens": 64
    }
}

4.3.4.3 More About NIM

More documentation on NIM can be found at:

https://docs.nvidia.com/nim/large-language-models/latest/getting-started.html

and

https://github.com/NVIDIA/nim-deploy/tree/main

4.4 Using GPUs With Kubernetes: AMD GPUs

4.4.1 Prerequisites

The GPUs have to be recognized by the node. One way to check this from within BCM is to run sysinfo for the node:

Example

[basecm10->device[basecm10]]% sysinfo | grep GPU
Number of GPUs                           1
GPU Driver Version                       4.18.0-193.el8.x86_64
GPU0 Name                                Radeon Instinct MI25

In order to make Kubernetes aware of nodes that have AMD GPUs, the AMD GPU device plugin has to be deployed as a DaemonSet inside Kubernetes. The official GitHub repository that hosts this plugin can be found at:

https://github.com/RadeonOpenCompute/k8s-device-plugin

The device plugin requires Kubernetes v1.16+, which has been around since BCM version 9.0. With some extra instructions, the plugin can also be made a part of BCM version 8.2.

The DaemonSet YAML file can be deployed with:

Example

kubectl create -f https://raw.githubusercontent.com/RadeonOpenCompute/k8s-device-plugin/v1.16/
k8s-ds-amdgpu-dp.yaml

4.4.2 Managing The YAML File Through CMDaemon

The plugin can be added by the user via the Kubernetes appgroups as an application. In the session that follows, it is given the arbitrary name device-plugin:

Example

[root@basecm10 ~]# wget https://raw.githubusercontent.com/RadeonOpenCompute/k8s-device-plugin/
v1.16/k8s-ds-amdgpu-dp.yaml -O /tmp/k8s-ds-amdgpu-dp.yaml
[root@basecm10 ~]# cmsh
[basecm10]% kubernetes
[basecm10->kubernetes[default]]% appgroups
[basecm10->kubernetes[default]->appgroups]% add amd
[basecm10->kubernetes*[default*]->appgroups*[amd*]]% applications
[basecm10->kubernetes*[default*]->appgroups*[amd*]->applications]% add device-plugin

The configuration of the plugin can be set to the YAML file, by setting the config parameter to take the value of the YAML file path.

Example

[basecm10->...[amd*]->applications*[device-plugin*]]% set config /tmp/k8s-ds-amdgpu-dp.yaml
[basecm10->kubernetes*[default*]->appgroups*[amd*]->applications*[device-plugin*]]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Name                             device-plugin
Revision
Format                           Yaml
Enabled                          yes
Config                           <914B>
Environment                      <0 in submode>
Exclude list snippets            <0 in submode>
[basecm10->kubernetes*[default*]->appgroups*[amd*]->applications*[device-plugin*]]% commit

The YAML file can also be edited within cmsh after it has been set, by running set config without a value.

There are older releases available, starting from Kubernetes v1.10, if needed. Saving this device-plugin YAML should result in pods being scheduled on all the non-tainted nodes, as seen by listing the pods (some columns elided):

[root@basecm10 ~]# kubectl get pod -n kube-system -l name=amdgpu-dp-ds -o wide
NAME                                   READY   STATUS    ... IP               NODE   ...
amdgpu-device-plugin-daemonset-66jl7   1/1     Running   ... 172.29.112.135   gpu001 ...
amdgpu-device-plugin-daemonset-8mh9w   1/1     Running   ... 172.29.152.130   gpu002 ...

4.4.3 Including Head Nodes as part of the DaemonSet:

BCM taints head nodes, so that they do not run non-critical pods. The taint can be removed with the “-” operator to allow non-critical pods to run:

Example

kubectl taint nodes basecm10 node-role.kubernetes.io/control-plane-

However, a more specific exception can be configured in the DaemonSet itself.

Within the YAML file, the following existing tolerations definition has to be modified, from:

      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists

to:

      tolerations:
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule
        operator: Exists

The modified toleration tolerates this taint, and therefore has the device plugin run on such tainted nodes.

4.4.3.1 Verifying That AMD GPUs Are Recognized By Kubernetes

If Kubernetes is aware of the AMD GPUs for a node then several mentions of amd.com/gpu are displayed when running the kubectl describe node command for the node. The following session shows output for a node gpu01, ellipsized for clarity:

Example

[root@basecm10 ~]# kubectl describe node gpu01
Name:                 gpu01
...
Capacity:
  amd.com/gpu:        3
  cpu:                64
  ephemeral-storage:  1813510Mi
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             527954676Ki
  pods:               50
...

4.4.4 Running The DaemonSet Only On Specific Nodes

The AMD GPU device plugin, unlike the NVIDIA GPU device plugin Daemonset, is scheduled to run on each Kubernetes host. This means that it runs even if the host has no GPU.

This can be prevented with the following steps:

A LabelSet can be created via cmsh, and the nodes or categories that have GPUs are assigned within the labelsets mode:

Example

[root@basecm10 ~]# cmsh
[basecm10]% kubernetes
[basecm10->kubernetes[default]]% labelsets
[basecm10->kubernetes[default]->labelsets]% use nvidia
[basecm10->kubernetes[default]->labelsets[nvidia]]% ..   #but, we're using AMD GPUs, so let's go back up:
[basecm10->kubernetes[default]->labelsets]% add amd
[basecm10->kubernetes*[default*]->labelsets*[amd*]]% set labels nvidia.com/amd-gpu-accelerator=
[basecm10->kubernetes*[default*]->labelsets*[amd*]]% append categories gpu-nodes
[basecm10->kubernetes*[default*]->labelsets*[amd*]]% commit

This assigns the labels to the nodes with GPUs. This can be verified with:

Example

kubectl get nodes -l nvidia.com/amd-gpu-accelerator=
NAME     STATUS   ROLES    AGE   VERSION
gpu001   Ready    master   66m   v1.18.8
gpu002   Ready    master   66m   v1.18.8
...

The DaemonSet YAML can now be adjusted to only run the device plugin on nodes with this new label. This can be done by adding an affinity block after the tolerations block:

Example

      tolerations:
      - key: CriticalAddonsOnly  # toleration may be different, if changes were made to it
        operator: Exists
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: 'nvidia.com/amd-gpu-accelerator'
                    operator: Exists

This results in the device plugin pods being removed immediately from all nodes that do not have the label.

4.4.5 Running An Example Workload

An example workload can be run as described in the official AMD GPU Kubernetes device plugin documentation at:

https://github.com/RadeonOpenCompute/k8s-device-plugin/tree/v1.16#example-workload

Thus it should now be possible to run:

[root@basecm10 ~]# kubectl create -f https://raw.githubusercontent.com/RadeonOpenCompute/
k8s-device-plugin/v1.16/example/pod/alexnet-gpu.yaml

The YAML requests only one GPU at the bottom of the YAML file:

apiVersion: v1
kind: Pod
metadata:
  name: alexnet-tf-gpu-pod
  labels:
    purpose: demo-tf-amdgpu
spec:
  containers:
    - name: alexnet-tf-gpu-container
      image: rocm/tensorflow:latest
      workingDir: /root
      env:
      - name: HIP_VISIBLE_DEVICES
        value: "0" # # 0,1,2,...,n for running on GPU and select the GPUs, -1 for running on CPU
      command: ["/bin/bash", "-c", "--"]
    args: ["python3 benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py --model=alexnet;\
trap : TERM INT; sleep infinity & wait"]
      resources:
        limits:
          amd.com/gpu: 1 # requesting a GPU

Container creation might take a while due to the image size. Once scheduled, it prints out that it found exactly one GPU, and proceeds to run a TensorFlow workload.

Example

[root@basecm10 ~]# kubectl logs -f alexnet-tf-gpu-pod
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/compat/v2_compat.py:96:
disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed
in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
2021-01-08 21:03:29.222293: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104]
    CPU Frequency: 2495445000 Hz
2021-01-08 21:03:29.222398: I tensorflow/compiler/xla/service/service.cc:168]
    XLA service 0x39f62f0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-01-08 21:03:29.222420: I tensorflow/compiler/xla/service/service.cc:176]
          StreamExecutor device (0): Host, Default Version
2021-01-08 21:03:29.223754: I tensorflow/stream_executor/platform/default/dso_loader.cc:48]
    Successfully opened dynamic library libamdhip64.so
2021-01-08 21:03:31.635339: I tensorflow/compiler/xla/service/service.cc:168]
    XLA service 0x3a40bb0 initialized for platform ROCM (this does not guarantee that XLA will be used). Devices:
2021-01-08 21:03:31.635363: I tensorflow/compiler/xla/service/service.cc:176]
          StreamExecutor device (0): Device 738c, AMDGPU ISA version: gfx908
2021-01-08 21:03:31.931125: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734]
    Found device 0 with properties:
          pciBusID: 0000:27:00.0 name: Device 738c     ROCm AMD GPU ISA: gfx908
          coreClock: 1.502GHz coreCount: 120 deviceMemorySize: 31.98GiB deviceMemoryBandwidth: 1.12TiB/s
...
TensorFlow:  2.3
Model:       alexnet
Dataset:     imagenet (synthetic)
Mode:        training
SingleSess:  False
Batch size:  512 global
             512 per device
Num batches: 100
Num epochs:  0.04
Devices:     ['/gpu:0']
...

Had more GPUs been requested, more would have been made available to the container.

For comparison, a CPU version of the container is also available. The official instructions can be referred to for these, too.

4.5 Kubernetes Configuration Overlays

A list of configuration overlays can be seen from within configurationoverlay mode:

Example

[basecm10->configurationoverlay]% list
Name (key)          Priority Nodes             Categories Roles
------------------- -------- ----------------- ---------- -------------------------...
kube-default-etcd   500      node001..node003             Etcd::Host
kube-default-master 510      node001..node003             generic::containerd, kube...
kube-default-worker 500      node004..node006  default    generic::containerd, kube...
kube-default-netq   501      node001                      generic::netq

The NetQ configuration overlay is only normally displayed if NetQ is deployed.

Configuration overlays can be used to manage the Kubernetes services used with a particular configuration. For example, when managing the Kubernetes services used for a Kubernetes engine within an Auto Scale tracker (section 8.4.9 of the Administrator Manual).

4.6 Removing A Kubernetes Cluster

A Kubernetes cluster can be removed using cm-kubernetes-setup with the --remove and --yes-i-really-mean-it options. Also, if there more than one cluster present, then the cluster name must be specified using the --cluster parameter.

A removal run looks as follows (some output ellipsized):

Example

[root@basecm10 ~]# cm-kubernetes-setup --remove --cluster default --yes-i-really-mean-it
Connecting to CMDaemon
Executing 20 stages
################### Starting execution for 'Kubernetes Setup'
  - kubernetes
  - docker
## Progress: 0
#### stage: kubernetes: Get Kube Cluster
## Progress: 5
#### stage: kubernetes: Check Kube Cluster Exists
## Progress: 10
#### stage: kubernetes: Find Installed Components
## Progress: 15
#### stage: kubernetes: Find Files On Headnodes
## Progress: 20
#### stage: kubernetes: Firewall Zone Close
## Progress: 25
#### stage: kubernetes: Firewall Interface Close
## Progress: 30
#### stage: kubernetes: Firewall Policy Close
## Progress: 35
#### stage: kubernetes: Nginx Reverse Proxy Close
## Progress: 40
#### stage: kubernetes: IP Ports Close
## Progress: 60
#### stage: kubernetes: Remove Installed Components
## Progress: 65
#### stage: kubernetes: Remove Files On Headnodes
## Progress: 70
#### stage: kubernetes: Remove Etcd Spool
## Progress: 80
#### stage: kubernetes: Set Reboot Required
You need to reboot 2 nodes to cleanup the network configuration
## Progress: 85
#### stage: kubernetes: Collection Update Provisioners
## Progress: 100
Took:     00:08 min.
Progress: 100/100
################### Finished execution for 'Kubernetes Setup', status: completed
Kubernetes Setup finished!

Using the --remove option removes the Kubernetes cluster configuration from BCM, unassigns Kubernetes-related roles—including the EtcdHost role—and removes Kubernetes health checks. The command does not remove packages that were installed with a cm-kubernetes-setup command before that.

After the disabling procedure has finished, the cluster has no Kubernetes configured and running.

4.7 Kubernetes Cluster Configuration Options

Kubernetes allows many Kubernetes clusters to be configured. These are separated sets of hosts with different certificates, users and other global settings.

When carrying out the Kubernetes setup run, a Kubernetes cluster name is asked, and a new object with the cluster settings is then added into the CMDaemon configuration. The administrator can change the settings of the cluster from within the kubernetes mode of cmsh, or within the Kubernetes Clusters options window of Base View accessible via the navigation path Containers > Kubernetes Clusters.

The cmsh equivalent looks like:

Example

[root@basecm10 ~]# cmsh
[basecm10]% kubernetes list
Name (key)
------------------
default
[basecm10]% kubernetes use default
[basecm10->kubernetes[default]]% show
Parameter                            Value
------------------------------------ ------------------------------------------------------
Name                                 default
Revision
Etcd Cluster                         kube-default
Pod Network                          kube-default-pod
Pod Network Node Mask
Internal Network                     internalnet
KubeDNS IP                           10.150.255.254
Kubernetes API server
Kubernetes API server proxy port     10443
App Groups                           <1 in submode>
Label Sets                           <3 in submode>
Notes
Version                              1.27.10-150500.1.1
Trusted domains                      basecm10.openstacklocal,master,localhost,10.141.255.254
Module file template                 <1.01KiB>
Kubeadm init file                    <1.19KiB>
Service Network                      kube-default-service
Kubeadm CERT Key                     *********
Kube CA Cert                         *********
Kube CA Key                          *********
Kubernetes users                     <0 in submode>
External                             no
External Kubernetes Ingress server
External port                        0
Capi template                        no
Capi namespace                       default
Kubernetes management cluster
[basecm10->kubernetes[default]]%

The preceding kubernetes mode parameters are described in table 4.1:

Table 4.1: kubernetes mode parameters

4.8 EtcdCluster

The EtcdCluster mode sets the global etcd cluster settings. It can be accessed via the top level etcd mode of cmsh.

* Boolean (takes yes or no as a value)

Table 4.2: EtcdCluster role parameters and etcd options

4.9 Kubernetes Roles

Kubernetes roles include the following roles:

• EtcdHost (here)

• KubernetesApiServerProxy (here)

• Kubelet (here)

• generic::containerd (here)

• generic::netq (here)

When nodes are configured using Kubernetes roles, then settings in these roles may sometimes use the same values (pointer variables).

Example

[basecm10->configurationoverlay[kube-default-etcd]->roles[Etcd::Host]]% get etcdcluster
kube-default

and

[basecm10->kubernetes[default]]% get etcdcluster
kube-default

Pointer variables such as these have definitions that are shared across the roles, as indicated by the parameter description tables for the roles, and which are described in the following pages.

In cmsh, the roles can be assigned:

• for individual nodes via the roles submode of device mode

• for a category via the roles submode of a category

• for a configuration overlay via the roles submode of configurationoverlay mode

4.9.1 EtcdHost Role

The EtcdHost role is used to configure and manage the etcd service for a node.

The etcd service manages the etcd database, which is a hierarchical distributed key-value database. The database is used by Kubernetes to store its configurations. The EtcdHost role parameters are described in table 4.3:

* Boolean (takes yes or no as a value)

Table 4.3: EtcdHost role parameters and etcd options

The etcd settings are updated by BCM in /cm/local/apps/etcd/current/etc/cm-etcd.conf.

4.9.2 The KubernetesAPIServerProxy Role

The KubernetesApiServerProxy role sets up a proxy that provides the entry point for one or more instances of the Kubernetes API server. The proxy runs on every node of a Kubernetes cluster instance, including the head node.

If multiple Kubernetes master nodes are present, then it enables HA for the Kubernetes master components (section 4.1).

This means that local port 10443 on all these nodes load balances over all control-plane node API servers.

Port 10443 can also be opened up for external access. The cm-kubernetes-setup wizard (section 4.2) in one of its screens (figure 4.5) prompts for this.

Figure 4.5: Wizard question

If the prompt is answered with no, then the firewall role on the head node can still be customized with cmsh later on:

Example

root@basecm10:~# cmsh
[basecm10]% device use master
[basecm10->device[basecm10]]% roles
[basecm10->device[basecm10]->roles]% use firewall
[basecm10->device[basecm10]->roles[firewall]]% openports
[basecm10->device[basecm10]->roles[firewall]->openports]% list
Index  Action    Network      Port         Destination  Protocol  Address      Description
------ --------- ------------ ------------ ------------ --------- ------------ ------------------------
0      ACCEPT    net          30080        fw           TCP       0.0.0.0/0    Kubernetes 'default' In+
1      ACCEPT    net          30443        fw           TCP       0.0.0.0/0    Kubernetes 'default' In+
[basecm10->device[basecm10]->roles[firewall]->openports]% add ACCEPT net 10443 tcp fw
ok.
[basecm10->device*[basecm10*]->roles*[firewall*]->openports[2]]% commit
[basecm10->device[basecm10]->roles[firewall]->openports[2]]% list
Index  Action    Network      Port         Destination  Protocol  Address      Description
------ --------- ------------ ------------ ------------ --------- ------------ ------------------------
0      ACCEPT    net          30080        fw           TCP       0.0.0.0/0    Kubernetes 'default' In+
1      ACCEPT    net          30443        fw           TCP       0.0.0.0/0    Kubernetes 'default' In+
2      ACCEPT    net          10443        fw           TCP       0.0.0.0/0
[basecm10->device[basecm10]->roles[firewall]->openports[2]]%
Thu Oct 31 15:58:56 2024 [notice] basecm10: Service shorewall was restarted

4.9.2.1 Ingress Server Proxying Using The Default HTTPS Port 443

In BCM 11 a new KubernetesIngressServerProxy role is introduced to allow Ingress Server Proxying.

In BCM 10, to maintain backward compatibility, new roles cannot be introduced. The proxying functionality of the Ingress server is therefore partly dealt with using the KubernetesApiServerProxy role.

The proxy provides the entry point for one or more instances of the Kubernetes Ingress server, and runs on every node of a Kubernetes cluster instance, including the head node. If multiple Kubernetes master nodes are present, then the role enables HA for the Kubernetes master components (section 4.1).

Before BCM 10.24.11, the Kubernetes Ingress Server bundled NGINX Ingress server as an option, and used non-standard HTTPS ports. The defaults have always been 30080 for HTTP, and 30443 for HTTPS as NodePort services (https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport). These ports are customizable, but a limitation of NodePort services is that they may only use the port range 30000–-32767.

Starting with BCM 10.24.11, during the cm-kubernetes-setup session to install Kubernetes, there is a screen which asks if the Ingress Server should be available on the default HTTPS port 443 (figure 4.6).

It is recommended to respond with yes, because nowadays services running inside Kubernetes are commonly run on the default HTTPS port. This in turn results in user redirects from the non-standard port (for example, 30443) to port 443. Using the default port is also more in line with how Ingress is offered by cloud-based solutions that provide Kubernetes.

Figure 4.6: Wizard question

Setting 443 for the proxy ensures that local port 443 on all the concerned nodes carries out loadbalancing over all the control-plane node Kubernetes Ingress Servers. It also ensures that the port is opened in the firewall.

Setting no means that an open port can still be configured later on by carrying the 4 steps that follow shortly. The discerning cluster administrator should be able to see from these steps how the Ingress server proxying functionality is implemented without the additional role.

The extra parameters are set per KubeCluster entity inside BCM. This is because Ingress ports can differ between multiple installations of Kubernetes.

The steps:

1. Ensuring HTTPS is not already in use: If port 443 it is already in use by BCM (for example, if the BCM landing page is served on port 443 via apache2 or httpd), then that port must be changed or disabled.

   For Ubuntu-based systems, this means commenting out Listen 443 in /etc/apache2/ports.conf. For RHEL-based systems, that line is located in /etc/httpd/conf.d/ssl.conf.

   The web service should then be restarted (apache2 for Ubuntu, httpd for RHEL and Rocky).

2. Assigning the configuration to the KubeCluster entities:

   This can be done as follows in cmsh:

   Example

   This is the KubeCluster entity that the parameters are added to:

   root@basecm10:~# cmsh
   [basecm10]% kubernetes
   [basecm10->kubernetes[default]]% show
   Parameter                            Value
   ------------------------------------ --------------------------------------------------
   Name                                 default
   Revision
   Etcd Cluster                         kube-default
   Pod Network                          kube-default-pod
   Pod Network Node Mask
   Internal Network                     internalnet
   KubeDNS IP                           10.150.255.254
   ...

   The following parameters are set next:

   • ingress_proxy_backend_port: a fallback port in case the NGINX NodePort cannot be read from the applications

   • ingress_proxy_listen_port: the port the proxy should listen on (default HTTPS port is 443)

   • ingress_proxy_enable: a flag to enable the proxying functionality

   In cmsh this can be carried out with special -e|--extra options as follows, which set up new parameters in the kubernetes instance:

   Example

   [basecm10->kubernetes[default]]% set -e ingress_proxy_backend_port 30443
   [basecm10->kubernetes*[default*]]% set -e ingress_proxy_listen_port 443
   [basecm10->kubernetes*[default*]]% set -e ingress_proxy_enable true
   [basecm10->kubernetes*[default*]]% commit
   [basecm10->kubernetes[default]]% show
   Parameter                            Value
   ------------------------------------ -------------------------------------------------------------------
   ingress_proxy_backend_port           30443
   ingress_proxy_enable                 true
   ingress_proxy_listen_port            443
   Name                                 default
   Revision
   Etcd Cluster                         kube-default
   Pod Network                          kube-default-pod
   Pod Network Node Mask
   Internal Network                     internalnet
   KubeDNS IP                           10.150.255.254
   ...

   In BCM 10, back in the KubernetesApiServerProxy role, a commit then triggers the NGINX configuration reload:

   [basecm10->kubernetes[default]]% device use master
   [basecm10->device[basecm10]]% roles
   [basecm10->device[basecm10]->roles]% use kubernetes::apiserverproxy
   [basecm10->device[basecm10]->roles[Kubernetes::ApiServerProxy]]% commit
   [basecm10->device[basecm10]->roles[Kubernetes::ApiServerProxy]]%
   Fri Nov 22 14:49:15 2024 [notice] basecm10: Service nginx was reloaded

   As the preceding session implies, the /etc/nginx/nginx.conf file and nginx service are affected.

   Local port 443 should now load balance the Kubernetes Ingress service to all control-plane nodes.

3. Opening the HTTPS port in the firewall:

   The port in the default firewall, Shorewall (section 7.2 of the Installation Manual), can be modified using the firewall role in cmsh.

   For historic reasons, ports 30080 and 30443 are still opened up by default in BCM 10. These ports can also be removed from the entries for the Ingress controller, as shown in the following session:

   root@basecm10:~# cmsh
   [basecm10]% device use master
   [basecm10->device[basecm10]]% roles
   [basecm10->device[basecm10]->roles]% use firewall
   [basecm10->device[basecm10]->roles[firewall]]% openports
   [basecm10->device[basecm10]->roles[firewall]->openports]% list -v
   Index  Action    Network  Port     Destination  Protocol  Address      Description
   ------ --------- -------- -------- ------------ --------- ------------ ------------------------
   0      ACCEPT    net      30080    fw           TCP       0.0.0.0/0    Kubernetes 'default' Ingress HTTP
   1      ACCEPT    net      30443    fw           TCP       0.0.0.0/0    Kubernetes 'default' Ingress HTTPS
   2      ACCEPT    net      10443    fw           TCP       0.0.0.0/0
   [basecm10->device[basecm10]->roles[firewall]->openports]% remove 0
   [basecm10->device*[basecm10*]->roles*[firewall*]->openports*]% remove 1
   [basecm10->device*[basecm10*]->roles*[firewall*]->openports*]% add ACCEPT net 443 tcp fw
   ok.
   [basecm10->device*[basecm10*]->roles*[firewall*]->openports*[3]]% list
   Index  Action    Network  Port     Destination  Protocol  Address      Description
   ------ --------- -------- -------- ------------ --------- ------------ ------------
   2      ACCEPT    net      10443    fw           TCP       0.0.0.0/0
   3      ACCEPT    net      443      fw           TCP       0.0.0.0/0
   [basecm10->device*[basecm10*]->roles*[firewall*]->openports*[3]]% ..
   [basecm10->device*[basecm10*]->roles*[firewall*]->openports*]% commit
   [basecm10->device[basecm10]->roles[firewall]->openports]%
   Thu Oct 31 16:35:19 2024 [notice] basecm10: Service shorewall was restarted
   [basecm10->device[basecm10]->roles[firewall]->openports]% list
   Index  Action    Network  Port     Destination  Protocol  Address      Description
   ------ --------- -------- -------- ------------ --------- ------------ ------------
   0      ACCEPT    net      10443    fw           TCP       0.0.0.0/0
   1      ACCEPT    net      443      fw           TCP       0.0.0.0/0

   As seen in the preceding session, this immediately affects the /etc/shorewall/rules file and shorewall service.

4. Fixing the BCM landing page: Finally the landig page YAML file is applied. The file should be available on the head nodes for BCM version 10.24.11 and beyond.

   root@basecm10# kubectl apply -f \
   /cm/local/apps/cm-setup/lib/python3.12/site-packages/cmsetup/plugins/kubernetes/addons/1.27/landingpage.yaml
   configmap/nginx-proxy-config created
   deployment.apps/nginx-proxy created
   service/nginx-proxy-service created
   Warning: annotation "kubernetes.io/ingress.class" is deprecated, please use 'spec.ingressClassName' instead
   ingress.networking.k8s.io/default-ingress created

   In the preceding session, a simple Deployment (https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) is deployed. The Deployment manages a single pod that runs an NGINX container with the configuration provided as a ConfigMap (https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/). This configuration sets a redirect to the single http://master backend, where the BCM landing page (section 2.4.1 of the Administrator Manual) is running, and running with the HTTP rather than the HTTPS protocol.

   The service exposes the pod, and an Ingress rule is created and used as a fallback/default handler for the Kubernetes NGINX Ingress server. This way, if no other Ingress rule matches, then the BCM landing page is served via the HTTPS port. In other words, the page is served when there is no other matching domain name or other path-based Ingress rule to take priority over the landing page. This means that the landing page is still served when the head node IP address is entered directly in the browser.

4.9.3 The Kubelet Role

The Kubelet role is used to configure and manage the kubelet service. BCM takes care of joining these nodes with kubeadm join when needed.

Control Plane The Kubelet role has a parameter Control plane that is set to the value yes for control plane nodes. In that case it also runs Kubernetes control plane services, such as:

• Kubernetes API server (kube-apiserver),

• Kubernetes scheduler (kube-scheduler),

• Kubernetes controller manager (kube-controller-manager),

• Kubernetes network proxy (kube-proxy),

• CoreDNS (coredns).

Workers The Kubelet role has a parameter worker that is set to yes for worker nodes. In that case, the control plane pods will not be running on the node.

The Kubelet role parameters are described in table 4.4:

* Boolean (takes yes or no as a value)

Table 4.4: Kubelet role parameters and kubelet options

Further details on the Kubelet service can be found at https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/.

4.9.3.1 The options Submode

[basecm10->configurationoverlay[kube-default-master]->roles[kubelet]]% help options
Name:
options - Manage kubelet container command options
Usage:
options [kubelet]
options [kubelet] <filename>
options [kubelet] <filename> set [name] key=value
options [kubelet] <filename> set [name] key
options [kubelet] <filename> clear [name] key
Examples:
options kube-apiserver.yaml set advertise-address=192.168.200.148
[basecm10->configurationoverlay[kube-default-master]->roles[kubelet]]% options
Filename             Container       Key                   Value
-------------------  --------------- --------------------- ------------------------------------
kube-apiserver.yaml  kube-apiserver  cors-allowed-origins  https://ui.example.com
kube-apiserver.yaml  kube-apiserver  oidc-client-id        kubernetes
kube-apiserver.yaml  kube-apiserver  oidc-issuer-url       https://sso.example.com/realms/bcm
kube-apiserver.yaml  kube-apiserver  oidc-username-prefix  -

The full path to the files should be specified for the options command. For example, kube-apiserver.yaml may be specified as /etc/kubernetes/manifests/kube-apiserver.yaml.

4.9.4 Containerd Role

The Containerd role is used to configure and manage the containerd daemon. This is done through a generic role in BCM. Generic roles under Kubernetes are found under the cmsh path indicated by: cmsh > configurationoverlay[kube-object] > roles[generic::role]

Example

[basecm10->configurationoverlay[kube-default-master]->roles[generic::containerd]]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Name                             generic::containerd
Revision
Type                             GenericRole
Add services                     yes
Services                         containerd
Configurations                   <2 in submode>
Environments                     <1 in submode>
Exclude list snippets            <2 in submode>
Data node                        no

The role has several submodes:

• The configurations submode: This contains various configuration drop-ins.

  Configuration drop-ins that BCM may manage in the generic::containerd role are:

  • NVIDIA Container Toolkit configuration.

  • Docker registry certs directory configuration.

  • Docker Hub credentials configuration.

  • Harbor Registry Mirror configuration.

  • CRI cgroup configuration.

  • CNI bin dir configuration.

  • CRI registry configuration.

  • CRI custom sandbox image configuration.

  The drop-ins that are actually available depend on choices made by the cluster administrator during deployment.

• The environment submode: This has optional environment variables that are made available to the configuration files and that are to be used as templates.

• The excludelistsnippets submode: This has files related to containerd that need to be excluded, such as the most obvious directory /var/lib/containerd.

4.10 Security Model

The Kubernetes security model allows authentication using a certificate authority (CA), with the user and daemon certificates signed by a Kubernetes CA. The Kubernetes CA should not be confused with the BCM CA.

BCM lets kubeadm create a CA specifically for issuing all Kubernetes-related certificates. The certificates are put into /etc/kubernetes/pki/<kube cluster label>/ by default.

In Kubernetes terminology a user is a unique identity accessing the Kubernetes API server. The user may be a human or an automated process. For example an admin or a developer are human users, but kubelet represents an infrastructure user. Both types of users are authorized and authenticated in the same way against the API server.

Kubernetes uses client certificates, tokens, or HTTP basic authentication methods to authenticate users for API calls. BCM configures client certificate usage by default. The authentication is performed by the API server which validates the user certificate using the common name part of the certificate subject.