Skip to navigation

Kubernetes Cluster API

View as Markdown

The Kubernetes Cluster API (CAPI), as explained in the introduction to the online Cluster API Book at https://cluster-api.sigs.k8s.io/, “is a Kubernetes sub-project focused on providing declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters”. The Cluster API Book is the official Kubernetes project documentation for CAPI.

This chapter describes the installation and usage of the NVIDIA Base Command Manager CAPI extension called BCM Kubernetes CAPI Infrastructure Provider.

8.1 Kubernetes Cluster API Components

An overview of the CAPI components is shown in figure 8.1. Further details about the components are given in the sections of this chapter that follow.

CAPI components

Figure 8.1: CAPI components

Figure 8.1 shows a standard Kubernetes cluster deployed on an active head node. The cluster has been modified to become a Kubernetes management cluster using the BCM CAPI Infrastructure Provider, and has successfully deployed two additional Kubernetes clusters through CAPI. These additional clusters may be running different versions of Kubernetes.

The system administrator receives module files and kubeconfig files for all three Kubernetes clusters.

8.1.1 Kubernetes Management Cluster

In BCM documentation, the term Kubernetes management cluster is used specifically and precisely to refer to a Kubernetes cluster that operates as the management cluster for CAPI.

The ability to modify an external Kubernetes cluster to operate as a Kubernetes management cluster is under preparation at the time of writing (June 2023) of this section. This includes scenarios such as using an existing Kubernetes cluster, hosted on a public cloud, to serve as the Kubernetes management cluster for a Kubernetes cluster deployed through BCM. At present, any Kubernetes cluster deployed through BCM (Chapter 4) can be modified into a CAPI management cluster.

8.1.2 Kubernetes CAPI Cluster

BCM documentation uses the term Kubernetes CAPI cluster to refer specifically to a Kubernetes cluster deployed via the Cluster API. Upstream Kubernetes documentation also sometimes refers to a Kubernetes cluster deployed via CAPI as a workload cluster.

8.1.3 BCM CAPI Infrastructure Provider

The BCM CAPI Infrastructure Provider is a derivative of the Bring Your Own Host (BYOH) CAPI provider. The BYOH CAPI provider can also be referred to as the BYOH CAPI Infrastructure Provider, and is available as a GitHub project at:
https://github.com/vmware-tanzu/cluster-api-provider-bringyourownhost

Similar to other CAPI providers, the BCM CAPI Infrastructure Provider is an extension of CAPI itself, and both the CAPI provider and CAPI are required on the Kubernetes management cluster.

8.1.3.1 The BCM CAPI Operator

The BCM CAPI operator is an operator deployed in the byoh-system namespace on the Kubernetes management cluster. The operator serves as the central point of connection for BCM CAPI host agents (section 8.1.3.2). It also monitors the state of clusters and machines for the Kubernetes CAPI clusters.

root@headnode:~# kubectl get pod -n byoh-system
NAME                                       READY   STATUS    RESTARTS        AGE
byoh-controller-manager-6c98cbf44c-7gdbx   2/2     Running   2 (5h55m ago)   23h

8.1.3.2 BCM CAPI Host Agents

BCM CAPI host agents are scheduled to run on selected hosts. These hosts are defined by the cluster administrator when the cm-kubernetes-capi-setup wizard (section 8.2.2) is run. The wizard ensures that a selected host gets the right packages in its software image, and that the host is assigned the right BCM roles.

For CAPI calls to function, the CAPI role is assigned. Assignment can be done either before or after creating a Kubernetes CAPI cluster via the Kubernetes management cluster. The role sets up a capi-agent service on the host for the associated Kubernetes management cluster.

When the role is:

  • assigned: BCM initiates and bootstraps the service

  • unassigned: BCM halts the service and makes it unavailable for use by CAPI on the associated Kubernetes management cluster

This differs from cloud-based CAPI providers, which do not require managing a limited number of pre-existing nodes, but instead start up new nodes on demand as needed.

If a Kubernetes CAPI cluster receives a cluster request and there are no hosts available, then machines remain in a Pending state until hosts become available. Once hosts become available, the status transitions to Provisioning, and then eventually to Running (section 8.3.1).

8.1.3.3 BCM CAPI Vs BYOH

Once the actual Kubernetes CAPI clusters are deployed on designated CAPI hosts, BCM uses its Python API to carry out the following operations:

  • The requested Kubernetes version for each node is stored in the BCM database.

  • The requested Kubernetes version is made available in the corresponding software image for the node.

  • The node is provisioned with the software image.

  • The installer is notified that the Kubernetes installation can proceed.

In contrast to the BYOH provider, BCM CAPI has the following extra features:

  • Linux distributions are not limited to Ubuntu 20.04.

  • Kubernetes versions can be other than only those for which parcels have been created, as the BCM Kubernetes integration does not rely on parcels.

8.2 The Kubernetes CAPI Wizard

The cm-kubernetes-capi-setup wizard installs the BCM CAPI Infrastructure Provider, and assigns the CAPI role to nodes.

CAPI setup wizard

Figure 8.2: CAPI setup wizard

8.2.1 The Install CAPI Option

The Install CAPI option of figure 8.2 leads to a screen that prompts the user to select the Kubernetes cluster instance on which to install the BCM CAPI Infrastructure Provider.

Selection of Kubernetes cluster for installation of the BCM CAPI Infrastructure Provider

Figure 8.3: Selection of Kubernetes cluster for installation of the BCM CAPI Infrastructure Provider

If needed:

  • cert-manager and the Cluster API itself are installed

  • the CAPI clusterctl tool is run in the back end to carry out initialization

8.2.1.1 The Install CAPI Option Actions

Execution of the Install CAPI operation:

  • prepares a Kubernetes cluster template for CAPI (section 8.8).

  • installs the BCM CAPI Infrastructure Provider on the Kubernetes cluster with the command: clusterctl init infrastructure byoh.

  • waits for the BCM CAPI operator to be ready for operation.

  • patches the BCM CAPI operator to use the appropriate image. Patching is required until the BCM changes get into the upstream repository.

  • makes the capi module available for loading

8.2.1.2 The Install CAPI Option Changes To Kubernetes Management Cluster

After the Install CAPI option has run, the operators installed and running on the Kubernetes management cluster can be seen (some output truncated):

Example

[root@basecm11 ~]# kubectl get pod -A | grep -E "byoh-system|capi|cert-manager"
byoh-system                         byoh-controller-manager-ff7f68bb4-vprrz                          2/2 ...
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-7945579f8c-4ntb2       1/1 ...
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-66cdfb477b-h5779   1/1 ...
capi-system                         capi-controller-manager-64cb86f545-8k9zt                         1/1 ...
cert-manager                        cert-manager-5d4c5bc8bc-j5hbp                                    1/1 ...
cert-manager                        cert-manager-cainjector-79bc559d9d-qfvph                         1/1 ...
cert-manager                        cert-manager-webhook-5cf45f5b6-9gwlx                             1/1 ...

8.2.1.3 The Install CAPI Changes To BCM

A new Kubernetes cluster template is generated in BCM on running Install CAPI. The template naming convention used follows the form:

capi- <mgmt_cluster_name>-template

In the present example, where the name of the Kubernetes management cluster is default, the resulting template would be called capi-default-template. The name default is set by default when creating a Kubernetes instance (figure 4.2).

[root@basecm11 ~]# cmsh
[basecm11]% kubernetes
[basecm11->kubernetes]% list
Name (key)
----------------------
capi-default-template
default
[basecm11->kubernetes]% show capi-default-template
Parameter                            Value
------------------------------------ ----------------------------------------------------------------------
Name                                 capi-default-template
Revision
Etcd Cluster
Pod Network
Pod Network Node Mask
Internal Network
KubeDNS IP                           0.0.0.0
Kubernetes API server
Kubernetes API server proxy port     6444
App Groups                           <0 in submode>
Label Sets                           <0 in submode>
Notes
Version
Trusted domains                      kubernetes,kubernetes.default,kubernetes.default.svc,master,localhost
Module file template                 <690B>
Kubeadm init file                    <0B>
Service Network
Kubeadm CERT Key                     < not set >
Kube CA Cert                         < not set >
Kube CA Key                          < not set >
Kubernetes users                     <0 in submode>
External                             no
External Kubernetes Ingress server
External port                        0
Capi template                        yes
Capi namespace                       default
Kubernetes management cluster        default

This becomes the default template used for generating Kubernetes CAPI clusters associated with this specific Kubernetes management cluster. Users can modify this template, and can even create additional templates as needed. The YAML configuration for a CAPI cluster also lets users specify a template, via annotations within the YAML file (section 8.8).

8.2.2 The Assign CAPI Role Option

After the Install CAPI operation has completed its run, the Assign CAPI Role operation can be carried out from the setup screen of figure 8.2. The Assign CAPI Role operation carries out a CAPI role assignment to a group of nodes by creating a configuration overlay (section 4.5). The role can then be assigned either to a category, or to individual nodes.

The software images on the chosen nodes get the packages needed to run the CAPI agents.

When the Assign CAPI Role operation is carried out, the screens in figures 8.4-8.7 may be displayed:

Select a Kubernetes management cluster

Figure 8.4: Select a Kubernetes management cluster

Customize the configuration overlay

Figure 8.5: Customize the configuration overlay

Choose nodes via categories

Figure 8.6: Choose nodes via categories

Choose nodes directly

Figure 8.7: Choose nodes directly

Some of the screens may be skipped if they are not needed. For example, the screen for selecting individual nodes (figure 8.7) is not displayed if all the nodes have already been assigned during selection via categories.

8.2.2.1 The Assign CAPI Role Actions

During execution of the Assign CAPI Role operation, the wizard:

  • prepares the Kubeadm and Helm repositories in the software images for the nodes.

  • writes the IP Forwarding configuration to the software images for the nodes.

  • installs necessary packages in the software images for the nodes (for example: cm-capi).

  • provisions the nodes from their software images.

  • creates a configuration overlay in BCM.

  • assigns the selected categories and nodes to the overlay.

  • assigns the capi role and containerd roles to the overlay.

8.2.2.2 The Assign CAPI Role Changes To BCM

The configuration overlay created during the execution of the Assign CAPI Role operation is now visible in the configurationoverlay mode (some output elided):

Example

[root@basecm11 ~]# cmsh
[basecm11]% configurationoverlay
[basecm11->configurationoverlay]% list
Name (key)           Priority   All head nodes Nodes            Categories       Roles
-------------------- ---------- -------------- ---------------- ---------------- -----------------
capi-default         500        no             node001,node002                   capi
kube-default-etcd    500        no             node003                           Etcd::Host
...
[basecm11->configurationoverlay]% show capi-default
Parameter                        Value
-------------------------------- ------------------------------------------------
Name                             capi-default
Revision
All head nodes                   no
Priority                         500
Nodes                            node001,node002
Categories
Roles                            capi
Customizations                   <0 in submode>

The preceding overlay shows that the capi role has been assigned to nodes node001 and node002.

The capi role sets the capi-agent service for a node. If it is set, then BCM bootstraps the agent (section 8.4) and initiates the capi-agent systemd service that runs on the node.

The containerd role takes care of the containerd service, so that containerd is also started on the nodes.

The containerd service is needed after the agent initiates the provisioning of its node to integrate it into a Kubernetes CAPI cluster. This is because a kubelet service is eventually started, which relies on containerd.

CAPI uses kubeadm for node provisioning and management, which means that containerd must be running in advance. Therefore, as part of its pre-launch checks, kubeadm verifies that the containerd service is operational.

8.2.2.3 The Assign CAPI Role Changes To the Kubernetes management cluster

The BCM CAPI host agents should register with the Kubernetes management cluster, and can then be seen on running kubectl with the get byohost option:

Example

root@basecm11:~# kubectl get byohost -A
NAMESPACE   NAME      OSNAME   OSIMAGE              ARCH
default     node001   linux    Ubuntu 22.04.4 LTS   amd64
default     node002   linux    Ubuntu 22.04.4 LTS   amd64

Registration does not necessarily make these hosts a part of any Kubernetes CAPI cluster. However, if there are clusters that have been created beforehand, and there are machines that are awaiting additional resources, then it might be that some or all of these hosts are immediately provisioned. A simple method to check this is by querying the machine resource.

[root@basecm11 ~]# kubectl get machine -A
No resources found

If however there were machines that were pending, the output might look like:

[root@basecm11 ~]# kubectl get machines
NAME                                       CLUSTER        NODENAME   PROVIDERID   PHASE          AGE   VERSION
byoh-cluster-control-plane-h2d64           byoh-cluster                           Provisioning   4s    v1.26.1
byoh-cluster-md-0-56985bf9d6xkhj68-dxvkc   byoh-cluster                           Pending        6s    v1.26.1

When the machines are provisioned, the Kubernetes CAPI cluster output state might look like:

[root@basecm11 ~]# kubectl get ma       #ma|machine|machines are synonymous. Also byom|byomachine|byomachines
NAME                                       CLUSTER        NODENAME   PROVIDERID              PHASE     AGE ...
byoh-cluster-control-plane-h2d64           byoh-cluster   node001    byoh://node001/e7fq1b   Running   3h3m...
byoh-cluster-md-0-56985bf9d6xkhj68-dxvkc   byoh-cluster   node002    byoh://node002/ngkju0   Running   3h3m...

This is discussed further in section 8.3.

8.3 Deploying A Kubernetes Cluster Through CAPI

The BCM CAPI Infrastructure Provider configures the Kubernetes management cluster. A Kubernetes cluster can then be created through CAPI, and deployed on the available CAPI nodes.

If resources are not available, then the cluster cannot assign nodes to the newly-created cluster definition for provisioning. This results in a persistent Pending state for various resources.

The capi module makes the command line tool clusterctl available, which can be used to generate a cluster manifest. In the following example, the tool is used to define a cluster with Kubernetes version 1.34.2, comprising one control plane node and one worker node.

Example

[root@headnode ~]# module load capi/1.10.2
[root@headnode ~]# CONTROL_PLANE_ENDPOINT_IP=10.141.168.1 clusterctl generate cluster byoh-cluster   \
                 --from /cm/local/apps/capi/var/cluster-template.yaml  --kubernetes-version v1.34.2 \
                 --control-plane-machine-count 1 --worker-machine-count 1 > cluster.yaml

The environment variable CONTROL_PLANE_ENDPOINT_IP must be set by the cluster administrator to a valid unused IP address. The IP address must be within the internal network of the nodes that have been assigned the CAPI role. Here the IP address of 10.141.168.1 is set.

The YAML file that is generated is typically several hundred lines long:

Example

[root@headnode ~]# wc -l cluster.yaml
285 cluster.yaml

The start of it looks like:

Example

[root@headnode ~]# head cluster.yaml
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
  name: byoh-cluster-md-0
  namespace: default
spec:
  template:
    spec: {}
---
apiVersion: cluster.x-k8s.io/v1beta1

The cluster can now be created with kubectl:

Example

[root@headnode ~]# kubectl create -f cluster.yaml
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/byoh-cluster-md-0 created
cluster.cluster.x-k8s.io/byoh-cluster created
machinedeployment.cluster.x-k8s.io/byoh-cluster-md-0 created
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/byoh-cluster-control-plane created
byocluster.infrastructure.cluster.x-k8s.io/byoh-cluster created
byomachinetemplate.infrastructure.cluster.x-k8s.io/byoh-cluster-control-plane created
byomachinetemplate.infrastructure.cluster.x-k8s.io/byoh-cluster-md-0 created
k8sinstallerconfigtemplate.infrastructure.cluster.x-k8s.io/byoh-cluster-control-plane created
k8sinstallerconfigtemplate.infrastructure.cluster.x-k8s.io/byoh-cluster-md-0 created

The very first control plane node is assigned this IP address initially, and any other control planes nodes do not try to take it at that time. Only one of the control plane nodes use this IP address at a time.

Load balancing between all three nodes is however possible—it is just not currently configured out-of-the-box at the time of writing (July 2023).

When all prerequisites are met, the BCM CAPI Infrastructure Provider initiates node provisioning, and creates the cluster. BCM generates a module file and kubeconfig for the cluster automatically.

8.3.1 Machine Provisioning

Applying the YAML for the cluster creates the machine resources. These are initially in a Pending state:

Example

root@basecm11:~# kubectl get machines
NAME                               CLUSTER        NODENAME   PROVIDERID   PHASE          AGE   VERSION
byoh-cluster-control-plane-gdm7n   byoh-cluster                           Pending        3s    v1.34.2
byoh-cluster-md-0-khrk6-mszxr      byoh-cluster                           Pending        5s    v1.34.2

The operator in the byoh-system namespace then allocates resources. It first selects the machine for the control plane:

[root@basecm11 ~]# kubectl logs -n byoh-system -l cluster.x-k8s.io/provider=infrastructure-byoh
I0601 06:16:18.202787 .. "msg"="Attempting host reservation" "cluster"="byoh-cluster" ...
I0601 06:16:18.318478 .. "msg"="Successfully attached Byohost" "byohost"="node001" "cluster"="byoh-cluster" ...

The capi-agent service on the node finds an installation script and invokes it, as indicated by the log entry:

Example

[root@node001 ~]# journalctl -u capi-agent.service -g executing
Jun 01 06:16:20 .. controller/byohost "msg"="executing install script" "name"="node001" ...

At this point the ByoHost resource should be linked to a ByoMachine resource, which is linked to a Machine resource. Until the installation script has completed, it is not easy to go the other way around from a Machine resource to find the related ByoHost resource. Once the installation script has completed, the Machine resource updates the ProviderID column with a value, but it can take some time to show up. It shows up later in this session as the value byoh://node001/f3j7mt in this example session.

The output for the installation script is only printed on completion, and the installation script is automatically removed. Figure 8.9 has more details on the installation process.

The machine transitions to the Provisioning state, and also the ByoHost resource is now tied to the given Machine:

Example

[root@basecm11 ~]# kubectl get machines
NAME                               CLUSTER        NODENAME   PROVIDERID   PHASE          AGE   VERSION
byoh-cluster-control-plane-gdm7n   byoh-cluster                           Provisioning    8s   v1.34.2
byoh-cluster-md-0-khrk6-mszxr      byoh-cluster                           Pending        10s   v1.34.2

The journal for BCM displays output similar to the following (some output ellipsized):

Example

[root@basecm11 ~]# journalctl -u cmd.service  | grep -i register
Nov 25 cmd[... [CapiNodeVersionManager::register_node], create new kube cluster: byoh-cluster, ...
Nov 25 cmd[... [CapiNodeVersionManager::register_node], add node: 1f6520c9-1475-4493-825c-ac83f...
Nov 25 cmd[... [CapiRegisterNode::async_work::async]: Registration of nodes completed: node002
Nov 25 cmd[... [CapiNodeVersionManager::register_node], add node: 42c06248-fae1-423a-82d0-9e2fd...
Nov 25 cmd[... [CapiRegisterNode::async_work::async]: Registration of nodes completed: node001

The registration of a node by the wizard is done automatically in the background, by using BCM’s Python API (PythonCM, Chapter 1 of the Developer Manual). The node is registered with the active head node, triggering a cascade of events using:

cm-kubernetes-capi-setup --register-node node001

Script logs can be found in the log file /var/log/cm-kubernetes-capi-setup.log. More on what the script does can be found in section 8.5.

If multiple nodes are being provisioned at the same time, then BCM invokes the script with more nodes as arguments so that the work is parallelized. On completion, the machine transitions to the Running state:

Example

[root@basecm11 ~]# kubectl get machine
NAME        CLUSTER        NODENAME   PROVIDERID              PHASE         AGE   VERSION
byoh-clus.. byoh-cluster   node001    byoh://node001/e7fq1b   Running       3m4s  v1.26.0
byoh-clus.. byoh-cluster                                      Provisioning  3m6s  v1.26.0

Section 8.5.1 covers the process from a different perspective, which may clarify matters further.

8.3.2 Accessing The Cluster

Assuming a Kubernetes CAPI cluster named byoh-cluster has been deployed, with three control planes, one worker, with the following machines all running:

Example

[root@basecm11 ~]# kubectl get machines
NAME         CLUSTER        NODENAME   PROVIDERID              PHASE     AGE   VERSION
byoh-clus... byoh-cluster   node006    byoh://node006/ytvpbd   Running   26m   v1.26.1
byoh-clus... byoh-cluster   node004    byoh://node004/7rczjg   Running   22m   v1.26.1
byoh-clus... byoh-cluster   node002    byoh://node002/i0tj4j   Running   44m   v1.26.1
byoh-clus... byoh-cluster   node003    byoh://node003/f18zem   Running   44m   v1.26.1

The KubeCluster entity can then be seen in cmsh:

Example

[root@basecm11 ~]# cmsh
[basecm11]% kubernetes
[basecm11->kubernetes]% show byoh-cluster
Parameter                            Value
------------------------------------ ----------------------------------------------------------------------
Name                                 byoh-cluster
Revision
Etcd Cluster
Pod Network
Pod Network Node Mask
Internal Network
KubeDNS IP                           0.0.0.0
Kubernetes API server
Kubernetes API server proxy port     6444
App Groups                           <0 in submode>
Label Sets                           <0 in submode>
Notes
Version                              1.26.1
Trusted domains                      kubernetes,kubernetes.default,kubernetes.default.svc,master,localhost
Module file template                 <690B>
Kubeadm init file                    <0B>
Service Network
Kubeadm CERT Key                     < not set >
Kube CA Cert                         < not set >
Kube CA Key                          < not set >
Kubernetes users                     <0 in submode>
External                             no
External Kubernetes Ingress server
External port                        0
Capi template                        no
Capi namespace                       default
Kubernetes management cluster        default
Ingress Proxy Enable                 no
Ingress Proxy Listen Port            443
Ingress Proxy Backend Port           0

There should also be a byoh-cluster module file available on the head node:

Example

[root@basecm11 ~]# module load kubernetes/byoh-cluster/1.26.1
[root@basecm11 ~]# kubectl get nodes
NAME      STATUS     ROLES           AGE   VERSION
node002   NotReady   control-plane   29m   v1.26.1
node003   NotReady   <none>          25m   v1.26.1
node004   NotReady   control-plane   23m   v1.26.1
node006   NotReady   control-plane   25m   v1.26.1

By default, Kubernetes CAPI clusters do not come with a networking implementation configured. This can be created by the cluster administrator, to see if this improves the state of the cluster (section 4.2.2):

Example

root@rb-capi2:~# kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.24.5 ...
                                   ... /manifests/calico-typha.yaml
poddisruptionbudget.policy/calico-kube-controllers created
poddisruptionbudget.policy/calico-typha created
serviceaccount/calico-kube-controllers created
serviceaccount/calico-node created
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
service/calico-typha created
daemonset.apps/calico-node created
deployment.apps/calico-kube-controllers created
deployment.apps/calico-typha created

The preceding session ends up with the nodes then ends up configured with Calico networking:

Example

root@rb-capi2:~# kubectl get nodes
NAME      STATUS  ROLES           AGE   VERSION
node002   Ready   control-plane   31m   v1.26.1
node003   Ready   <none>          27m   v1.26.1
node004   Ready   control-plane   25m   v1.26.1
node006   Ready   control-plane   28m   v1.26.1

8.3.3 Scaling Control Planes Or Workers

Control planes can be scaled at the level of the KubeadmControlPlane resource (some output ellipsized):

Example

[root@basecm11 ~]# module load kubernetes/default
[root@basecm11 ~]# kubectl get kubeadmcontrolplane
NAME                         CLUSTER        INITIALIZED  ... REPLICAS ... UPDATED   UNAVAILABLE ...
byoh-cluster-control-plane   byoh-cluster   true         ... 1        ... 1         1           ...
[root@basecm11 ~]# kubectl patch kubeadmcontrolplane byoh-cluster-control-plane \
                                                   --patch='{"spec": {"replicas": 3}}' \
                                                   --type=merge
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/byoh-cluster-control-plane patched

Workers can be scaled at the level of the MachineDeployment resource:

Example

[root@basecm11 ~]# kubectl get machinedeployment
NAME                CLUSTER        REPLICAS   ... UPDATED   UNAVAILABLE   PHASE       AGE   VERSION
byoh-cluster-md-0   byoh-cluster   1          ... 1         1             ScalingUp   14h   v1.26.1
[root@basecm11 ~]# kubectl scale --replicas=2 machinedeployment/byoh-cluster-md-0
machinedeployment.cluster.x-k8s.io/byoh-cluster-md-0 scaled

8.3.4 Upgrading Control Planes Or Workers

For this section, some background understanding of how Kubernetes upgrades work is recommended. A good introduction can be found at:

https://cluster-api.sigs.k8s.io/tasks/upgrading-clusters.html

Further hints and suggestions on upgrading can be found at:

One suggestion from these resources that should be followed, is first to upgrade to the latest patch version of the current minor version, and then to upgrade to the next minor version.

Typically all the control planes are upgraded first, and then the workers.

8.3.4.1 Rolling Upgrades

The default upgrade method is with rolling upgrades. More information on rolling upgrades can be found at:

https://cluster-api.sigs.k8s.io/tasks/upgrading-clusters.html#upgrading-machines-managed-by-a-machinedeployment

The rolling upgrades method requires that at least one spare ByoHost is available, since machines are replaced one by one, and both the old and new machine need to run at the same time during a part of the procedure. If assigning an extra CAPI role is a problem, then the upgrade strategy based on OnDelete can be followed instead, which is also described at that URL.

Another option, depending on the cluster, could be to temporarily scale down the workers by one during the rolling upgrade, via the MachineDeployment resource.

8.3.4.2 Deprecated APIs

8.3.4.3 Upgrading The Control Plane

The following example session upgrades the control plane from version v1.26.1 to v1.26.2:

Example

[root@basecm11 ~]# kubectl get kubeadmcontrolplane
NAME                         CLUSTER        INITIALIZED   .. REPLICAS   .. UPDATED   UNAVAILABLE   AGE   VERSION
byoh-cluster-control-plane   byoh-cluster   true          .. 1          .. 1         1             16h   v1.26.1
[root@basecm11 ~]# kubectl patch kubeadmcontrolplane byoh-cluster-control-plane \
                               --type=merge \
                               -p '{"spec": {"version": "v1.26.2"}}'
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/byoh-cluster-control-plane patched

The resource immediately shows the new version. However, the upgrade is not performed immediately:

[root@basecm11 ~]# kubectl get kubeadmcontrolplane
NAME                         CLUSTER        INITIALIZED   .. REPLICAS   .. UPDATED   UNAVAILABLE   AGE   VERSION
byoh-cluster-control-plane   byoh-cluster   true          .. 1          .. 0         1             16h   v1.26.2

First a new control plane is provisioned with the new version. Only when it is fully up, is the old control plane deleted:

[root@basecm11 ~]# kubectl get machine
NAME        CLUSTER        NODENAME   PROVIDERID              PHASE          AGE    VERSION
byoh-clus.. byoh-cluster   node001    byoh://node005/gije0o   Running        125m   v1.26.1
byoh-clus.. byoh-cluster                                      Provisioning   7s     v1.26.2
byoh-clus.. byoh-cluster   node002    byoh://node004/98s8gu   Running        98m    v1.26.1

The deletion is not carried out immediately. There is a grace period in which both control planes run. The old control plane is deleted shortly afterwards.

[root@basecm11 ~]# kubectl get machine
NAME        CLUSTER        NODENAME   PROVIDERID              PHASE     AGE     VERSION
byoh-clus.. byoh-cluster   node001    byoh://node005/gije0o   Running   127m    v1.26.1
byoh-clus.. byoh-cluster   node003    byoh://node003/mjvvvg   Running   2m30s   v1.26.2
byoh-clus.. byoh-cluster   node004    byoh://node004/98s8gu   Running   101m    v1.26.1

If there is than one control plane, then a rolling upgrade takes place, one control plane at a time.

8.3.4.4 Upgrading The Workers

The following session shows the workers being upgraded from version v1.26.1 to v1.26.2.

Example

[root@basecm11 ~]# kubectl get machinedeployment
NAME                CLUSTER        REPLICAS   .. UPDATED   UNAVAILABLE   PHASE       AGE   VERSION
byoh-cluster-md-0   byoh-cluster   3          .. 3         3             ScalingUp   17h   v1.26.1

In this case there are three workers. The version is patched with:

Example

[root@basecm11 ~]# kubectl patch machinedeployment byoh-cluster-md-0 \
                               --type=merge \
                               -p '{"spec": {"template": {"spec": {"version": "v1.26.2"}}}}'
machinedeployment.cluster.x-k8s.io/byoh-cluster-md-0 patched

The resource immediately shows the new desired version. However, the upgrade is not performed immediately:

Example

[root@basecm11 ~]# kubectl get machinedeployment
NAME                CLUSTER        REPLICAS   .. UPDATED   UNAVAILABLE   PHASE       AGE   VERSION
byoh-cluster-md-0   byoh-cluster   4          .. 1         4             ScalingUp   17h   v1.26.2

First a new worker is provisioned with the new version. Only when it is fully up, is the old worker deleted:

[root@basecm11 ~]# kubectl get machine
NAME        CLUSTER        NODENAME   PROVIDERID              PHASE          AGE     VERSION
byoh-clus.. byoh-cluster   node003    byoh://node003/mjvvvg   Running        5m45s   v1.26.2
byoh-clus.. byoh-cluster   node004    byoh://node004/98s8gu   Running        104m    v1.26.1
byoh-clus.. byoh-cluster   node002    byoh://node002/f3j7mt   Running        107m    v1.26.1
byoh-clus.. byoh-cluster   node006    byoh://node006/paityf   Running        139m    v1.26.1
byoh-clus.. byoh-cluster                                      Provisioning   7s      v1.26.2

The process repeats itself until all the workers are upgraded.

8.4 BCM Host Agent Registration

The registration process in figure 8.8 is sourced from the documentation at the upstream BYOH project:

https://github.com/vmware-tanzu/cluster-api-provider-bringyourownhost/blob/main/docs/

BYOH bootstrap flow

Figure 8.8: BYOH bootstrap flow

The roles of the platform operator and site operator in the illustration are automated by BCM.

The process initiated by BCM CAPI Infrastructure Provider starts with BCM.

When initiating, the capi-agent service verifies the existence of a configuration before launching. If a configuration does not exist, then the BCM API is used to request one.

BCM then automates all the steps necessary to generate this configuration, and prepares it for the capi-agent so it can start its process.

To clarify figure 8.8 further, before step 9 (create a new kubeconfig file at ~/.kube/config), only the bootstrap configuration exists on the node. However, after this step, two configuration files exist: the bootstrap and the definitive configuration files. The presence of these files enables BCM to determine the current phase of the bootstrap process that the capi-agent is in.

Manual generation of bootstrap configurations is also possible. Details on this are given in the documentation at https://github.com/vmware-tanzu/cluster-api-provider-bringyourownhost/blob/main/docs/getting_started.md#register-byoh-host-to-management-cluster

The host agent registration process described in the preceding is part of what is carried out for nodes when running the Assign CAPI Role option in section 8.2.2.

[root@basecm11 ~]# kubectl get byohost -A
NAMESPACE   NAME      OSNAME   OSIMAGE                            ARCH
default     node001   linux    Rocky Linux 8.7 (Green Obsidian)   amd64
default     node002   linux    Rocky Linux 8.7 (Green Obsidian)   amd64
default     node003   linux    Rocky Linux 8.7 (Green Obsidian)   amd64
default     node004   linux    Rocky Linux 8.7 (Green Obsidian)   amd64
default     node005   linux    Rocky Linux 8.7 (Green Obsidian)   amd64
default     node006   linux    Rocky Linux 8.7 (Green Obsidian)   amd64

8.5 Install Process BCM CAPI

The preceding section (section 8.4) focused on BCM CAPI host agent registration. The current section (section 8.5) discusses the deployment of Kubernetes clusters on these CAPI Hosts.

The deployment uses installation scripts transmitted to the designated nodes through “installation secrets”.

Node registration with BCM is also carried out as part of the deployment of Kubernetes on the CAPI hosts, and is distinct from the host agent registration of section 8.4. The deployment process flow is illustrated in figure 8.9:

BYOH installer flow

Figure 8.9: BYOH installer flow

8.5.1 Registration Process Of The Node With BCM

With BCM CAPI Infrastructure Provider, the installation script is a pythoncm script. It establishes communication with BCM on the active Head Node. This is referred to as “registering” the node with BCM.

The registration process involves essential bookkeeping tasks, such as tracking which nodes belong to specific Kubernetes CAPI clusters, and preparing software images with the appropriate Kubernetes versions.

The example that follows illustrates various components involved in creating a Kubernetes cluster using CAPI. In the example, a single head node operates as the Kubernetes management cluster, and multiple nodes are assigned the CAPI role. A cluster definition is generated for a control plane node and a worker node using the procedure described in section 8.3. An overview of the components involved when the cluster definition is applied is seen in figure 8.10:

CAPI summary

Figure 8.10: CAPI summary

In figure 8.10, the Kubernetes CAPI cluster has the control plane deployed on node001, followed by the deployment of the worker node software image on node002. The diagram of the is expanded from the node001 CAPI agent block illustrates the steps executed by the node001 capi-agent.

Upon registering with the BCM API, the more important actions that BCM carries out are:

  • Step 1: Preparation of Kubernetes version 1.26.0 within the software image for the registering node.

  • Step 2: Provisioning of the node through an image update, for example using the imageupdate command of cmsh (section 5.6.2 of the Administrator Manual), to ensure that it stays synchronized with its software image.

  • Step 3: Handing control back to the capi-agent, which proceeds with the subsequent step in the CAPI cluster creation process.

The steps also include processes beside the ones illustrated in figure 8.10. The more complete sequence is:

  • Step 1: The installation script is invoked with contextual information, such as the desired Kubernetes version (for example: 1.26.0) and the cluster name.

  • Step 2: The installation script, implemented as a PythonCM script, establishes communication with BCM. It registers itself with the active head node and remains in a waiting state until notified to terminate.

  • Step 3: BCM creates a mapping that specifies which node should be provisioned with the corresponding Kubernetes version.

  • Step 4: BCM ensures that the software image for the node contains the requested Kubernetes version.

  • Step 5: BCM ensures that the node is provisioned with its designated software image.

  • Step 6: BCM signals to the node that the installation script has completed.

From this point onward, the default logic for the BCM CAPI operator takes over, and kubeadm initializes the node accordingly.

8.5.2 Creating A Kubernetes Cluster Via CAPI

The steps in section 8.5.1, are about the node registration process and node provisioning during Kubernetes cluster creation with CAPI. The following steps zoom out further, and describe the complete process of creating a Kubernetes cluster through CAPI.

  • Step 1: The system administrator defines a cluster to be deployed by the BCM CAPI operator, and feeds it to the Kubernetes API server using kubectl.

  • Step 2: The definition results in a number of Kubernetes resources being created, such as a Cluster, ByoCluster, MachineDeployment.

  • Step 3: The BCM CAPI operator responds to these newly-created resources.

  • Step 4: CAPI starts assigning CAPI agents to reconcile specific Machines, beginning with the first control plane. This is indicated by the red provisioning line, numbered with a 1.

  • Step 5: BCM prepares a module file and kubeconfig for the new Kubernetes CAPI cluster, and writes these to disk for the system administrator.

  • Step 6: BCM CAPI operator updates its records using the newly-created control plane and proceeds with provisioning the additional control planes or workers, such as the additional worker. This is indicated by the red line numbered with a 2.

8.6 Configuring CAPI Versions In Software Images

BCM takes care of configuring CAPI versions in software images automatically. This is part of its BCM CAPI node registration process, as mentioned earlier in section 8.5.

An alternative is to manually pre-install a specific version of Kubernetes in a software image. This can be done from within the softwareimage mode of cmsh:

[root@basecm11 ~]# cmsh
[basecm11]% softwareimage
[basecm11->softwareimage]% use default-image
[basecm11->softwareimage[default-image]]% help capi
Name:
capi - Manage Kubernetes CAPI versions on the image
Usage:
capi [OPTIONS] list
capi [OPTIONS] add <version> [<version> ...]
capi [OPTIONS] remove <version> [<version> ...]
capi [OPTIONS] clear
Options:
-v, --verbose
Be more verbose
-d, --delimiter <string>
Use <string> as delimiter between columns. Use {} for JSON, and {<digit>} for JSON with a specific indentation.
--image, -i <list of images>
Perform action on comma separated list of images
--repo-refresh, -r
Refresh the repository cache before adding new versions
--debug
Run script with debug on
Examples:
capi list                 List CAPI versions on all / current image
capi clear                Remove all CAPI versions on all / current image
capi add 1.26.0 1.27.*    Add specified versions on all / current image

Example

[basecm11->softwareimage[default-image]]% capi list
Node          image                     versions                        Result   Error
------------- ------------------------- ------------------------------- -------- -----
basecm11      /cm/images/default-image  1.24.9, 1.23.0, 1.26.1, 1.26.2  good
[basecm11->softwareimage[default-image]]% capi add 1.27.*
Node          image                     versions                                                Result   Error
------------- ------------------------- ------------------------------------------------------- -------- -----
basecm11      /cm/images/default-image  1.24.9, 1.23.0, 1.26.1, 1.26.2, 1.27.0, 1.27.1, 1.27.2  good

8.7 Removing Kubernetes CAPI Clusters

The following steps remove the CAPI clusters:

  • Step 1: The removal of the CAPI clusters is started with the kubectl delete command:

    [root@basecm11 ~]# kubectl delete -f cluster.yaml
    ...

  • Step 2: If all CAPI hosts are not part of any significant clusters, then the configuration overlay configuration overlays are removed:

    [root@basecm11 ~]# cmsh
    [basecm11]% configurationoverlay
    [basecm11->configurationoverlay]% remove capi-mgmt
    [basecm11->configurationoverlay*]% commit
    Successfully removed 1 ConfigurationOverlays
    Successfully committed 0 ConfigurationOverlays

    The removal of the configuration overlays causes containerd and the CAPI agents to stop on the hosts.

  • Step 3: For additional cleanliness, the bootstrap configurations for each of the hosts should be removed:

    [root@basecm11 ~]# kubectl delete bootstrapkubeconfig --all
    bootstrapkubeconfig.infrastructure.cluster.x-k8s.io "bootstrap-kubeconfig-node001" deleted
    bootstrapkubeconfig.infrastructure.cluster.x-k8s.io "bootstrap-kubeconfig-node002" deleted
    bootstrapkubeconfig.infrastructure.cluster.x-k8s.io "bootstrap-kubeconfig-node003" deleted
    bootstrapkubeconfig.infrastructure.cluster.x-k8s.io "bootstrap-kubeconfig-node004" deleted
    bootstrapkubeconfig.infrastructure.cluster.x-k8s.io "bootstrap-kubeconfig-node005" deleted
    bootstrapkubeconfig.infrastructure.cluster.x-k8s.io "bootstrap-kubeconfig-node006" deleted

  • Step 4: The infrastructure provider is then removed:

    [root@basecm11 ~]# module load capi/1.3.0
    [root@basecm11 ~]# clusterctl delete --infrastructure byoh
    Deleting Provider="infrastructure-byoh" Version="" Namespace="byoh-system"

  • Step 5: The removal of all the ByoHost resources can be checked:

    [root@basecm11 ~]# kubectl get byohost
    No resources found

    For clean-up the command

    kubectl delete byohost ...

    can be used.

  • Step 6: Finally, the Kubernetes CAPI clusters is eliminated from BCM itself:

    [root@basecm11 ~]# cmsh
    [basecm11]% kubernetes
    [basecm11->kubernetes]% remove byoh-cluster
    [basecm11->kubernetes*]% commit
    [basecm11->kubernetes]%

8.8 Kubernetes CAPI Templates

In BCM, each Kubernetes cluster is represented as a KubeCluster entity stored in the BCM database. These entities can be viewed in the kubernetes submode of cmsh:

Example

[root@basecm11 ~]# cmsh
[basecm11]% kubernetes
[basecm11->kubernetes]% list
Name (key)
-------------------
default

  • For every Kubernetes management cluster, a default KubeCluster CAPI template is generated once the CAPI role node is assigned using the wizard, as discussed in section 8.2.2. This template serves as a base for all subsequent Kubernetes clusters created via CAPI.

  • For example, if an administrator creates a cluster my-capi-cluster via CAPI, for the Kubernetes management cluster mgmt, then BCM clones the capi-mgmt-template KubeCluster entity to create a new KubeCluster my-capi-cluster.

  • At the time of CAPI cluster creation, the KubeCluster template can also be customized using an annotation in the Cluster resource definition:

    Example

    apiVersion: cluster.x-k8s.io/v1beta1
    kind: Cluster
    metadata:
      labels:
        cni: byoh-cluster-crs-0
        crs: "true"
        infrav1.nvidia.x-k8s.io/capi-kube-template: "my-capi-cluster-template"
      name: byoh-cluster
      namespace: default
    spec:
      clusterNetwork:
        pods:
          cidrBlocks:
          - 192.168.0.0/16
        serviceDomain: cluster.local
        services:
          cidrBlocks:
          - 10.128.0.0/12
      controlPlaneRef:
        apiVersion: controlplane.cluster.x-k8s.io/v1beta1
        kind: KubeadmControlPlane
        name: byoh-cluster-control-plane
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: ByoCluster
        name: byoh-cluster

    The provided YAML illustrates a cluster byoh-cluster definition, with the specified capi-template set to my-capi-cluster-template. If present, then this is used instead of the default.

  • The fallback sequence for the CAPI template is: The specified label (for example: my-capi-cluster-template) is used if it exists; otherwise the default capi-mgmt-template is used. If neither is available, then capi-template is used.