> This page is for version 10. > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Kubernetes Cluster API The Kubernetes Cluster API (CAPI), as explained in the introduction to the online [Cluster API Book at `https://cluster-api.sigs.k8s.io/`](https://cluster-api.sigs.k8s.io/), "is a Kubernetes sub-project focused on providing declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters". The Cluster API Book is the official Kubernetes project documentation for CAPI. This chapter describes the installation and usage of the NVIDIA Base Command Manager CAPI extension called BCM Kubernetes CAPI Infrastructure Provider. ## 8.1 Kubernetes Cluster API Components An overview of the CAPI components is shown in figure [8.1](#fig-capi-components). Further details about the components are given in the sections of this chapter that follow. ![CAPI components](/base-command-manager/product-manuals/_fern-img/8f329188d47039f3d358bee73d444ef35d7660da20444086fe625798f018e6ef.webp) *Figure 8.1: CAPI components* Figure [8.1](#fig-capi-components) shows a standard Kubernetes cluster deployed on an active head node. The cluster has been modified to become a Kubernetes management cluster using the BCM CAPI Infrastructure Provider, and has successfully deployed two additional Kubernetes clusters through CAPI. These additional clusters may be running different versions of Kubernetes. The system administrator receives module files and kubeconfig files for all three Kubernetes clusters. ### 8.1.1 Kubernetes Management Cluster In BCM documentation, the term Kubernetes management cluster is used specifically and precisely to refer to a Kubernetes cluster that operates as the management cluster for CAPI. The ability to modify an *external* Kubernetes cluster to operate as a Kubernetes management cluster is under preparation at the time of writing (June 2023) of this section. This includes scenarios such as using an existing Kubernetes cluster, hosted on a public cloud, to serve as the Kubernetes management cluster for a Kubernetes cluster deployed through BCM. At present, any Kubernetes cluster deployed through BCM (Chapter [4](/containerization/kubernetes)) can be modified into a CAPI management cluster. ### 8.1.2 Kubernetes CAPI Cluster BCM documentation uses the term Kubernetes CAPI cluster to refer specifically to a Kubernetes cluster deployed via the Cluster API. Upstream Kubernetes documentation also sometimes refers to a Kubernetes cluster deployed via CAPI as a workload cluster. ### 8.1.3 BCM CAPI Infrastructure Provider The BCM CAPI Infrastructure Provider is a derivative of the Bring Your Own Host (BYOH) CAPI provider. The BYOH CAPI provider can also be referred to as the BYOH CAPI Infrastructure Provider, and is available as a GitHub project at:\ [https://github.com/vmware-tanzu/cluster-api-provider-bringyourownhost](https://github.com/vmware-tanzu/cluster-api-provider-bringyourownhost) Similar to other CAPI providers, the BCM CAPI Infrastructure Provider is an extension of CAPI itself, and both the CAPI provider and CAPI are required on the Kubernetes management cluster. #### 8.1.3.1 The BCM CAPI Operator The BCM CAPI operator is an operator deployed in the `byoh-system` namespace on the Kubernetes management cluster. The operator serves as the central point of connection for BCM CAPI host agents (section [8.1.3.2](#8132-bcm-capi-host-agents)). It also monitors the state of clusters and machines for the Kubernetes CAPI clusters. ```bash root@headnode:~# kubectl get pod -n byoh-system NAME READY STATUS RESTARTS AGE byoh-controller-manager-6c98cbf44c-7gdbx 2/2 Running 2 (5h55m ago) 23h ``` #### 8.1.3.2 BCM CAPI Host Agents BCM CAPI host agents are scheduled to run on selected hosts. These hosts are defined by the cluster administrator when the `cm-kubernetes-capi-setup` wizard (section [8.2.2](#822-the-assign-capi-role-option)) is run. The wizard ensures that a selected host gets the right packages in its software image, and that the host is assigned the right BCM roles. For CAPI calls to function, the `CapiRole` is assigned. Assignment can be done either before or after creating a Kubernetes CAPI cluster via the Kubernetes management cluster. The role sets up a capi-agent service on the host for the associated Kubernetes management cluster. When the role is: * assigned: BCM initiates and bootstraps the service * unassigned: BCM halts the service and makes it unavailable for use by CAPI on the associated Kubernetes management cluster This differs from cloud-based CAPI providers, which do not require managing a limited number of pre-existing nodes, but instead start up new nodes on demand as needed. If a Kubernetes CAPI cluster receives a cluster request and there are no hosts available, then machines remain in a `Pending` state until hosts become available. Once hosts become available, the status transitions to `Provisioning`, and then eventually to `Running` (section [8.3.1](#831-machine-provisioning)). #### 8.1.3.3 BCM CAPI Vs BYOH Once the actual Kubernetes CAPI clusters are deployed on designated CAPI hosts, BCM uses its Python API to carry out the following operations: * The requested Kubernetes version for each node is stored in the BCM database. * The requested Kubernetes version is made available in the corresponding software image for the node. * The node is provisioned with the software image. * The installer is notified that the Kubernetes installation can proceed. In contrast to the BYOH provider, BCM CAPI has the following extra features: * Linux distributions can be other than just Ubuntu 20.04. * Kubernetes versions can be other than only those for which parcels have been created, as the BCM Kubernetes integration does not rely on parcels. ## 8.2 The Kubernetes CAPI Wizard The `cm-kubernetes-capi-setup` wizard installs the BCM CAPI Infrastructure Provider, and assigns the CAPI role to nodes. ![CAPI setup wizard](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/6da2e64290596c09c9615cfc0a196eca7004bd5b4e200f0d24bb84e74d16af19/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/capi-setup-wizard.png) *Figure 8.2: CAPI setup wizard* ### 8.2.1 The `Install CAPI` Option The `Install CAPI` option of figure [8.2](#fig-capi-setup-wizard) leads to a screen that prompts the user to select the Kubernetes cluster instance on which to install the BCM CAPI Infrastructure Provider. ![Selection of Kubernetes cluster for installation of the BCM CAPI Infrastructure Provider](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/a1e4f9772e275058ab1e3b6bd594a9b97dfb8d7e3f811cf8afacfe06cdb27cc1/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/capi-install-capi-select-kc.png) *Figure 8.3: Selection of Kubernetes cluster for installation of the BCM CAPI Infrastructure Provider* If needed: * `cert-manager` and the Cluster API itself are installed * the CAPI `clusterctl` tool is run in the back end to carry out initialization #### 8.2.1.1 The `Install CAPI` Option Actions Execution of the `Install CAPI` operation: * prepares a Kubernetes cluster template for CAPI (section [8.8](#88-kubernetes-capi-templates)). * installs the BCM CAPI Infrastructure Provider on the Kubernetes Cluster with the command: `clusterctl init infrastructure byoh`. * waits for the BCM CAPI operator to be ready for operation. * patches the BCM CAPI operator to use the appropriate image. Patching is required until the BCM changes get into the upstream repository. * makes the `capi` module available for loading #### 8.2.1.2 The `Install CAPI` Option Changes To Kubernetes Management Cluster After the `Install CAPI` option has run, the operators installed and running on the Kubernetes management cluster can be seen (some output truncated): **Example** ```bash [root@basecm10 ~]# module load kubernetes [root@basecm10 ~]# kubectl get pod -A | grep -E "byoh-system|capi|cert-manager" byoh-system byoh-controller-manager-ff7f68bb4-vprrz 2/2 ... capi-kubeadm-bootstrap-system capi-kubeadm-bootstrap-controller-manager-7945579f8c-4ntb2 1/1 ... capi-kubeadm-control-plane-system capi-kubeadm-control-plane-controller-manager-66cdfb477b-h5779 1/1 ... capi-system capi-controller-manager-64cb86f545-8k9zt 1/1 ... cert-manager cert-manager-5d4c5bc8bc-j5hbp 1/1 ... cert-manager cert-manager-cainjector-79bc559d9d-qfvph 1/1 ... cert-manager cert-manager-webhook-5cf45f5b6-9gwlx 1/1 ... ``` #### 8.2.1.3 The `Install CAPI` Changes To BCM A new Kubernetes cluster template is generated in BCM on running `Install CAPI`. The template naming convention used follows the form: `capi-` \<*mgmt_cluster_name*>`-template` In the present example, where the name of the Kubernetes management cluster is `default`, the resulting template would be called `capi-default-template`. The name `default` is set by default when creating a Kubernetes instance (figure [4.2](/containerization/kubernetes#421-kubernetes-networking)). ```bash [root@basecm10 ~]# cmsh [basecm10]% kubernetes [basecm10->kubernetes]% list Name (key) ---------------------- capi-default-template default [basecm10->kubernetes]% show capi-default-template Parameter Value ------------------------------------ ---------------------------------------------------------------------- Name capi-default-template Revision Etcd Cluster Pod Network Pod Network Node Mask Internal Network KubeDNS IP 0.0.0.0 Kubernetes API server Kubernetes API server proxy port 6444 App Groups <0 in submode> Label Sets <0 in submode> Notes Version Trusted domains kubernetes,kubernetes.default,kubernetes.default.svc,master,localhost Module file template <690B> Kubeadm init file <0B> Service Network Kubeadm CERT Key < not set > Kube CA Cert < not set > Kube CA Key < not set > Kubernetes users <0 in submode> External no External Kubernetes Ingress server External port 0 Capi template yes Capi namespace default Kubernetes management cluster default ``` This becomes the default template used for generating Kubernetes CAPI clusters associated with this specific Kubernetes management cluster. Users can modify this template, and can even create additional templates as needed. The YAML configuration for a CAPI cluster also lets users specify a template, via annotations within the YAML file (section [8.8](#88-kubernetes-capi-templates)). ### 8.2.2 The `Assign CAPI Role` Option After the `Install CAPI` operation has completed its run, the `Assign CAPI Role` operation can be carried out from the setup screen of figure [8.2](#fig-capi-setup-wizard). The `Assign CAPI Role` operation carries out a CAPI role assignment to a group of nodes by creating a configuration overlay (section [4.5](/containerization/kubernetes#45-kubernetes-configuration-overlays)). The role can then be assigned either to a category, or to individual nodes. The software images on the chosen nodes get the packages needed to run the CAPI agents. When the `Assign CAPI Role` operation is carried out, the screens in figures [8.4](#fig-capi-assign-capi-role-kc)-[8.7](#fig-capi-assign-capi-role-nodes) may be displayed: ![Select a Kubernetes management cluster](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/1cdc2c2a1d825fffd00caf532aaa7439e3e9cd0af8412a77cd24b20ba48d0146/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/capi-assign-capi-role-kc.png) *Figure 8.4: Select a Kubernetes management cluster* ![Customize the configuration overlay](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/ffb5d56373e93fa1a4334e2494c55f3375de91147cf974ae1ddf0ffbc8838f8d/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/capi-assign-capi-role-overlay.png) *Figure 8.5: Customize the configuration overlay* ![Choose nodes via categories](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/f0a8eb356c9f113525520cb8f2b9c145e912fecf920002378bff19b2131f9e6e/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/capi-assign-capi-role-categories.png) *Figure 8.6: Choose nodes via categories* ![Choose nodes directly](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/1cddc4e6e6cc121ef6bfb957b146ef13531ab8ee430f33cb66a7cb6aae815432/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/capi-assign-capi-role-nodes.png) *Figure 8.7: Choose nodes directly* Some of the screens may be skipped if they are not needed. For example, the screen for selecting individual nodes (figure [8.7](#fig-capi-assign-capi-role-nodes)) is not displayed if all the nodes have already been assigned during selection via categories. #### 8.2.2.1 The `Assign CAPI Role` Actions During execution of the `Assign CAPI Role` operation, the wizard: * prepares the Kubeadm and Helm repositories in the software images for the nodes. * writes the IP Forwarding configuration to the software images for the nodes. * installs necessary packages in the software images for the nodes (for example: `cm-capi`). * provisions the nodes from their software images. * creates a configuration overlay in BCM. * assigns the selected categories and nodes to the overlay. * assigns the `capi` role and `containerd` roles to the overlay. #### 8.2.2.2 The `Assign CAPI Role` Changes To BCM The configuration overlay created during the execution of the `Assign CAPI Role` operation is now visible in the `configurationoverlay` mode (some output elided): **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% configurationoverlay [basecm10->configurationoverlay]% list Name (key) Priority All head nodes Nodes Categories Roles -------------------- ---------- -------------- ---------------- ---------------- ----------------- capi-default 500 no node001,node002 capi kube-default-etcd 500 no node003 Etcd::Host ... [basecm10->configurationoverlay]% show capi-default Parameter Value -------------------------------- ------------------------------------------------ Name capi-default Revision All head nodes no Priority 500 Nodes node001,node002 Categories Roles capi Customizations <0 in submode> ``` The preceding overlay shows that the `capi` role has been assigned to nodes `node001` and `node002`. The `capi` role sets the capi-agent service for a node. If it is set, then BCM bootstraps the agent (section [8.4](#84-bcm-host-agent-registration)) and initiates the `capi-agent` systemd service that runs on the node. The `containerd` role takes care of the containerd service, so that `containerd` is also started on the nodes. The containerd service is needed after the agent initiates the provisioning of its node to integrate it into a Kubernetes CAPI cluster. This is because a kubelet service is eventually started, which relies on `containerd`. CAPI uses `kubeadm` for node provisioning and management, which means that `containerd` must be running in advance. Therefore, as part of its pre-launch checks, `kubeadm` verifies that the containerd service is operational. #### 8.2.2.3 The `Assign CAPI Role` Changes To the Kubernetes management cluster The BCM CAPI host agents should register with the Kubernetes management cluster, and can then be seen on running `kubectl` with the `get byohost` option: **Example** ```bash [root@basecm10 ~]# module load kubernetes/default/1.26.5-0 [root@basecm10 ~]# kubectl get byohost -A NAMESPACE NAME OSNAME OSIMAGE ARCH default node001 linux Rocky Linux 8.7 (Green Obsidian) amd64 default node002 linux Rocky Linux 8.7 (Green Obsidian) amd64 ``` Registration does not necessarily make these hosts a part of any Kubernetes CAPI cluster. However, if there are clusters that have been created beforehand, and there are machines that are awaiting additional resources, then it might be that some or all of these hosts are immediately provisioned. A simple method to check this is by querying the machine resource. ```bash [root@basecm10 ~]# kubectl get machine -A No resources found ``` If however there were machines that were pending, the output might look like: ```bash [root@basecm10 ~]# kubectl get machines NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION byoh-cluster-control-plane-h2d64 byoh-cluster Provisioning 4s v1.26.1 byoh-cluster-md-0-56985bf9d6xkhj68-dxvkc byoh-cluster Pending 6s v1.26.1 ``` When the machines are provisioned, the Kubernetes CAPI cluster output state might look like: ```bash [root@basecm10 ~]# kubectl get ma #ma|machine|machines are synonymous. Also byom|byomachine|byomachines NAME CLUSTER NODENAME PROVIDERID PHASE AGE ... byoh-cluster-control-plane-h2d64 byoh-cluster node001 byoh://node001/e7fq1b Running 3h3m... byoh-cluster-md-0-56985bf9d6xkhj68-dxvkc byoh-cluster node002 byoh://node002/ngkju0 Running 3h3m... ``` This is discussed further in section [8.3](#83-deploying-a-kubernetes-cluster-through-capi). ## 8.3 Deploying A Kubernetes Cluster Through CAPI The BCM CAPI Infrastructure Provider configures the Kubernetes management cluster. A Kubernetes cluster can then be created through CAPI, and deployed on the available CAPI nodes. If resources are not available, then the cluster cannot assign nodes to the newly-created cluster definition for provisioning. This results in a persistent `Pending` state for various resources. The `capi` module makes the command line tool `clusterctl` available, which can be used to generate a cluster manifest. In the following example, the tool is used to define a cluster with Kubernetes version 1.26.0, comprising one control plane node and one worker node. **Example** ```bash [root@headnode ~]# module load capi/1.3.0 [root@headnode ~]# CONTROL_PLANE_ENDPOINT_IP=10.141.168.1 clusterctl generate cluster byoh-cluster \ --infrastructure byoh --kubernetes-version v1.26.0 \ --control-plane-machine-count 1 --worker-machine-count 1 > cluster.yaml ``` The environment variable `CONTROL_PLANE_ENDPOINT_IP` must be set by the cluster administrator to a valid unused IP address. The IP address must be within the internal network of the nodes that have been assigned the CAPI role. Here the IP address of `10.141.168.1` is set. The YAML file that is generated is typically several hundred lines long: **Example** ```bash [root@headnode ~]# wc -l cluster.yaml 218 cluster.yaml ``` The start of it looks like: **Example** ```bash [root@headnode ~]# head cluster.yaml apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 kind: KubeadmConfigTemplate metadata: name: byoh-cluster-md-0 namespace: default spec: template: spec: {} --- apiVersion: cluster.x-k8s.io/v1beta1 ``` The cluster can now be created with `kubectl`: **Example** ```bash [root@headnode ~]# kubectl create -f cluster.yaml kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/byoh-cluster-md-0 created cluster.cluster.x-k8s.io/byoh-cluster created machinedeployment.cluster.x-k8s.io/byoh-cluster-md-0 created kubeadmcontrolplane.controlplane.cluster.x-k8s.io/byoh-cluster-control-plane created byocluster.infrastructure.cluster.x-k8s.io/byoh-cluster created byomachinetemplate.infrastructure.cluster.x-k8s.io/byoh-cluster-control-plane created byomachinetemplate.infrastructure.cluster.x-k8s.io/byoh-cluster-md-0 created k8sinstallerconfigtemplate.infrastructure.cluster.x-k8s.io/byoh-cluster-control-plane created k8sinstallerconfigtemplate.infrastructure.cluster.x-k8s.io/byoh-cluster-md-0 created ``` The very first control plane node is assigned this IP address initially, and any other control planes nodes do not try to take it at that time. Only one of the control plane nodes use this IP address at a time. Load balancing between all three nodes is however possible—it is just not currently configured out-of-the-box at the time of writing (July 2023). When all prerequisites are met, the BCM CAPI Infrastructure Provider initiates node provisioning, and creates the cluster. BCM generates a module file and kubeconfig for the cluster automatically. ### 8.3.1 Machine Provisioning Applying the YAML for the cluster creates the machine resources. These are initially in a `Pending` state: **Example** ```bash [root@basecm10 ~]# kubectl get machines NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION byoh-cluster-control-plane-jzm22 byoh-cluster Pending 2s v1.26.0 byoh-cluster-md-0-5c594b479cx9jcz8-ws56w byoh-cluster Pending 4s v1.26.0 ``` The operator in the `byoh-system` namespace then allocates resources. It first selects the machine for the control plane: ```bash [root@basecm10 ~]# kubectl logs -n byoh-system -l cluster.x-k8s.io/provider=infrastructure-byoh I0601 06:16:18.202787 .. "msg"="Attempting host reservation" "cluster"="byoh-cluster" ... I0601 06:16:18.318478 .. "msg"="Successfully attached Byohost" "byohost"="node001" "cluster"="byoh-cluster" ... ``` The capi-agent service on the node finds an installation script and invokes it, as indicated by the log entry: **Example** ```bash [root@node001 ~]# journalctl -u capi-agent.service -g executing Jun 01 06:16:20 .. controller/byohost "msg"="executing install script" "name"="node001" ... ``` At this point the `ByoHost` resource should be linked to a `ByoMachine` resource, which is linked to a `Machine` resource. Until the installation script has completed, it is not easy to go the other way around from a `Machine` resource to find the related `ByoHost` resource. Once the installation script has completed, the `Machine` resource updates the `ProviderID` column with a value, but it can take some time to show up. It shows up later in this session as the value `byoh://node001/f3j7mt` in this example session. The output for the installation script is only printed on completion, and the installation script is automatically removed. Figure [8.9](#fig-byoh-installer-flow) has more details on the installation process. The machine transitions to the `Provisioning` state, and also the `ByoHost` resource is now tied to the given `Machine`: **Example** ```bash [root@basecm10 ~]# kubectl get machines NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION byoh-cluster-control-plane-jzm22 byoh-cluster Provisioning 4s v1.26.0 byoh-cluster-md-0-5c594b479cx9jcz8-ws56w byoh-cluster Pending 6s v1.26.0 ``` The journal for BCM displays output similar to the following (some output ellipsized): **Example** ```bash [root@basecm10 ~]# journalctl -u cmd.service | grep -i register Jun 01...Info: [Capi...::register_node], create new kube cluster: byoh-cluster, unable to find template:... Jun 01...Info: [Capi...::register_node], create new kube cluster: byoh-cluster, from: capi-mgmt-template... Jun 01...Info: [Capi...::register_node], add node: 3b45eb7c-c93b-4102-914f-4bddbeb3a8a4, version: 1.26.0... Jul 01...Info: [CapiRegisterNode::async_work]: registered: node002 ``` The registration of a node by the wizard is done automatically in the background, by using BCM’s Python API (PythonCM, [Chapter 1 of the *Developer Manual*](/developer/pythonapi)). The node is registered with the active head node, triggering a cascade of events using: `cm-kubernetes-capi-setup --register-node node001` Script logs can be found in the log file `/var/log/cm-kubernetes-capi-setup.log`. More on what the script does can be found in section [8.5](#85-install-process-bcm-capi). If multiple nodes are being provisioned at the same time, then BCM invokes the script with more nodes as arguments so that the work is parallelized. On completion, the machine transitions to the `Running` state: **Example** ```bash [root@basecm10 ~]# kubectl get machine NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION byoh-clus.. byoh-cluster node001 byoh://node001/e7fq1b Running 3m4s v1.26.0 byoh-clus.. byoh-cluster Provisioning 3m6s v1.26.0 ``` Section [8.5.1](#851-registration-process-of-the-node-with-bcm) covers the process from a different perspective, which may clarify matters further. ### 8.3.2 Accessing The Cluster Assuming a Kubernetes CAPI cluster named `byoh-cluster` has been deployed, with three control planes, one worker, with the following machines all running: **Example** ```bash [root@basecm10 ~]# kubectl get machines NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION byoh-clus... byoh-cluster node006 byoh://node006/ytvpbd Running 26m v1.26.1 byoh-clus... byoh-cluster node004 byoh://node004/7rczjg Running 22m v1.26.1 byoh-clus... byoh-cluster node002 byoh://node002/i0tj4j Running 44m v1.26.1 byoh-clus... byoh-cluster node003 byoh://node003/f18zem Running 44m v1.26.1 ``` The `KubeCluster` entity can then be seen in `cmsh`: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% kubernetes [basecm10->kubernetes]% show byoh-cluster Parameter Value ------------------------------------ ---------------------------------------------------------------------- Name byoh-cluster Revision Etcd Cluster Pod Network Pod Network Node Mask Internal Network KubeDNS IP 0.0.0.0 Kubernetes API server Kubernetes API server proxy port 6444 App Groups <0 in submode> Label Sets <0 in submode> Notes Version 1.26.1 Trusted domains kubernetes,kubernetes.default,kubernetes.default.svc,master,localhost Module file template <690B> Kubeadm init file <0B> Service Network Kubeadm CERT Key < not set > Kube CA Cert < not set > Kube CA Key < not set > Kubernetes users <0 in submode> External no External Kubernetes Ingress server External port 0 Capi template no Capi namespace default Kubernetes management cluster default ``` There should also be a `byoh-cluster` module file available on the head node: **Example** ```bash [root@basecm10 ~]# module unload kubernetes/mgmt/1.24.9-00 [root@basecm10 ~]# module load kubernetes/byoh-cluster/1.26.1 [root@basecm10 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION node002 NotReady control-plane 29m v1.26.1 node003 NotReady 25m v1.26.1 node004 NotReady control-plane 23m v1.26.1 node006 NotReady control-plane 25m v1.26.1 ``` By default, Kubernetes CAPI clusters do not come with a networking implementation configured. This can be created by the cluster administrator, to see if this improves the state of the cluster (section [4.2.2](/containerization/kubernetes#422-kubernetes-core-add-ons)): **Example** ```bash root@rb-capi2:~# kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.24.5 ... ... /manifests/calico-typha.yaml poddisruptionbudget.policy/calico-kube-controllers created poddisruptionbudget.policy/calico-typha created serviceaccount/calico-kube-controllers created serviceaccount/calico-node created configmap/calico-config created customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created clusterrole.rbac.authorization.k8s.io/calico-node created clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created clusterrolebinding.rbac.authorization.k8s.io/calico-node created service/calico-typha created daemonset.apps/calico-node created deployment.apps/calico-kube-controllers created deployment.apps/calico-typha created ``` The preceding session ends up with the nodes then ends up configured with Calico networking: **Example** ```bash root@rb-capi2:~# kubectl get nodes NAME STATUS ROLES AGE VERSION node002 Ready control-plane 31m v1.26.1 node003 Ready 27m v1.26.1 node004 Ready control-plane 25m v1.26.1 node006 Ready control-plane 28m v1.26.1 ``` ### 8.3.3 Scaling Control Planes Or Workers Control planes can be scaled at the level of the `KubeadmControlPlane` resource (some output ellipsized): **Example** ```bash [root@basecm10 ~]# module load kubernetes/default [root@basecm10 ~]# kubectl get kubeadmcontrolplane NAME CLUSTER INITIALIZED ... REPLICAS ... UPDATED UNAVAILABLE ... byoh-cluster-control-plane byoh-cluster true ... 1 ... 1 1 ... [root@basecm10 ~]# kubectl patch kubeadmcontrolplane byoh-cluster-control-plane \ --patch='{"spec": {"replicas": 3}}' \ --type=merge kubeadmcontrolplane.controlplane.cluster.x-k8s.io/byoh-cluster-control-plane patched ``` Workers can be scaled at the level of the `MachineDeployment` resource: **Example** ```bash [root@basecm10 ~]# kubectl get machinedeployment NAME CLUSTER REPLICAS ... UPDATED UNAVAILABLE PHASE AGE VERSION byoh-cluster-md-0 byoh-cluster 1 ... 1 1 ScalingUp 14h v1.26.1 [root@basecm10 ~]# kubectl scale --replicas=2 machinedeployment/byoh-cluster-md-0 machinedeployment.cluster.x-k8s.io/byoh-cluster-md-0 scaled ``` ### 8.3.4 Upgrading Control Planes Or Workers For this section, some background understanding of how Kubernetes upgrades work is recommended. A good introduction can be found at: [https://cluster-api.sigs.k8s.io/tasks/upgrading-clusters.html](https://cluster-api.sigs.k8s.io/tasks/upgrading-clusters.html) Further hints and suggestions on upgrading can be found at: * [https://github.com/kubernetes/sig-release/blob/master/release-engineering/versioning.md#kubernetes-release-versioning](https://github.com/kubernetes/sig-release/blob/master/release-engineering/versioning.md#kubernetes-release-versioning): discusses release versioning, and is a recommended first read * [https://kubernetes.io/releases/version-skew-policy/](https://kubernetes.io/releases/version-skew-policy/): discusses version skew policy, but is also a document that provides more detailed information One suggestion from these resources that should be followed, is first to upgrade to the latest patch version of the current minor version, and then to upgrade to the next minor version. Typically all the control planes are upgraded first, and then the workers. #### 8.3.4.1 Rolling Upgrades The default upgrade method is with rolling upgrades. More information on rolling upgrades can be found at: [https://cluster-api.sigs.k8s.io/tasks/upgrading-clusters.html#upgrading-machines-managed-by-a-machinedeployment](https://cluster-api.sigs.k8s.io/tasks/upgrading-clusters.html#upgrading-machines-managed-by-a-machinedeployment) The rolling upgrades method requires that at least one spare `ByoHost` is available, since machines are replaced one by one, and both the old and new machine need to run at the same time during a part of the procedure. If assigning an extra CAPI role is a problem, then the upgrade strategy based on `OnDelete` can be followed instead, which is also described at that URL. Another option, depending on the cluster, could be to temporarily scale down the workers by one during the rolling upgrade, via the `MachineDeployment` resource. #### 8.3.4.2 Deprecated APIs * The deprecation guide at: [https://kubernetes.io/docs/reference/using-api/deprecation-guide/](https://kubernetes.io/docs/reference/using-api/deprecation-guide/) should be read before carrying out upgrades. The cluster administrator should check the changelogs and upstream documentation for obsolete APIs in the target version. * The `pluto` utility ([https://github.com/FairwindsOps/pluto](https://github.com/FairwindsOps/pluto)) can check for deprecated API usage. * The `kubent` utility ([https://github.com/doitintl/kube-no-trouble](https://github.com/doitintl/kube-no-trouble)) can also be used, but seems a few Kubernetes versions behind at the time of writing (June 2023). #### 8.3.4.3 Upgrading The Control Plane The following example session upgrades the control plane from version `v1.26.1` to `v1.26.2`: **Example** ```bash [root@basecm10 ~]# kubectl get kubeadmcontrolplane NAME CLUSTER INITIALIZED .. REPLICAS .. UPDATED UNAVAILABLE AGE VERSION byoh-cluster-control-plane byoh-cluster true .. 1 .. 1 1 16h v1.26.1 [root@basecm10 ~]# kubectl patch kubeadmcontrolplane byoh-cluster-control-plane \ --type=merge \ -p '{"spec": {"version": "v1.26.2"}}' kubeadmcontrolplane.controlplane.cluster.x-k8s.io/byoh-cluster-control-plane patched ``` The resource immediately shows the new version. However, the upgrade is not performed immediately: ```bash [root@basecm10 ~]# kubectl get kubeadmcontrolplane NAME CLUSTER INITIALIZED .. REPLICAS .. UPDATED UNAVAILABLE AGE VERSION byoh-cluster-control-plane byoh-cluster true .. 1 .. 0 1 16h v1.26.2 ``` First a new control plane is provisioned with the new version. Only when it is fully up, is the old control plane deleted: ```bash [root@basecm10 ~]# kubectl get machine NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION byoh-clus.. byoh-cluster node001 byoh://node005/gije0o Running 125m v1.26.1 byoh-clus.. byoh-cluster Provisioning 7s v1.26.2 byoh-clus.. byoh-cluster node002 byoh://node004/98s8gu Running 98m v1.26.1 ``` The deletion is not carried out immediately. There is a grace period in which both control planes run. The old control plane is deleted shortly afterwards. ```bash [root@basecm10 ~]# kubectl get machine NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION byoh-clus.. byoh-cluster node001 byoh://node005/gije0o Running 127m v1.26.1 byoh-clus.. byoh-cluster node003 byoh://node003/mjvvvg Running 2m30s v1.26.2 byoh-clus.. byoh-cluster node004 byoh://node004/98s8gu Running 101m v1.26.1 ``` If there is than one control plane, then a rolling upgrade takes place, one control plane at a time. #### 8.3.4.4 Upgrading The Workers The following session shows the workers being upgraded from version `v1.26.1` to `v1.26.2`. **Example** ```bash [root@basecm10 ~]# kubectl get machinedeployment NAME CLUSTER REPLICAS .. UPDATED UNAVAILABLE PHASE AGE VERSION byoh-cluster-md-0 byoh-cluster 3 .. 3 3 ScalingUp 17h v1.26.1 ``` In this case there are three workers. The version is patched with: **Example** ```bash [root@basecm10 ~]# kubectl patch machinedeployment byoh-cluster-md-0 \ --type=merge \ -p '{"spec": {"template": {"spec": {"version": "v1.26.2"}}}}' machinedeployment.cluster.x-k8s.io/byoh-cluster-md-0 patched ``` The resource immediately shows the new desired version. However, the upgrade is not performed immediately: **Example** ```bash [root@basecm10 ~]# kubectl get machinedeployment NAME CLUSTER REPLICAS .. UPDATED UNAVAILABLE PHASE AGE VERSION byoh-cluster-md-0 byoh-cluster 4 .. 1 4 ScalingUp 17h v1.26.2 ``` First a new worker is provisioned with the new version. Only when it is fully up, is the old worker deleted: ```bash [root@basecm10 ~]# kubectl get machine NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION byoh-clus.. byoh-cluster node003 byoh://node003/mjvvvg Running 5m45s v1.26.2 byoh-clus.. byoh-cluster node004 byoh://node004/98s8gu Running 104m v1.26.1 byoh-clus.. byoh-cluster node002 byoh://node002/f3j7mt Running 107m v1.26.1 byoh-clus.. byoh-cluster node006 byoh://node006/paityf Running 139m v1.26.1 byoh-clus.. byoh-cluster Provisioning 7s v1.26.2 ``` The process repeats itself until all the workers are upgraded. ## 8.4 BCM Host Agent Registration The registration process in figure [8.8](#fig-byoh-bootstrap-flow) is sourced from the documentation at the upstream BYOH project: [https://github.com/vmware-tanzu/cluster-api-provider-bringyourownhost/blob/main/docs/](https://github.com/vmware-tanzu/cluster-api-provider-bringyourownhost/blob/main/docs/) ![BYOH bootstrap flow](/base-command-manager/product-manuals/_fern-img/9eb5f9eb790b0d34340dcb6f3b55124c34c280fff20c48fabc7e65376e352774.webp) *Figure 8.8: BYOH bootstrap flow* The roles of the platform operator and site operator in the illustration are automated by BCM. The process initiated by BCM CAPI Infrastructure Provider starts with BCM. When initiating, the capi-agent service verifies the existence of a configuration before launching. If a configuration does not exist, then the BCM API is used to request one. BCM then automates all the steps necessary to generate this configuration, and prepares it for the `capi-agent` so it can start its process. To clarify figure [8.8](#fig-byoh-bootstrap-flow) further, before step 9 (create a new kubeconfig file at `~/.kube/config`), only the *bootstrap* configuration exists on the node. However, after this step, *two* configuration files exist: the *bootstrap* and the *definitive* configuration files. The presence of these files enables BCM to determine the current phase of the bootstrap process that the `capi-agent` is in. Manual generation of bootstrap configurations is also possible. Details on this are given in the documentation at [https://github.com/vmware-tanzu/cluster-api-provider-bringyourownhost/blob/main/docs/getting_started.md#register-byoh-host-to-management-cluster](https://github.com/vmware-tanzu/cluster-api-provider-bringyourownhost/blob/main/docs/getting_started.md#register-byoh-host-to-management-cluster) The host agent registration process described in the preceding is part of what is carried out for nodes when running the `Assign Capi Role` option in section [8.2.2](#822-the-assign-capi-role-option). ```bash [root@basecm10 ~]# kubectl get byohost -A NAMESPACE NAME OSNAME OSIMAGE ARCH default node001 linux Rocky Linux 8.7 (Green Obsidian) amd64 default node002 linux Rocky Linux 8.7 (Green Obsidian) amd64 default node003 linux Rocky Linux 8.7 (Green Obsidian) amd64 default node004 linux Rocky Linux 8.7 (Green Obsidian) amd64 default node005 linux Rocky Linux 8.7 (Green Obsidian) amd64 default node006 linux Rocky Linux 8.7 (Green Obsidian) amd64 ``` ## 8.5 Install Process BCM CAPI The preceding section (section [8.4](#84-bcm-host-agent-registration)) focused on BCM CAPI host agent registration. The current section (section [8.5](#85-install-process-bcm-capi)) discusses the deployment of Kubernetes clusters on these CAPI Hosts. The deployment uses installation scripts transmitted to the designated nodes through “installation secrets”. Node registration with BCM is also carried out as part of the deployment of Kubernetes on the CAPI hosts, and is distinct from the host agent registration of section [8.4](#84-bcm-host-agent-registration). The deployment process flow is illustrated in figure [8.9](#fig-byoh-installer-flow): ![BYOH installer flow](/base-command-manager/product-manuals/_fern-img/025230bb0f8411049211b637c998704c21abc46f25a06c8adb4ffb7a54700cfe.webp) *Figure 8.9: BYOH installer flow* ### 8.5.1 Registration Process Of The Node With BCM With BCM CAPI Infrastructure Provider, the installation script is a `pythoncm` script. It establishes communication with BCM on the active Head Node. This is referred to as “registering” the node with BCM. The registration process involves essential bookkeeping tasks, such as tracking which nodes belong to specific Kubernetes CAPI clusters, and preparing software images with the appropriate Kubernetes versions. The example that follows illustrates various components involved in creating a Kubernetes cluster using CAPI. In the example, a single head node operates as the Kubernetes management cluster, and multiple nodes are assigned the CAPI role. A cluster definition is generated for a control plane node and a worker node using the procedure described in section [8.3](#83-deploying-a-kubernetes-cluster-through-capi). An overview of the components involved when the cluster definition is applied is seen in figure [8.10](#fig-capi-summary): ![CAPI summary](/base-command-manager/product-manuals/_fern-img/8832677a715f98f4de60936b47656150c8fcd9301eb393f0c2d8853d850963a4.webp) *Figure 8.10: CAPI summary* In figure [8.10](#fig-capi-summary), the Kubernetes CAPI cluster has the control plane deployed on `node001`, followed by the deployment of the worker node software image on `node002`. The diagram of the is expanded from the `node001` CAPI agent block illustrates the steps executed by the `node001` `capi-agent`. Upon registering with the BCM API, the more important actions that BCM carries out are: * **Step 1:** Preparation of Kubernetes version 1.26.0 within the software image for the registering node. * **Step 2:** Provisioning of the node through an image update, for example using the `imageupdate` command of `cmsh` ([section 5.6.2 of the *Administrator Manual*](/admin/provisioning#562-updating-running-nodes-with-cmsh-using-imageupdate)), to ensure that it stays synchronized with its software image. * **Step 3:** Handing control back to the `capi-agent`, which proceeds with the subsequent step in the CAPI cluster creation process. The steps also include processes beside the ones illustrated in figure [8.10](#fig-capi-summary). The more complete sequence is: * **Step 1:** The installation script is invoked with contextual information, such as the desired Kubernetes version (for example: 1.26.0) and the cluster name. * **Step 2:** The installation script, implemented as a PythonCM script, establishes communication with BCM. It registers itself with the active head node and remains in a waiting state until notified to terminate. * **Step 3:** BCM creates a mapping that specifies which node should be provisioned with the corresponding Kubernetes version. * **Step 4:** BCM ensures that the software image for the node contains the requested Kubernetes version. * **Step 5:** BCM ensures that the node is provisioned with its designated software image. * **Step 6:** BCM signals to the node that the installation script has completed. From this point onward, the default logic for the BCM CAPI operator takes over, and `kubeadm` initializes the node accordingly. ### 8.5.2 Creating A Kubernetes Cluster Via CAPI The steps in section [8.5.1](#851-registration-process-of-the-node-with-bcm), are about the node registration process and node provisioning during Kubernetes cluster creation with CAPI. The following steps zoom out further, and describe the complete process of creating a Kubernetes cluster through CAPI. * **Step 1:** The system administrator defines a cluster to be deployed by the BCM CAPI operator, and feeds it to the Kubernetes API server using `kubectl`. * **Step 2:** The definition results in a number of Kubernetes resources being created, such as a `Cluster`, `ByoCluster`, `MachineDeployment`. * **Step 3:** The BCM CAPI operator responds to these newly-created resources. * **Step 4:** CAPI starts assigning CAPI agents to reconcile specific `Machine`s, beginning with the first control plane. This is indicated by the red provisioning line, numbered with a `1`. * **Step 5:** BCM prepares a module file and kubeconfig for the new Kubernetes CAPI cluster, and writes these to disk for the system administrator. * **Step 6:** BCM CAPI operator updates its records using the newly-created control plane and proceeds with provisioning the additional control planes or workers, such as the additional worker. This is indicated by the red line numbered with a `2`. ## 8.6 Configuring CAPI Versions In Software Images BCM takes care of configuring CAPI versions in software images automatically. This is part of its BCM CAPI node registration process, as mentioned earlier in section [8.5](#85-install-process-bcm-capi). An alternative is to manually pre-install a specific version of Kubernetes in a software image. This can be done from within the `softwareimage` mode of `cmsh`: ```bash [root@basecm10 ~]# cmsh [basecm10]% softwareimage [basecm10->softwareimage]% use default-image [basecm10->softwareimage[default-image]]% help capi Name: capi - Manage Kubernetes CAPI versions on the image ``` ```bash Usage: capi [OPTIONS] list capi [OPTIONS] add [ ...] capi [OPTIONS] remove [ ...] capi [OPTIONS] clear Options: -v, --verbose Be more verbose -d, --delimiter Use as delimiter between columns. Use {} for JSON, and {} for JSON with a specific indentation. --image, -i Perform action on comma separated list of images --repo-refresh, -r Refresh the repository cache before adding new versions --debug Run script with debug on Examples: capi list List CAPI versions on all / current image capi clear Remove all CAPI versions on all / current image capi add 1.26.0 1.27.* Add specified versions on all / current image ``` **Example** ```bash [basecm10->softwareimage[default-image]]% capi list Node image versions Result Error ------------- ------------------------- ------------------------------- -------- ----- basecm10 /cm/images/default-image 1.24.9, 1.23.0, 1.26.1, 1.26.2 good [basecm10->softwareimage[default-image]]% capi add 1.27.* Node image versions Result Error ------------- ------------------------- ------------------------------------------------------- -------- ----- basecm10 /cm/images/default-image 1.24.9, 1.23.0, 1.26.1, 1.26.2, 1.27.0, 1.27.1, 1.27.2 good ``` ## 8.7 Removing Kubernetes CAPI clusters The following steps remove the CAPI clusters: * **Step 1:** The removal of the CAPI clusters is started with the `kubectl delete` command: ```bash [root@basecm10 ~]# kubectl delete -f cluster.yaml ... ``` * **Step 2:** If all CAPI hosts are not part of any significant clusters, then the configuration overlay configuration overlays are removed: ```bash [root@basecm10 ~]# cmsh [basecm10]% configurationoverlay [basecm10->configurationoverlay]% remove capi-mgmt [basecm10->configurationoverlay*]% commit Successfully removed 1 ConfigurationOverlays Successfully committed 0 ConfigurationOverlays ``` The removal of the configuration overlays causes `containerd` and the CAPI agents to stop on the hosts. * **Step 3:** For additional cleanliness, the bootstrap configurations for each of the hosts should be removed: ```bash [root@basecm10 ~]# kubectl delete bootstrapkubeconfig --all bootstrapkubeconfig.infrastructure.cluster.x-k8s.io "bootstrap-kubeconfig-node001" deleted bootstrapkubeconfig.infrastructure.cluster.x-k8s.io "bootstrap-kubeconfig-node002" deleted bootstrapkubeconfig.infrastructure.cluster.x-k8s.io "bootstrap-kubeconfig-node003" deleted bootstrapkubeconfig.infrastructure.cluster.x-k8s.io "bootstrap-kubeconfig-node004" deleted bootstrapkubeconfig.infrastructure.cluster.x-k8s.io "bootstrap-kubeconfig-node005" deleted bootstrapkubeconfig.infrastructure.cluster.x-k8s.io "bootstrap-kubeconfig-node006" deleted ``` * **Step 4:** The infrastructure provider is then removed: ```bash [root@basecm10 ~]# module load capi/1.3.0 [root@basecm10 ~]# clusterctl delete --infrastructure byoh Deleting Provider="infrastructure-byoh" Version="" Namespace="byoh-system" ``` * **Step 5:** The removal of all the ByoHost resources can be checked: ```bash [root@basecm10 ~]# kubectl get byohost No resources found ``` For clean-up the command `kubectl delete byohost ...` can be used. * **Step 6:** Finally, the Kubernetes CAPI clusters is eliminated from BCM itself: ```bash [root@basecm10 ~]# cmsh [basecm10]% kubernetes [basecm10->kubernetes]% remove byoh-cluster [basecm10->kubernetes*]% commit [basecm10->kubernetes]% ``` ## 8.8 Kubernetes CAPI Templates In BCM, each Kubernetes cluster is represented as a `KubeCluster` entity stored in the BCM database. These entities can be viewed in the `kubernetes` submode of `cmsh`: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% kubernetes [basecm10->kubernetes]% list Name (key) ------------------- default ``` * For every Kubernetes management cluster, a default KubeCluster CAPI template is generated once the CAPI role node is assigned using the wizard, as discussed in section [8.2.2](#822-the-assign-capi-role-option). This template serves as a base for all subsequent Kubernetes clusters created via CAPI. * For example, if an administrator creates a cluster `my-capi-cluster` via CAPI, for the Kubernetes management cluster `mgmt`, then BCM clones the `capi-mgmt-template` KubeCluster entity to create a new KubeCluster `my-capi-cluster`. * At the time of CAPI cluster creation, the KubeCluster template can also be customized using an annotation in the Cluster resource definition: **Example** ```bash apiVersion: cluster.x-k8s.io/v1beta1 kind: Cluster metadata: labels: cni: byoh-cluster-crs-0 crs: "true" infrav1.nvidia.x-k8s.io/capi-kube-template: "my-capi-cluster-template" name: byoh-cluster namespace: default spec: clusterNetwork: pods: cidrBlocks: - 192.168.0.0/16 serviceDomain: cluster.local services: cidrBlocks: - 10.128.0.0/12 controlPlaneRef: apiVersion: controlplane.cluster.x-k8s.io/v1beta1 kind: KubeadmControlPlane name: byoh-cluster-control-plane infrastructureRef: apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: ByoCluster name: byoh-cluster ``` The provided YAML illustrates a cluster `byoh-cluster` definition, with the specified capi-template set to `my-capi-cluster-template`. If present, then this is used instead of the default. * The fallback sequence for the CAPI template is: The specified label (for example: `my-capi-cluster-template`) is used if it exists; otherwise the default `capi-mgmt-template` is used. If neither is available, then `capi-template` is used.