Kubernetes Node Setup

This section addresses configuration steps to be performed on the BCM head node.

1. Set the IP addresses for the interfaces on knode-01 and ensure that the network device names match the site survey.

   [bcm10-headnode]% device
   [bcm10-headnode->device]% use knode-01
   [bcm10-headnode->device[knode-01]]% interfaces
   [bcm10-headnode->device[knode-01]->interfaces]% set ipmi0 ip 10.133.3.25
   [bcm10-headnode->device*[knode-01*]->interfaces*]% set bond0 ip 10.133.4.25
   [bcm10-headnode->device*[knode-01*]->interfaces*]% set ens3f1np1 networkdevicename ens1f1np1
   Found bond0 member interface ens3f1np1
   [bcm10-headnode->device*[knode-01*]->interfaces*]% commit
   

2. Create the other knodes by cloning the first one.

   [bcm10-headnode]% device
   [bcm10-headnode->device]% foreach --clone knode-01 -n knode-02..knode-03 --next-ip ()
   [bcm10-headnode->device*]% commit
   Successfully committed 2 Devices
   

3. Now as with the DGX nodes, assign the MAC addresses for each of the interfaces and the node itself by the site survey.

   Repeat this for all 3 knodes.

   [bcm10-headnode]% device
   [bcm10-headnode->device]% use knode-01
   [bcm10-headnode->device[knode-01]]% interfaces
   [bcm10-headnode->device[knode-01]->interfaces]% set enp65s0np0 mac 08:C0:EB:F5:71:87
   [bcm10-headnode->device*[knode-01*]->interfaces*]% set ens1f1np1 mac 08:C0:EB:F5:74:F7
   [bcm10-headnode->device*[knode-01*]->interfaces*]% exit
   [bcm10-headnode->device*[knode-01*]]% set mac 08:C0:EB:F5:71:87
   [bcm10-headnode->device*[knode-01*]]% commit
   

4. The knodes will need to be differentiated from the default image.

5. Create a new softwareimage for the k8s control nodes.

   [bcm10-headnode]% softwareimage
   [bcm10-headnode->softwareimage]% clone default-image k8s-control-plane-image
   [bcm10-headnode->softwareimage*[k8s-control-plane-image*]]% commit
   [bcm10-headnode->softwareimage[k8s-control-plane-image]]% kernelmodules
   [bcm10-headnode->softwareimage[k8s-control-plane-image]->kernelmodules]% add mlx5_core
   [bcm10-headnode->softwareimage*[k8s-control-plane-image*]->kernelmodules*[mlx5_core*]]% commit
   

6. Create a new category for the k8s control nodes.

   [bcm10-headnode]% category
   [bcm10-headnode->category]% clone default k8s-control-plane
   [bcm10-headnode->category*[k8s-control-plane*]]% commit
   [bcm10-headnode->category[k8s-control-plane]]% set softwareimage k8s-control-plane-image
   [bcm10-headnode->category*[k8s-control-plane*]]% commit
   

7. Exit cmsh and go back to the root shell. Save the following XML code to a new file /cm/local/apps/cmd/etc/htdocs/disk-setup/k8s-disksetup.xml.

   <?xml version="1.0" encoding="ISO-8859-1"?>
   <diskSetup xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
       <device>
   
           <blockdev>/dev/nvme0n1</blockdev>
   
           <partition id="a0" partitiontype="esp">
           <size>100M</size>
           <type>linux</type>
           <filesystem>fat</filesystem>
           <mountPoint>/boot/efi</mountPoint>
           <mountOptions>defaults,noatime,nodiratime</mountOptions>
           </partition>
           <partition id="a1">
           <size>max</size>
           <type>linux</type>
           <filesystem>xfs</filesystem>
           <mountPoint>/</mountPoint>
           <mountOptions>defaults,noatime,nodiratime</mountOptions>
           </partition>
       </device>
   </diskSetup>
   

8. Go back into cmsh and assign the disk layout to the k8s control nodes.

   # cmsh
   [bcm10-headnode]% category
   [bcm10-headnode->category]% use k8s-control-plane
   [bcm10-headnode->category[k8s-control-plane]]% set disksetup /cm/local/apps/cmd/etc/htdocs/disk-setup/k8s-disksetup.xml
   [bcm10-headnode->category*[k8s-control-plane*]]% commit
   

9. Assign the new category to the knodes.

   [bcm10-headnode]% device
   [bcm10-headnode->device]% foreach -n knode-01..knode-03 (set category k8s-control-plane)
   [bcm10-headnode->device*]% commit
   Successfully committed 3 Devices
   

10. Power on and provision the k8s control nodes.