> This page is for version 10. > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Node Provisioning This chapter covers *node provisioning*. Node provisioning is the process of how nodes obtain an image. Typically, this happens during their stages of progress from power-up to becoming active in a cluster, but node provisioning can also take place when updating a running node. Section [5.1](#51-before-the-kernel-loads) describes the stages leading up to the loading of the kernel onto the node. Section [5.2](#52-provisioning-nodes) covers configuration and behavior of the provisioning nodes that supply the software images. Section [5.3](#53-the-kernel-image-ramdisk-and-kernel-modules) describes the configuration and loading of the kernel, the ramdisk, and kernel modules. Section [5.4](#54-node-installer) elaborates on how the node-installer identifies and places the software image on the node in a 13-step process. Section [5.5](#55-node-states) explains node states during normal boot, as well node states that indicate boot problems. Section [5.6](#56-updating-running-nodes) describes how running nodes can be updated, and modifications that can be done to the update process. Section [5.7](#57-adding-new-nodes) explains how to add new nodes to a cluster so that node provisioning will work for these new nodes too. The Base View and `cmsh` front ends for creating new node objects and properties in CMDaemon are described. Section [5.8](#58-troubleshooting-the-node-boot-process) describes troubleshooting the node provisioning process. ## 5.1 Before The Kernel Loads Immediately after powering up a node, and before it is able to load up the Linux kernel, a node starts its boot process in several possible ways: ### 5.1.1 PXE Booting By default, nodes boot from the network when using BCM. This is called a *network boot*. On the x86 architectures it is known as a *PXE boot* (often pronounced as “pixie boot”). It is recommended as a BIOS setting for nodes. The head node runs a tftpd server that is managed by `systemd`. The tftpd server supplies the boot loader from within the default software image (section [2.1.2](/admin/clustermanagement#212-software-images)) offered to nodes. The boot loader runs on the node and displays a menu (figure [5.1](#fig-PXEbootmenuoptions)) based on loading a menu module within a configuration file. The default configuration files offered to nodes are located under `/tftpboot/pxelinux.cfg/` on the head node. To implement changes in the files, CMDaemon may need to be restarted, or the `updateprovisioners` command ([here](#5244-provisioning-role-change-notification-with-updateprovisioners)) can be run. ![PXE boot menu options](/base-command-manager/product-manuals/_fern-img/136cab139eff407d829f495e024426c6ad89c696884883eefff828987b6e58d3.webp) *Figure 5.1: PXE boot menu options* The default configuration files give instructions to the menu module of `PXElinux`. The instruction set used is documented at [http://www.syslinux.org/wiki/index.php/Comboot/menu.c32](http://www.syslinux.org/wiki/index.php/Comboot/menu.c32), and includes the `TIMEOUT`, `LABEL`, `MENU LABEL`, `DEFAULT`, and `MENU DEFAULT` instructions. #### 5.1.1.1 The PXE `TIMEOUT` Instruction During the display of the PXE boot menu, a selection can be made within a timeout period to boot the node in a several ways. Among the options are some of the install mode options (section [5.4.4](#544-determining-install-mode-type-and-execution-mode)). If no selection is made by the user within the timeout period, then the `AUTO` install mode option is chosen by default. In the PXE menu configuration files under `pxelinux.cfg/`, the default timeout of 5 seconds can be adjusted by changing the value of the “`TIMEOUT 50`” line. This value is specified in deciseconds. **Example** ```bash TIMEOUT 300 # changed timeout from 50 (=5 seconds) ``` #### 5.1.1.2 The PXE `LABEL` And `MENU LABEL` Instructions **LABEL:** The menu configuration files under `pxelinux.cfg/` contain several multiline `LABEL` statements. Each `LABEL` statement is associated with a kernel image that can be loaded from the PXE boot menu along with appropriate kernel options. Each `LABEL` statement also has a text immediately following the `LABEL` tag. Typically the text is a description, such as `linux`, `main`, `RESCUE`, and so on. If the PXE menu module is not used, then tab completion prompting displays the list of possible text values at the PXE boot prompt so that the associated kernel image and options can be chosen by user intervention. **MENU LABEL:** By default, the PXE menu module is used, and by default, each `LABEL` statement also contains a `MENU LABEL` instruction. Each `MENU LABEL` instruction also has a text immediately following the `MENU LABEL` tag. Typically the text is a description, such as `AUTO`, `RESCUE` and so on (figure [5.1](#fig-PXEbootmenuoptions)). Using the PXE menu module means that the list of the `MENU LABEL` text values is displayed when the PXE boot menu is displayed, so that the associated kernel image and options can conveniently be selected by user intervention. #### 5.1.1.3 The PXE `DEFAULT` And `MENU DEFAULT` Instructions **DEFAULT:** If the PXE menu module is not used and if no `MENU` instructions are used, and if there is no user intervention, then setting the same text that follows a `LABEL` tag immediately after the `DEFAULT` instruction, results in the associated kernel image and its options being run by default after the timeout. By default, as already explained, the PXE menu module is used. In particular it uses the setting: `DEFAULT menu.c32` to enable the menu. **MENU DEFAULT:** If the PXE menu module is used and if `MENU` instructions are used, and if there is no user intervention, then setting a `MENU DEFAULT` tag as a line within the multiline `LABEL` statement results in the kernel image and options associated with that `LABEL` statement being loaded by default after the timeout. #### 5.1.1.4 The CMDaemon `PXE Label` Setting For Specific Nodes The `MENU DEFAULT` value by default applies to every node using the software image that the PXE menu configuration file under `pxelinux.cfg/` is loaded from. To override its application on a per-node basis, the value of `PXE Label` can be set for each node. * Some simple examples of overriding the default `MENU DEFAULT` value are as follows: * For example, using `cmsh`: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% device use node001 [basecm10->device[node001]]% set pxelabel MEMTEST ; commit ``` Carrying it out for all nodes in the `default` category can be done, for example, with: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% device [basecm10->device]% foreach -c default (set pxelabel MEMTEST) [basecm10->device*]% commit ``` The value of `pxelabel` can be cleared with: **Example** ```bash [root@basecm10 ~]# cmsh -c "device; foreach -c default (clear pxelabel); commit" ``` * In Base View, the PXE label can be set for a node `node001` using the navigation path `Devices` > `Nodes` > `Physical Node list[node001]` > `Settings[Physical Node node001]` >\ `Provisioning[PXE Label]` which leads to a screen as in (figure [5.2](#fig-bright-view-PXElabeloption)): ![Base View PXE Label option](/base-command-manager/product-manuals/_fern-img/82e37cd6188b655ca4a1f189a760821c514b47c688550db1c9397367722e7b62.webp) *Figure 5.2: Base View PXE Label option* * A more complicated example of overriding the default `MENU DEFAULT` value now follows. Although it helps in understanding how PXE labels can be used, it can normally be skipped because the use case for it is unlikely, and the details are involved. In this example, `pxelabel` is set by the administrator via Base View or `cmsh` to `localdrive`. This will then set the node to boot from the first local drive and not the node-installer. This is a setting that is discouraged since it usually makes node management harder, but it can be used by administrators who do not wish to answer any prompt during node boot, and also want the node drives to have no risk of being overwritten by the actions of the node-installer, and also want the system to be up and running quite fully, even if not necessarily provisioned with the latest image from the head node. Here, the overwriting-avoidance method relies on the nodes being associated with a configuration file under `pxelinux.cfg` at the time that the `localdrive` setting is done. However, nodes that are unidentified, or are identified later on, will have their `MENU DEFAULT` value still set to a default `pxelabel` value set in the files under `/tftpboot/pxelinux.cfg/`, which is the value `linux` by default, and which is associated with a code block in that file with the label `LABEL linux`. To make such (as yet) unidentified nodes boot to a `localdrive` setting instead, requires modifying the files under `/tftpboot/pxelinux.cfg/`, so that the `MENU DEFAULT` line is associated with the code block of `LABEL localdrive` rather than the code block of `LABEL linux`. There are two methods other than using the preceding `pxelabel` method to deal with the risk of overwriting. Unlike the `pxelabel` method however, these methods can interrupt node booting, so that the node does not progress to being fully up until the administrator takes further action: 1. If it is acceptable that the administrator manually enters a confirmation as part of the boot process when a possible overwrite risk is found, then the `datanode` method (section [5.4.4.4](#5444-full-install-confirmation-via-datanode-setting)) can be used. 2. If it is acceptable that the boot process halts on detecting a possible overwrite risk, then the XML assertions method (Appendix [D.11](/admin/partitioning#d11-example-preventing-accidental-data-loss)) is recommended. #### 5.1.1.5 Changing The Install Mode Or Default Image Offered To Nodes The selections offered by the PXE menu are pre-configured by default so that the `AUTO` menu option by default loads a kernel, runs the `AUTO` install mode, and eventually the `default-image` software image is provisioned. Normally administrators should not be changing the install mode, kernel, or kernel options in the PXE menu configuration files under `pxelinux.cfg/`. More on changing the install mode is given in section [5.4.4](#544-determining-install-mode-type-and-execution-mode). More on changing software images, image package management, kernels, and kernel options, is to be found in Chapter [11](/admin/softwareimagemanagement). ### 5.1.2 iPXE Booting From A Disk Drive Also by default, on disked nodes, iPXE software is placed on the drive during node installation. If the boot instructions from the BIOS for PXE booting fail, and if the BIOS instructions are that a boot attempt should then be made from the hard drive, it means that a PXE network boot attempt is done again, as instructed by the bootable hard drive. This can be a useful fallback option that works around certain BIOS features or problems. ### 5.1.3 iPXE Booting Using InfiniBand On clusters that have InfiniBand hardware, it is normally used for data transfer as a service after the nodes have fully booted up (section [3.6](/admin/configuring#36-configuring-infiniband-interfaces)). InfiniBand can also be used for PXE booting (described here) and used for node provisioning (section [5.3.3](#533-infiniband-provisioning)). However these uses are not necessary, even if InfiniBand is used for data transfer as a service later on, because booting and provisioning is available over Ethernet by default. This section (about boot over InfiniBand) may therefore safely be skipped when first configuring a cluster. Booting over InfiniBand via PXE is enabled by carrying out these 3 steps: 1. Making BCM aware that nodes are to be booted over InfiniBand. Node booting (section [3.2.3.4](/admin/configuring#3234-changing-internal-network-parameters-for-the-cluster), [here](/admin/configuring#3234-changing-internal-network-parameters-for-the-cluster)) can be set from `cmsh` or Base View as follows: 1. From `cmsh`’s `network` mode: If the InfiniBand network name is `ibnet`, then a `cmsh` command that will set it is:\ `cmsh -c "network; set ibnet nodebooting yes; commit"` 2. From Base View: The `Settings` window for the InfiniBand network, for example `ibnet`, can be accessed from the `Networking` resource via the navigation path `Networking` > `Networks[ibnet]` > `Edit` > `Settings` (this is similar to figure [3.5](/admin/configuring#321-configuring-networks), but for `ibnet`). The `Node booting` option for `ibnet` is then enabled and saved. If the InfiniBand network does not yet exist, then it must be created (section [3.2.2](/admin/configuring#322-adding-networks)). The recommended default values used are described in section [3.6.3](/admin/configuring#363-infiniband-network-settings). The MAC address of the interface in CMDaemon defaults to using the GUID of the interface. The administrator should also be aware that the interface from which a node boots, (conveniently labeled `BOOTIF`), must not be an interface that is already configured for that node in CMDaemon. For example, if `BOOTIF` is the device `ib0`, then `ib0` must not already be configured in CMDaemon. Either `BOOTIF` or the `ib0` configuration should be changed so that node installation can succeed. It is recommended to set `BOOTIF` to `eth0` if the `ib0` device should exist. 2. Flashing iPXE onto the InfiniBand HCAs. (The ROM image is obtained from the HCA vendor). 3. Configuring the BIOS of the nodes to boot from the InfiniBand HCA. All MAC addresses become invalid for identification purposes when changing from booting over Ethernet to booting over InfiniBand. Administrators who enable iPXE booting almost always wish to provision over InfiniBand too. Configuring provisioning over InfiniBand is described in section [5.3.3](#533-infiniband-provisioning). ### 5.1.4 Using PXE To Boot From The Drive Besides PXE booting from only the network, a node can also be configured via PXE to step over to using its own drive to start booting and get to the stage of loading up its kernel entirely from its drive, just like a normal standalone machine. This can be done by setting `PXE LABEL` to `localdrive` ([here](#511-pxe-booting)). ### 5.1.5 Network Booting Without PXE On The ARMv8 Architecture ARMv8 nodes use a network boot implementation that differs slightly from the x86 PXE boot implementation. The actual firmware that starts up on ARMv8 nodes depends on the environment the hardware is in. Networking is then started by the firmware and the network requests what to boot. The head node however then sends out a GRUB binary instead of an iPXE binary. The GRUB binary then runs on the regular node, and fetches the kernel and initrd via TFTP. The node-installer then runs on the nodes and follows the same steps as in the x86 process. ### 5.1.6 Network Booting Protocol The protocol used by network booting is set with the parameter `bootloaderprotocol`. It is set to HTTP by default at category level: **Example** ```bash [basecm10->category[default-centos7-x86_64]]% get bootloaderprotocol HTTP ``` It can be modified at category or node level, to one of the values `HTTP`, `HTTPS`, or `TFTP`: **Example** ```bash [basecm10->device[node001]]% get bootloaderprotocol HTTP (default-centos7-x86_64) [basecm10->device[node001]]% set bootloaderprotocol http https tftp [basecm10->device[node001]]% set bootloaderprotocol tftp [basecm10->device*[node001*]]% commit [basecm10->device[node001]]% get bootloaderprotocol TFTP ``` The HTTPS protocol for node booting should almost never be used, because it is rarely implemented in hardware. ### 5.1.7 The Boot Role The action of providing a boot image to a node via DHCP and TFTP is known as providing *node booting*. Node provisioning (section [5.2](#52-provisioning-nodes)), on the other hand, is about provisioning the node with the rest of the node image. Roles in general are introduced in section [2.1.5](/admin/clustermanagement#215-roles). The *boot role* is one such role that can be assigned to a regular node. The boot role configures a regular node so that it can then provide node booting. The role cannot be assigned or removed from the head node—the head node always has a boot role. The boot role is assigned by administrators to regular nodes if there is a need to cope with the scaling limitations of TFTP and DHCP. TFTP and DHCP services can be overwhelmed when there are large numbers of nodes making use of them during boot. An example of the scaling limitations may be observed, for example, when, during the powering up and network booting attempts of a large number of regular nodes from the head node, it turns out that random different regular nodes are unable to boot, typically due to network effects. One implementation of boot role assignment might therefore be, for example, to have a several groups of racks, with each rack in a subnet, and with one regular node in each subnet that is assigned the boot role. The boot role regular nodes would thus take the DHCP and TFTP load off the head node and onto themselves for all the nodes in their associated subnet, so that all nodes of the cluster are then able to boot without networking issues. ## 5.2 Provisioning Nodes The action of transferring the software image to the nodes is called *node provisioning*, and is done by special nodes called the *provisioning nodes*. More complex clusters can have several provisioning nodes configured by the administrator, thereby distributing network traffic loads when many nodes are booting. Creating provisioning nodes is done by assigning a *provisioning role* to a node or category of nodes. Similar to how the head node always has a boot role (section [5.1.7](#517-the-boot-role)), the head node also always has a provisioning role. ### 5.2.1 Provisioning Nodes: Configuration Settings The provisioning role has several parameters that can be set:
**Property** **Description**
`allImages` The following values decide what images the provisioning node provides: \ • onlocaldisk (the default): all images on the local disk, regardless of any other parameters set \ • onlocaldiskexceptsharedimages: all images on the local disk, except for shared images \ • onsharedstorage: all images on the shared storage, regardless of any other parameters set \ • no: only images listed in the localimages or sharedimages parameters, described next
`localimages` A list of software images on the local disk that the provisioning node accesses and provides. The list is used only if `allImages` is “ `no` ”.
`sharedimages` A list of software images on the shared storage that the provisioning node accesses and provides. The list is used only if `allImages` is “ `no` ”
`Provisioning slots` The maximum number of nodes that can be provisioned in parallel by the provisioning node. The optimum number depends on the infrastructure. The default value is 10, which is safe for typical cluster setups. Setting it lower may sometimes be needed to prevent network and disk overload.
`nodegroups` A list of node groups (section [2.1.4](/admin/clustermanagement#214-node-groups) ). If set, the provisioning node only provisions nodes in the listed groups. Conversely, nodes in one of these groups can only be provisioned by provisioning nodes that have that group set. Nodes without a group, or nodes in a group not listed in `nodegroups` , can only be provisioned by provisioning nodes that have no `nodegroups` values set. By default, the `nodegroups` list is unset in the provisioning nodes. The `nodegroups` setting is typically used to set up a convenient hierarchy of provisioning, for example based on grouping by rack and by groups of racks.
`localprovisioning` If set to `yes` (the default), then it speeds up the initial provisioning of cloud directors and cloud provisioning nodes ( [section 3.2 of the *Cloudbursting Manual*](/cloudbursting/cexcloud#32-cluster-extension-with-aws-cloud-director-startup-from-scratch) ) . Also if set to `yes` , if a software image is used as the rootfs of the provisioning node and is also to be used by that node to provision other cloud nodes, then during the initial FULL install ( [here](#5441-install-mode-values) ) that image is transferred only once to the provisioning node, instead of twice.
A provisioning node keeps a copy of all the images it provisions on its local drive, in the same directory as where the head node keeps such images. The local drive of a provisioning node must therefore have enough space available for these images, which may require changes in its disk layout. ### 5.2.2 Provisioning Nodes: Role Setup With `cmsh` In the following `cmsh` example the administrator creates a new category called `misc`. The default category `default` already exists in a newly installed cluster. The administrator then assigns the role called `provisioning`, from the list of available assignable roles, to nodes in the `misc` category. After the `assign` command has been typed in, but before entering the command, tab-completion prompting can be used to list all the possible roles. Assignment creates an association between the role and the category. When the `assign` command runs, the shell drops into the level representing the `provisioning` role. If the role called `provisioning` were already assigned, then the `use provisioning` command would drop the shell into the `provisioning` role, without creating the association between the role and the category. As an aside from the topic of provisioning, from an organizational perspective, other assignable roles include `monitoring`, `storage`, and `failover`. Once the shell is within the role level, the role properties can be edited conveniently. For example, the nodes in the `misc` category assigned the `provisioning` role can have `default-image` set as the image that they provision to other nodes, and have `20` set as the maximum number of other nodes to be provisioned simultaneously (some text is elided in the following example): **Example** ```bash [basecm10]% category add misc [basecm10->category*[misc*]]% roles [basecm10->category*[misc*]->roles]% assign provisioning [basecm10...*]->roles*[provisioning*]]% set allimages no [basecm10...*]->roles*[provisioning*]]% set localimages default-image [basecm10...*]->roles*[provisioning*]]% set provisioningslots 20 [basecm10...*]->roles*[provisioning*]]% show Parameter Value --------------------------------- --------------------------------- Name provisioning Revision Type ProvisioningRole Add services yes Provisioning slots 20 All images no Include revisions of local images yes Local provisioning yes Local images default-image Shared images Nodegroups Categories Racks Load weight 0.000000 [basecm10->category*[misc*]->roles*[provisioning*]]% commit [basecm10->category[misc]->roles[provisioning]]% ``` Assigning a `provisioning` role can also be done for an individual node instead, if using a category is deemed overkill: **Example** ```bash [basecm10]% device use node001 [basecm10->device[node001]]% roles [basecm10->device[node001]->roles]% assign provisioning [basecm10->device*[node001*]->roles*[provisioning*]]% ... ``` A role change configures a provisioning node, but does not directly update the provisioning node with images. After carrying out a role change, BCM runs the `updateprovisioners` command described in section [5.2.4](#524-provisioning-nodes-housekeeping) automatically, so that regular images are propagated to the provisioners. The propagation can be done by provisioners themselves if they have up-to-date images. CMDaemon tracks the provisioning nodes role changes, as well as which provisioning nodes have up-to-date images available, so that provisioning node configurations and regular node images propagate efficiently. Thus, for example, image update requests by provisioning nodes take priority over provisioning update requests from regular nodes. ### 5.2.3 Provisioning Nodes: Role Setup With Base View The provisioning configuration outlined in `cmsh` mode in section [5.2.2](#522-provisioning-nodes-role-setup-with-cmsh) can be done via Base View too, as follows: A `misc` category can be added via the navigation path\ `Grouping` > `Categories` > `Add` > `Settings` > \<*name*>\ Within the `Settings` tab, the node category should be given a name `misc` (figure [5.3](#fig-bright-view-provisioningnodecategoriesadding)), and saved: ![Base View: Adding A misc Category](/base-command-manager/product-manuals/_fern-img/0e1a7aefe7eb5bee3a01e3c4e32149224bfaea095df8a0c19cd11b231ee74f95.webp) *Figure 5.3: Base View: Adding A misc Category* The `Roles` window can then be opened from within the `JUMP TO` section of the settings pane. To add a role, the `Add` button in the `Roles` window is clicked. A scrollable list of available roles is then displayed, (figure [5.4](#fig-bright-view-provisioningnodecategoriesprovisioningroles)): ![Base View: Setting A provisioning Role](/base-command-manager/product-manuals/_fern-img/6711caf51d47b9acc66db6101963b25d92888f53d1bbe5941c60711dbba084ea.webp) *Figure 5.4: Base View: Setting A provisioning Role* After selecting a role, then navigating via the `Back` buttons to the `Settings` menu of figure [5.3](#fig-bright-view-provisioningnodecategoriesadding), the role can be saved using the `Save` button there. The role has properties which can be edited (figure [5.5](#fig-bright-view-provisioningnodecategoriesprovisioningroles-properties)): ![Base View: Configuring A provisioning Role](/base-command-manager/product-manuals/_fern-img/684a97030417aa41c026b5d26e30d98265722e414e9567ac3d39296c5c6e0cf5.webp) *Figure 5.5: Base View: Configuring A provisioning Role* For example: * the `Provisioning slots` setting decides how many images can be supplied simultaneously from the provisioning node * the `All images` setting decides if the role provides all images * the `Local images` setting decides what images the provisioning node supplies from local storage * the `Shared images` setting decides what images the provisioning node supplies shared storage. The settings can be saved with the `Save` button of figure [5.5](#fig-bright-view-provisioningnodecategoriesprovisioningroles-properties). The images offered by the provisioning role should not be confused with the software image setting of the `misc` category itself, which is the image the provisioning node requests for itself from the category. ### 5.2.4 Provisioning Nodes: Housekeeping The head node does housekeeping tasks for the entire provisioning system. Provisioning is done on request for all non-head nodes on a first-come, first-serve basis. Since provisioning nodes themselves, too, need to be provisioned, it means that to cold boot an entire cluster up quickest, the head node should be booted and be up first, followed by provisioning nodes, and finally by all other non-head nodes. Following this start-up sequence ensures that all provisioning services are available when the other non-head nodes are started up. Some aspects of provisioning housekeeping are discussed next: #### 5.2.4.1 Provisioning Node Selection When a node requests provisioning, the head node allocates the task to a provisioning node. If there are several provisioning nodes that can provide the image required, then the task is allocated to the provisioning node with the lowest number of already-started provisioning tasks. #### 5.2.4.2 Limiting Provisioning Tasks With `MaxNumberOfProvisioningThreads` Besides limiting how much simultaneous provisioning per provisioning node is allowed with `Provisioning Slots` (section [5.2.1](#521-provisioning-nodes-configuration-settings)), the head node also limits how many simultaneous provisioning tasks are allowed to run on the entire cluster. This is set using the `MaxNumberOfProvisioningThreads` directive in the head node’s CMDaemon configuration file, `/etc/cmd.conf`, as described in Appendix [C](/admin/cmdconf). #### 5.2.4.3 Provisioning Tasks Deferral and Failure A provisioning request is *deferred* if the head node is not able to immediately allocate a provisioning node for the task. Whenever an ongoing provisioning task has finished, the head node tries to re-allocate deferred requests. A provisioning request *fails* if an image is not transferred. 5 retry attempts at provisioning the image are made in case a provisioning request fails. A provisioning node that is carrying out requests, and which loses connectivity, has its provisioning requests remain allocated to it for 180 seconds from the time that connectivity was lost. After this time the provisioning requests fail. #### 5.2.4.4 Provisioning Role Change Notification With `updateprovisioners` The `updateprovisioners` command can be accessed from the `softwareimage` mode in `cmsh`. It can also be accessed from Base View, via the navigation path `Provisioning` > `Provisioning requests` > `Update provisioning nodes`. In the examples in section [5.2.2](#522-provisioning-nodes-role-setup-with-cmsh), changes were made to provisioning role attributes for an individual node as well as for a category of nodes. This automatically ran the `updateprovisioners` command. The `updateprovisioners` command runs automatically if CMDaemon is involved during software image changes or during a provisioning request. If on the other hand, the software image is changed outside of the CMDaemon front ends (Base View and `cmsh`), for example by an administrator adding a file by copying it into place from the bash prompt, then `updateprovisioners` should be run manually to update the provisioners. In any case, if it is not run manually, then by default it runs every midnight (UTC). The scheduling period can be adjusted with the `autoupdateperiod` setting: **Example** ```bash [root@basecm10 ]# cmsh [basecm10]% partition use base [basecm10->partition[base]]% provisioningsettings [basecm10->partition[base]->provisioningsettings]% get autoupdateperiod [basecm10->partition[base]->provisioningsettings]% 86400 [basecm10->partition[base]->provisioningsettings]% # is UTC epoch start time modulo 24 hours [basecm10->partition[base]->provisioningsettings]% # set UTC epoch start time modulo 18 hours: [basecm10->partition[base]->provisioningsettings]% set autoupdateperiod 64800 [basecm10->partition*[base*]->provisioningsettings*]% commit ``` When the default `updateprovisioners` is invoked manually, the provisioning system waits for all running provisioning tasks to end, and then updates all images located on any provisioning nodes by using the images on the head node. It also re-initializes its internal state with the updated provisioning role properties, i.e. keeps track of what nodes are provisioning nodes. The default `updateprovisioners` command, run with no options, updates all images. If run from `cmsh` with a specified image as an option, then the command only does the updates for that particular image. A provisioning node undergoing an image update does not provision other nodes until the update is completed. **Example** ```bash [basecm10]% softwareimage updateprovisioners Provisioning nodes will be updated in the background. Sun Dec 12 13:45:09 2010 basecm10: Starting update of software image(s)\ provisioning node(s). (user initiated). [basecm10]% softwareimage updateprovisioners [basecm10]% Sun Dec 12 13:45:41 2010 basecm10: Updating image default-image on prov\ isioning node node001. [basecm10]% Sun Dec 12 13:46:00 2010 basecm10: Updating image default-image on prov\ isioning node node001 completed. Sun Dec 12 13:46:00 2010 basecm10: Provisioning node node001 was updated Sun Dec 12 13:46:00 2010 basecm10: Finished updating software image(s) \ on provisioning node(s). ``` #### 5.2.4.5 Provisioning Role Draining And Undraining Nodes With `drain`, `undrain` The `drain` and `undrain` commands to control provisioning nodes are accessible from within the `softwareimage` mode of `cmsh`. If a node is put into a `drain` state, then all currently active provisioning requests continue until they are completed. However, the node is not assigned any further pending requests, until the node is put back into an `undrain` state. **Example** ```bash [basecm10->softwareimage]% drain -n master Nodes drained [basecm10->softwareimage]% provisioningstatus Provisioning subsystem status Pending request: node001, node002 Provisioning node status: + basecm10 Slots: 1 / 10 State: draining Active nodes: node003 Up to date images: default-image [basecm10->softwareimage]% provisioningstatus Provisioning subsystem status Pending request: node001, node002 Provisioning node status: + basecm10 Slots: 0 / 10 State: drained Active nodes: none Up to date images: default-image ``` To drain all nodes at once, the `--role` option can be used, with `provisioning` role as its value. All pending requests then remain in the queue, until the nodes are undrained again. **Example** ```bash [basecm10->softwareimage]% drain --role provisioning ...Time passes. Pending requests stay in the queue. Then admin undrains it... [basecm10->softwareimage]% undrain --role provisioning ``` #### 5.2.4.6 Provisioning Node Update Safeguards And `dirtyautoupdatetimeout` The `updateprovisioners` command is subject to safeguards that prevent it running too frequently. The minimum period between provisioning updates can be adjusted with a timeout parameter `dirtyautoupdatetimeout`, which has a default value of 300s. Exceeding the timeout does not by itself trigger an update to the provisioning node. When the head node receives a provisioning request, it checks if the last update of the provisioning nodes is more than the timeout period. If true, then an update is triggered to the provisioning node. The update is disabled if the `dirtyautoupdatetimeout` is set to zero (`false`). The parameter can be accessed and set within `cmsh` from `partition` mode: **Example** ```bash [root@basecm10 ]# cmsh [basecm10]% partition use base [basecm10->partition[base]]% provisioningsettings [basecm10->partition[base]->provisioningsettings]% get dirtyautoupdatetimeout [basecm10->partition[base]->provisioningsettings]% 300 [basecm10->partition[base]->provisioningsettings]% set dirtyautoupdatetimeout 0 [basecm10->partition*[base*]->provisioningsettings*]% commit ``` Within Base View the parameter is accessible via the navigation path:\ `Cluster` > `Settings` > `Provisioning Settings` > `Dirty auto update timeout`. To prevent provisioning an image to the nodes, it can be locked (section [5.4.7](#547-synchronizing-the-local-drive-with-the-software-image)). The provisioning request is then deferred until the image is once more unlocked. #### 5.2.4.7 Synchronization Of Fspart Subdirectories To Provisioning Nodes In BCM, an *fspart* is a subdirectory, and it is a filesystem part that can be synced during provisioning. The fsparts can be listed with: **Example** ```bash [root@basecm10 ]# cmsh [basecm10]% fspart [basecm10->fspart]% list Path (key) Type Image ------------------------------ --------------- ------------------------ /cm/images/default-image image default-image /cm/images/default-image/boot boot default-image:boot /cm/node-installer node-installer /cm/shared cm-shared /tftpboot tftpboot /var/spool/cmd/monitoring monitoring ``` The `updateprovisioners` command ([here](#5244-provisioning-role-change-notification-with-updateprovisioners)) is used to update image fsparts to all nodes with a provisioning role. **The `trigger` command:** is used to update non-image fsparts to off-premises nodes, such as cloud directors and edge directors. The directors have a provisioning role for the nodes that they direct. All of the non-image types can be updated with the `--all` option: **Example** ```bash [basecm10->fspart]% trigger --all ``` The command `help trigger` in `fspart` mode gives further details. **The `info` command:** shows the architecture, OS, and the number of inotify watchers that track rsyncs in the fspart subdirectory. ```bash [basecm10->fspart]% info Path Architecture OS Inotify watchers ------------------------------ ---------------- ---------------- ---------------- /cm/images/default-image x86_64 rhel9 0 /cm/images/default-image/boot - - 0 /cm/node-installer x86_64 rhel9 0 /cm/shared x86_64 rhel9 0 /tftpboot - - 0 /var/spool/cmd/monitoring - - 0 [basecm10->fspart]% info -s (!#with size, takes longer) Path Architecture OS Inotify watchers Size ------------------------------ ---------------- ---------------- ---------------- ---------------- /cm/images/default-image x86_64 rhel9 0 4.8 GiB /cm/images/default-image/boot - - 0 313 MiB /cm/node-installer x86_64 rhel9 0 2.84 GiB /cm/shared x86_64 rhel9 0 1.16 GiB /tftpboot - - 0 3.5 MiB /var/spool/cmd/monitoring - - 0 1.02 GiB ``` **The `locked`, `lock`, and `unlock` commands:** * The `locked` command lists fsparts that are prevented from syncing. **Example** ```bash [basecm10->fspart]% locked No locked fsparts ``` * The `lock` command prevents a specific fspart from syncing. **Example** ```bash [basecm10->fspart]% lock /var/spool/cmd/monitoring [basecm10->fspart]% locked /var/spool/cmd/monitoring ``` * The `unlock` command unlocks a specific locked fspart again. **Example** ```bash [basecm10->fspart]% unlock /var/spool/cmd/monitoring [basecm10->fspart]% locked No locked fsparts ``` **Access to `excludelistsnippets`:** The properties of `excludelistsnippets` for a specific fspart can be accessed from the `excludelistsnippets` submode: **Example** ```bash [basecm10->fspart]% excludelistsnippets /tftpboot [basecm10->fspart[/tftpboot]->excludelistsnippets]% list Name (key) Lines Disabled Mode sync Mode full Mode update Mode grab Mode grab new ------------ ------- ------------ ----------- ----------- -------------- ----------- -------------- Default 2 no yes yes yes no no [basecm10->fspart[/tftpboot]->excludelistsnippets]% show default Parameter Value -------------------------------- ----------------------------------------------------------------- Lines 2 Name Default Revision Exclude list # no need for rescue on nodes with a boot role,/rescue,/rescue/* Disabled no No new files no Mode sync yes Mode full yes Mode update yes Mode grab no Mode grab new no [basecm10->fspart[/tftpboot]->excludelistsnippets]% get default excludelist # no need for rescue on nodes with a boot role /rescue /rescue/* ``` ## 5.3 The Kernel Image, Ramdisk And Kernel Modules A *software image* is a complete Linux filesystem that is to be installed on a non-head node. Chapter [11](/admin/softwareimagemanagement) describes images and their management in detail. The head node holds the head copy of the software images. Whenever files in the head copy are changed using CMDaemon, the changes automatically propagate to all provisioning nodes via the `updateprovisioners` command (section [5.2.4.4](#5244-provisioning-role-change-notification-with-updateprovisioners)). ### 5.3.1 Booting To A “Good State” Software Image When nodes boot from the network in simple clusters, the head node supplies them with a *known good state* during node start up. The known good state is maintained by the administrator and is defined using a software image that is kept in a directory of the filesystem on the head node. Supplementary filesystems such as `/home` are served via NFS from the head node by default. For a diskless node the known good state is copied over from the head node, after which the node becomes available to cluster users. For a disked node, by default, the hard disk contents on specified local directories of the node are checked against the known good state on the head node. Content that differs on the node is changed to that of the known good state. After the changes are done, the node becomes available to cluster users. Each software image contains a Linux kernel and a ramdisk. These are the first parts of the image that are loaded onto a node during early boot. The kernel is loaded first. The ramdisk is loaded next, and contains driver modules for the node’s network card and local storage. The rest of the image is loaded after that, during the node-installer stage (section [5.4](#54-node-installer)). ### 5.3.2 Selecting Kernel Driver Modules To Load Onto Nodes Kernel modules can be managed in `softwareimage` mode (using an image), in `category` mode (using a category), or in `device` mode (using a node), as indicated by the following `cmsh` tree view of a newly-installed cluster with default values: ```bash cmsh |-- category[default] | |-- kernelmodules ... |-- device][node001] | |-- kernelmodules ... |-- softwareimage[default-image] | |-- kernelmodules ... ``` As is usual in BCM, if there are values specified at the lower levels in the hierarchy, then their values override the values set higher up in the hierarchy. For example, modules specified at node level override modules specified at category or software image level. Similarly modules specified at the category level override whatever is specified at the software image level. The cluster administrators should be aware that “override” for kernel modules appended to a kernel image means that any kernel modules defined at a higher level are totally ignored—the modules from a lower level exclude the modules at the higher level. A misconfiguration of kernel modules in the lower levels can thus prevent the node from starting up. Modules are normally just set at `softwareimage` level, using `cmsh` or Base View. #### 5.3.2.1 Kernel Driver Modules With `cmsh` In `cmsh`, the modules that are to go on the ramdisk can be placed using the `kernelmodules` submode of the `softwareimage` mode. The order in which they are listed is the attempted load order. Within the `kernelmodules` submode, the `import` command can be used to import the kernel modules list from: * a software image (`-i|--image`) * a node (`-n|--node`) * a category (`-c|--category`) The import replaces the original kernel modules list. **Example** ```bash [basecm10]% softwareimage use target-image [basecm10->softwareimage[target-image]]% kernelmodules [basecm10->softwareimage[target-image]->kernelmodules]% import -i default-image 54 module(s) successfully imported. [basecm10->softwareimage[target-image]->kernelmodules*]% commit ``` Whenever a change is made via the `kernelmodules` submode to the kernel module selection of a software image, CMDaemon automatically runs the `createramdisk` command. The `createramdisk` command regenerates the ramdisk inside the initrd image and sends the updated image to all provisioning nodes, to the image directory, set by default to `/cm/images/default-image/boot/`. The original initrd image is saved as a file with suffix “`.orig`” in that directory. An attempt is made to generate the image for all software images that CMDaemon is aware of, regardless of category assignment, unless the image is protected from modification by CMDaemon with a `FrozenFile` directive (Appendix [C](/admin/cmdconf)). The `createramdisk` command can also be run manually from within the `softwareimage` mode. #### 5.3.2.2 Kernel Driver Modules With Base View In Base View the kernel modules for a particular image are managed through the `Software images` resource, and then choosing the `Kernel modules` menu option of that image. For example, for the image `default-image`, the navigation path that can be followed is:\ `Provisioning` > `Software images[default-image]` > `Edit` > `Settings` > `Kernel modules`\ which opens up the `Kernel Module list` screen, which allows kernel modules to be removed or added (figure [5.6](#fig-bright-view-provisioning-updateramdisk-button)): ![Base View: Selecting Kernel Modules For Software Images](/base-command-manager/product-manuals/_fern-img/e3cd793f06af24321989e0c31ead67fba2982368e997b49c9e34a2ad8ece6800.webp) *Figure 5.6: Base View: Selecting Kernel Modules For Software Images* New kernel modules can be added using the `Add` button, existing kernel modules can be removed using the `Delete` button, and kernel module parameters can be edited using the `Edit` button. #### 5.3.2.3 Manually Regenerating A Ramdisk Regenerating a ramdisk manually via `cmsh` or Base View is useful if the kernel or modules have changed without using CMDaemon. For example, after running a YUM update which has modified the kernel or modules of the nodes (section [11.3](/admin/softwareimagemanagement#113-kernel-management-on-a-head-node-or-image)). In such a case, the distribution would normally update the ramdisk on the machine, but this is not done for the extended ramdisk for nodes in BCM. Not regenerating the BCM ramdisk for nodes after such an update means the nodes may fail on rebooting during the loading of the ramdisk (section [5.8.4](#584-ramdisk-fails-during-loading-or-sometime-later)). An example of regenerating the ramdisk is seen in section [5.8.6](#586-ramdisk-cannot-start-network). #### 5.3.2.4 Implementation Of Kernel Driver Via Ramdisk Or Kernel Parameter Sometimes, testing or setting a kernel driver as a kernel parameter may be more convenient. How to do that is covered in section [11.3.4](/admin/softwareimagemanagement#1134-setting-kernel-options-for-software-images). ### 5.3.3 InfiniBand Provisioning On clusters that have InfiniBand hardware, it is normally used for data transfer as a service after the nodes have fully booted up (section [3.6](/admin/configuring#36-configuring-infiniband-interfaces)). It can also be used for PXE booting (section [5.1.3](#513-ipxe-booting-using-infiniband)) and for node provisioning (described here), but these are not normally a requirement. This section (about InfiniBand node provisioning) may therefore safely be skipped in almost all cases when first configuring a cluster. During node start-up on a setup for which InfiniBand networking has been enabled, the `init` process runs the `rdma` script. For SLES the `openib` script is used instead of the `rdma` script. The script loads up InfiniBand modules into the kernel. When the cluster is finally fully up and running, the use of InfiniBand is thus available for all processes that request it. Provisioning nodes over InfiniBand is not implemented by default, because the `init` process, which handles initialization scripts and daemons, takes place only after the node-provisioning stage launches. InfiniBand modules are therefore not available for use during provisioning, which is why, for default kernels, provisioning in BCM is done via Ethernet. Provisioning at the faster InfiniBand speeds rather than Ethernet speeds is however a requirement for some clusters. To get the cluster to provision using InfiniBand requires both of the following two configuration changes to be carried out: 1. configuring InfiniBand drivers for the ramdisk image that the nodes first boot into, so that provisioning via InfiniBand is possible during this pre-`init` stage 2. defining the provisioning interface of nodes that are to be provisioned with InfiniBand. It is assumed that InfiniBand networking is already configured, as described in section [3.6](/admin/configuring#36-configuring-infiniband-interfaces). The administrator should be aware that the interface from which a node boots, (conveniently labeled `BOOTIF`), must not be an interface that is already configured for that node in CMDaemon. For example, if `BOOTIF` is the device `ib0`, then `ib0` must not already be configured in CMDaemon. Either `BOOTIF` or the `ib0` configuration should be changed so that node installation can succeed. How these two changes are carried out is described next: #### 5.3.3.1 InfiniBand Provisioning: Ramdisk Image Configuration An easy way to see what modules must be added to the ramdisk for a particular HCA can be found by running `rdma` (or `openibd`), and seeing what modules do load up on a fully booted regular node. One way to do this is to run the following lines as root: ```bash [root@basecm10 ~]# { service rdma stop; lsmod | cut -f1 -d" "; }>/tmp/a [root@basecm10 ~]# { service rdma start; lsmod | cut -f1 -d" "; }>/tmp/b ``` The `rdma` service in the two lines should be replaced by `openibd` service instead when using SLES, or distributions based on versions of Red Hat prior to version 6. The first line stops the InfiniBand service, just in case it is running, in order to unload its modules, and then lists the modules on the node. The second line starts the service, so that the appropriate modules are loaded, and then lists the modules on the node again. The output of the first step is stored in a file `a`, and the output from the second step is stored in a file `b`. Running `diff` on the output of these two steps then reveals the modules that get loaded. For `rdma`, the output may display something like: **Example** ```bash [root@basecm10 ~]# diff /tmp/a /tmp/b 1,3c1 < Unloading OpenIB kernel modules: < Failed to unload ib_core < [FAILED] --- > Loading OpenIB kernel modules: [ OK ] 4a3,14 > ib_ipoib > rdma_ucm > ib_ucm > ib_uverbs > ib_umad > rdma_cm > ib_cm > iw_cm > ib_addr > ib_sa > ib_mad > ib_core ``` As suggested by the output, the modules `ib_ipoib`, `rdma_ucm` and so on are the modules loaded when `rdma` starts, and are therefore the modules that are needed for this particular HCA. Other HCAs may cause different modules to be loaded. For a default Red Hat from version 7 onward, the `rdma` service can only be started; it cannot be stopped. Finding the modules that load can therefore only be done once for the default configuration, until the next reboot. The preceding `lsmod` lines in that case can be generated with: **Example** ```bash [root@basecm10 ~]# { lsmod | cut -f1 -d" "; }>/tmp/a [root@basecm10 ~]# { systemctl start rdma-load-modules@rdma; lsmod | cut -f1 -d" "; }>/tmp/b ``` Here, `systemctl` is used instead of the older `service` command just because it is the modern way to run such commands. The InfiniBand modules that load are the ones that the initrd image needs, so that InfiniBand can be used during the node provisioning stage. The administrator can therefore now create an initrd image with the required InfiniBand modules. Loading kernel modules into a ramdisk is covered in general in section [5.3.2](#532-selecting-kernel-driver-modules-to-load-onto-nodes). A typical Mellanox HCA may have an initrd image created as follows (some text ellipsized in the following example): **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% softwareimage use default-image [basecm10->softwareimage[default-image]]% kernelmodules [basecm10...age[default-image]->kernelmodules]% add mlx4_ib [basecm10...age*[default-image*]->kernelmodules*[mlx4_ib*]]% add ib_ipoib [basecm10...age*[default-image*]->kernelmodules*[ib_ipoib*]]% add ib_umad [basecm10...age*[default-image*]->kernelmodules*[ib_umad*]]% commit [basecm10->softwareimage[default-image]->kernelmodules[ib_umad]]% Tue May 24 03:45:35 2011 basecm10: Initial ramdisk for image default-image was regenerated successfully. ``` If the modules are put in another image instead of `default-image`, then the default image that nodes boot from should be set to the new image (section [3.20.2](/admin/configuring#3202-making-nodes-function-differently-by-image)). #### 5.3.3.2 InfiniBand Provisioning: Network Configuration It is assumed that the networking configuration for the final system for InfiniBand is configured following the general guidelines of section [3.6](/admin/configuring#36-configuring-infiniband-interfaces). If it is not, that should be checked first to see if all is well with the InfiniBand network. The provisioning aspect is set by defining the provisioning interface. An example of how it may be set up for 150 nodes with a working InfiniBand interface `ib0` in `cmsh` is: **Example** ```bash [root@basecm10~]# cmsh [basecm10]% device [basecm10->device]% foreach -n node001..node150 (set provisioninginterface ib0) [basecm10->device*]% commit ``` ### 5.3.4 VLAN Provisioning Nodes can be configured for provisioning over a VLAN interface, starting in NVIDIA Base Command Manager version 8.2. This requires: * A VLAN network and node interface. The VLAN network is typically specified by the network switch. The interface that connects the node to the switch can be configured as a VLAN interface as outlined in section [3.4](/admin/configuring#34-configuring-vlan-interfaces). * The `8021q` (`rtnl-link-vlan`) driver to be available in the software image that is provisioned. In recent distributions this driver is not part of the base kernel, and is instead available as a module. The module should be loaded into the software image that is to be provisioned. For example, for `node001` that is missing the module in the software image, the module could be configured to run on the node from the software image as follows: **Example** ```bash [root@basecm10 ~]# ssh node001 "lsmod | grep 8021q" [root@basecm10 ~]# cmsh [basecm10]% softwareimage [basecm10]->softwareimage% use default-image [basecm10->softwareimage[default-image]]% kernelmodules [basecm10->softwareimage[default-image]->kernelmodules]% list | grep 8021q [basecm10->softwareimage[default-image]->kernelmodules]% add 8021q [basecm10->softwareimage*[default-image*]->kernelmodules*[8021q*]]% commit [basecm10->softwareimage[default-image]->kernelmodules[8021q]]% device use node001 [basecm10->device[node001]]% reboot ... some time after boot [root@basecm10 ~]# ssh node001 "lsmod | grep 8021q" 8021q 40960 0 garp 16384 1 8021q mrp 20480 1 8021q ``` Rebooted nodes that use the modified software image then have the VLAN module available in the running kernel. * The BIOS of the node must have the VLANID value set within the BIOS network options. If the BIOS does not support this setting, then PXE over VLAN cannot work. For a NIC that is missing this in the BIOS, the NIC hardware provider may sometimes have a BIOS update that supports this setting. * The VLANID value should be set in the kernel parameters. Kernel parameters for a node can be specified in `cmsh` with the `kernelparameters` setting of `softwareimage` mode (section [11.3.4](/admin/softwareimagemanagement#1134-setting-kernel-options-for-software-images)). An example where the VLANID is appended to some existing parameters could be: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% softwareimage use default-image [basecm10->softwareimage[default-image]]% append kernelparameters " VLANID=89" [basecm10->softwareimage*[default-image*]]% commit ``` ## 5.4 Node-Installer After the kernel has started up, and the ramdisk kernel modules are in place on the node, the node launches the node-installer. The node-installer is a software image (section [11.4.4](/admin/softwareimagemanagement#1144-managing-a-package-in-the-node-installer-image)) provided by the head node. It interacts with CMDaemon on the head node and takes care of the rest of the boot process. As an aside, the node-installer modifies some files (Appendix [A.2.3](/admin/generatedfiles#a23-files-created-on-regular-nodes-by-the-node-installer)) on the node it is installing to, so that they differ from the otherwise-expected pre-init stage Linux system. Such modifications can be prevented by a `frozenFilesPerNode` or `frozenFilesPerCategory` directive, as documented within the `node-installer.conf` file, and explained in greater detail [here](/admin/cmdconf#c6913-frozenfile-directive-for-regular-nodes-for-the-node-installer). Once the node-installer has completed its tasks, the local drive of the node has a complete Linux pre-init stage system. The node-installer ends by calling `/sbin/init` from the local drive and the boot process then proceeds as a normal Linux boot. The steps the node-installer goes through for each node are: 1. requesting a node certificate (section [5.4.1](#541-requesting-a-node-certificate)) 2. deciding or selecting node configuration (section [5.4.2](#542-deciding-or-selecting-node-configuration)) 3. starting up all network interfaces (section [5.4.3.1](#5431-starting-up-all-provisioning-network-interfaces)) 4. determining install-mode type and execution mode (section [5.4.4](#544-determining-install-mode-type-and-execution-mode)) 5. running `initialize` scripts (section [5.4.5](#545-running-initialize-scripts)) 6. checking partitions, mounting filesystems (section [5.4.6](#546-checking-partitions-raid-configuration-mounting-filesystems)) 7. synchronizing the local drive with the correct software image (section [5.4.7](#547-synchronizing-the-local-drive-with-the-software-image)) 8. writing network configuration files to the local drive (section [5.4.8](#548-writing-network-configuration-files)) 9. creating an `/etc/fstab` file on the local drive (section [5.4.9](#549-creating-a-local-etcfstab-file)) 10. installing GRUB bootloader if configured by BCM (section [5.4.10](#5410-booting-from-the-local-hard-drive)), and initializing SELinux if it has been installed and configured ([Chapter 12 of the *Installation Manual*](/installation/selinux)) 11. running `finalize` scripts (section [5.4.11](#5411-running-finalize-scripts)) 12. unloading specific drivers no longer needed (section [5.4.12](#5412-unloading-specific-drivers)) 13. switching the root device to the local drive and calling `/sbin/init` (section [5.4.13](#5413-switching-to-the-local-init-process)) These 13 node-installer steps and related matters are described in detail in the corresponding sections .1–.13. ### 5.4.1 Requesting A Node Certificate Each node communicates with the CMDaemon on the head node using a certificate. If no certificate is found, it automatically requests one from CMDaemon running on the head node (figure [5.7](#fig-prov-certreq)). ![Certificate Request](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/bde5a5b5d7b24c155f29e0eca2ca017b8bfc9fff822d86097b9e5cc2e6a2bc7e/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/prov-certreq.png) *Figure 5.7: Certificate Request* The certificate is stored on the head node in `/cm/node-installer/certificates/` by MAC address. #### 5.4.1.1 Certificate Auto-signing *Certificate auto-signing* means the cluster management daemon automatically signs a certificate signing request (CSR) that has been requested by a node. Certificate auto-signing can be configured from within `partition` mode of `cmsh`, with the `signinstallercertificates` parameter. It can take one of the following values: * `AUTO` (the default) * `MANUAL` **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% partition [basecm10->partition[base]]% set signinstallercertificates auto ``` For untrusted networks, it may be wiser to approve certificate requests manually to prevent new nodes being added automatically without getting noticed. Disabling certificate auto-signing for all networks can be done by setting `signinstallercertificates` to `MANUAL`. Instead of disabling certificate autosigning for all networks, a finer tuning can be carried out for individual networks. This requires that `signinstallercertificates` be set to `AUTO` in `partition` mode. The `allowautosign` parameter in `network` mode can then be set for a particular network, and it can take one of the following values: * `Always` * `Automatic` (the default) * `Never` * `Secret` **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% network use internalnet [basecm10->network[internalnet]]% set allowautosign automatic[TAB][TAB] always automatic never secret ``` If `Always` is set, then incoming CSRs from all types of networks are automatically auto-signed. If `Automatic` is set, then only networks that are of type internal are automatically auto-signed. If `Never` is set, then all incoming CSRs that come in for that network have to be manually approved. The value `Secret` is required for the `globalnet` network, for edge sites. A node on an edge site uses a shared secret that is passed along with the node request. The secret is set during edge site setup by BCM ([section 2.1.1 of the *Edge Manual*](/edge/edge#211-defining-the-edge-site)). #### 5.4.1.2 Manual Approval Of A CSR **Approval of the CSR from a regular node (not an edge node):** Manual approval of a CSR is typically done from within `certs` mode. A list of requests can be found, and from the list, the appropriate unsigned request can be signed and issued. The following session illustrates the process: ```bash [basecm10->cert]% listrequests Request ID Client type Session ID Autosign Name ------------ ------------ ------------ -------- ------------------ 6 installer 42949672986 No fa-16-3e-22-cd-13 [basecm10->cert]% issuecertificate 6 Issued 6 ``` **Approval of the CSR from an edge node:** If the shared secret has not been set for the edge director—that is, if it has not been stored locally on the edge director, or if it has not been passed on via the installation medium—then the node-installer prompts for the secret the first time that it boots. If the secret that is typed in matches the site secret, then a CSR from the edge director is handled by the head node, and a signed certificate is issued. The edge compute nodes pick up their secret from the director. If the director does not have the secret, then the compute node’s node-installer prompts for the secret on first boot. Once the secret is set, then edge compute node sends its CSR to the head node (via the edge director) and gets a signed certificate automatically. Section [2.3](/admin/clustermanagement#23-authentication) has more information on certificate management in general. #### 5.4.1.3 Certificate Storage And Removal Implications After receiving a valid certificate, the node-installer stores it in `/cm/node-installer/certificates/` \<*`node mac address`*>`/` on the head node. This directory is NFS exported to the nodes, but can only be accessed by the `root` user. The node-installer does not request a new certificate if it finds a certificate in this directory, valid or invalid. If an invalid certificate is received, the screen displays a communication error. Removing the node’s corresponding certificate directory allows the node-installer to request a new certificate and proceed further. ### 5.4.2 Deciding Or Selecting Node Configuration Once communication with the head node CMDaemon is established, the node-installer tries to identify the node it is running on so that it can select a configuration from CMDaemon’s record for it, if any such record exists. It correlates any node configuration the node is expected to have according to network hardware detected. If there are issues during this correlation process then the administrator is prompted to select a node configuration until all nodes finally have a configuration. #### 5.4.2.1 Possible Node Configuration Scenarios The correlations process and corresponding scenarios are now covered in more detail: It starts with the node-installer sending a query to CMDaemon to check if the MAC address used for net booting the node is already associated with a node in the records of CMDaemon. In particular, it checks the MAC address for a match against the existing *node configuration* properties, and decides whether the node is *known* or *new*. * the node is **known** if the query matches a node configuration. It means that node has been booted before. * the node is **new** if no configuration is found. In both cases the node-installer then asks CMDaemon to find out if the node is connected to an Ethernet switch, and if so, to which port. Setting up Ethernet switches for port detection is covered in section [3.9](/admin/configuring#39-configuring-switches-and-pdus). If a port is detected for the node, the node-installer queries CMDaemon for a node configuration associated with the detected Ethernet switch port. If a port is not detected for the node, then either the hardware involved with port detection needs checking, or a node configuration must be selected manually. There are thus several scenarios: 1. The node is new, and an Ethernet switch port is detected. A node configuration associated with the port is found. The node-installer suggests to the administrator that the new node should use this configuration, and displays the configuration along with a confirmation dialog (figure [5.8](#fig-prov-node)). This suggestion can be interrupted, and other node configurations can be selected manually instead through a sub-dialog (figure [5.9](#fig-prov-select-node)). By default (in the main dialog), the original suggestion is accepted after a timeout. ![Scenarios: Configuration Found, Confirm Node Configuration](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/a0c6ec2ac9051f9da5e1f883241a02105931c342384e353e72aa61ccf58d4600/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/prov-node.png) *Figure 5.8: Scenarios: Configuration Found, Confirm Node Configuration* ![Scenarios: Node Selection Sub-Dialog](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/ff575e814994334931473034dd02639744b7d01b71aca75a9c2663abf0567d83/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/prov-select-node.png) *Figure 5.9: Scenarios: Node Selection Sub-Dialog* 2. The node is new, and an Ethernet switch port is detected. A node configuration associated with the port is not found. The node-installer then displays a dialog that allows the administrator to either retry Ethernet switch port detection (figure [5.10](#fig-prov-node-unknown)) or to drop into a sub-dialog to manually select a node configuration (figure [5.9](#fig-prov-select-node)). By default, port detection is retried after a timeout. ![Scenarios: Unknown Node, Ethernet Port Detected](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/496e7ec2655cb61715623fb83bfa2c28d4c8b200ab80023c04c434e6567cf45f/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/prov-node-unknown.png) *Figure 5.10: Scenarios: Unknown Node, Ethernet Port Detected* 3. The node is new, and an Ethernet switch port is not detected. The node-installer then displays a dialog that allows the user to either retry Ethernet switch port detection (figure [5.11](#fig-prov-node-unknown-noport)) or to drop into a sub-dialog to manually select a node configuration (figure [5.9](#fig-prov-select-node)). By default, port detection is retried after a timeout. ![Scenarios: Unknown Node, No Ethernet Port Detected](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/4228461b99e6617055b245ccb1c19537f462498e1cd2c8f404b8066e2ded495c/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/prov-node-unknown-noport.png) *Figure 5.11: Scenarios: Unknown Node, No Ethernet Port Detected* 4. The node is known, and an Ethernet switch port is detected. The configuration associated with the port is the same as the configuration associated with the node’s MAC address. The node-installer then displays the configuration as a suggestion along with a confirmation dialog (figure [5.8](#fig-prov-node)). The suggestion can be interrupted, and other node configurations can be selected manually instead through a sub-dialog (figure [5.9](#fig-prov-select-node)). By default (in the main dialog), the original suggestion is accepted after a timeout. 5. The node is known, and an Ethernet switch port is detected. However, the configuration associated with the port is not the same as the configuration associated with the node’s MAC address. This is called a *port mismatch*. This type of port mismatch situation occurs typically during a mistaken *node swap*, when two nodes are taken out of the cluster and returned, but their positions are swapped by mistake (or equivalently, they are returned to the correct place in the cluster, but the switch ports they connect to are swapped by mistake). To prevent configuration mistakes, the node-installer displays a port mismatch dialog (figure [5.12](#fig-prov-port-mismatch)) allowing the user to retry, accept a node configuration that is associated with the detected Ethernet port, or to manually select another node configuration via a sub-dialog (figure [5.9](#fig-prov-select-node)). By default (in the main port mismatch dialog), port detection is retried after a timeout. ![Scenarios: Port Mismatch Dialog](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/ea0af9a79d1e2d4e44a73f477b39eb25cb9e938ef5274fc6a5e2dcc389c7e56b/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/prov-port-mismatch.png) *Figure 5.12: Scenarios: Port Mismatch Dialog* 6. The node is known, and an Ethernet switch port is not detected. However, the configuration associated with the node’s MAC address does have an Ethernet port associated with it. This is also considered a port mismatch. To prevent configuration mistakes, the node-installer displays a port mismatch dialog similar to figure [5.12](#fig-prov-port-mismatch), allowing the user to retry or to drop into a sub-dialog and manually select a node configuration that may work. However, a more likely solution in most cases is to: * either clear the switch port configuration in the cluster manager so that switch port detection is not attempted. For example, for `node001`, this can be done by running this `cmsh` command on the head node:\ `cmsh -c "device clear node001 switchports; commit"` * or enable switch port detection on the switch. This is usually quite straightforward, but may require going through the manuals or software application that the switch manufacturer has provided. By default (in the port mismatch dialog), port detection is retried after a timeout. This means that if the administrator clears the switch port configuration or enables switch port detection, the node-installer is able to continue automatically with a consistent configuration. 7. The node is known, and an Ethernet switch port is detected. However, the configuration associated with the node’s MAC address has no Ethernet switch port associated with it. This is not considered a port mismatch but an unset switch port configuration, and it typically occurs if switch port configuration has not been carried out, whether by mistake or deliberately. The node-installer displays the configuration as a suggestion along with a confirmation dialog (figure [5.13](#fig-prov-ports-not-configured)). The suggestion can be interrupted, and other node configurations can be selected manually instead using a sub-dialog. By default (in the main dialog) the configuration is accepted after a timeout. ![Scenarios: Port Unset Dialog](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/9c1d9d2ee46ca0c0f47515d35e9c75428bb88d0c86937565a79cf6826d5a1cc3/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/prov-ports-not-configured.png) *Figure 5.13: Scenarios: Port Unset Dialog* A truth table summarizing the scenarios is helpful:
**Scenario** **Node known?** **Switch port detected?** **Switch port configuration found?** **Switch port configuration conflicts with node configuration?**
1 No Yes Yes No
2 No Yes No No
3 No No No No
4 Yes Yes Yes No
5 Yes Yes Yes Yes (configurations differ)
6 Yes No Yes Yes (port expected by MAC configuration not found)
7 Yes Yes No No (port not expected by MAC configuration)
In these scenarios, whenever the user manually selects a node configuration in the prompt dialog, an attempt to detect an Ethernet switch port is repeated. If a port mismatch still occurs, it is handled by the system as if the user has not made a selection. #### 5.4.2.2 Summary Of Behavior During Hardware Changes The logic of the scenarios means that an unpreconfigured node always boots to a dialog loop requiring manual intervention during a first install (scenarios 2 and 3). For subsequent boots the behavior is: * If the node MAC hardware has changed (scenarios 1, 2, 3): * if the node is new and the detected port has a configuration, the node automatically boots to that configuration (scenario 1). * else manual intervention is needed (scenarios 2, 3) * If the node MAC hardware has not changed (scenarios 4, 5, 6, 7): * if there is no port mismatch, the node automatically boots to its last configuration (scenarios 4, 7). * else manual intervention is needed (scenarios 5, 6). #### 5.4.2.3 The `newnodes` Command **`newnodes` basic use:** New nodes that have not been configured yet can be detected using the `newnodes` command from within the `device` mode of `cmsh`. A new node is detected when it reaches the node-installer stage after booting, and contacts the head node. **Example** ```bash [basecm10->device]% newnodes The following nodes (in order of appearance) are waiting to be assigned: MAC First appeared Detected on switch port ----------------- ----------------------------- ----------------------- 00:0C:29:01:0F:F8 Mon, 14 Feb 2011 10:16:00 CET [no port detected] ``` At this point the node-installer is seen by the administrator to be looping, waiting for input on what node name is to be assigned to the new node. The nodes can be uniquely identified by their MAC address or switch port address. The port and switch to which a particular MAC address is connected can be discovered by using the `showport` command (section [3.10.4](/admin/configuring#3104-the-showport-mac-address-to-port-matching-tool)). After confirming that they are appropriate, the `switchports` property for the specified device can be `set` to the port and switch values. **Example** ```bash [basecm10->device]% showport 00:0C:29:01:0F:F8 switch01:8 [basecm10->device]% set node003 switchports switch01:8 [basecm10->device*]% commit ``` When the node name (`node003` in the preceding example) is assigned, the node-installer stops looping and goes ahead with the installation to the node. The preceding basic use of `newnodes` is useful for small numbers of nodes. For larger number of nodes, the advanced options of `newnodes` may help carry out node-to-MAC assignment with less effort. **`newnodes` advanced use—options:** The list of MAC addresses discovered by a `newnodes` command can be assigned in various ways to nodes specified by the administrator. Node objects should be created in advance to allow the assignment to take place. The easiest way to set up node objects in `cmsh` is to use the `--clone` option of the `foreach` command (section [2.5.5.9](/admin/clustermanagement#2559-looping-over-objects-with-foreach), [here](/admin/clustermanagement#2559-looping-over-objects-with-foreach)). The advanced options of `newnodes` are particularly useful for quickly assigning node names to specific physical nodes. All that is needed is to power the nodes up in the right order. For nodes with the same hardware, the node that is powered up first reaches the stage where it tries to connect with the node-installer first. So its MAC address is detected first, and arrives on the list generated by `newnodes` first. If some time after the first node is powered up, the second node is powered up, then its MAC address becomes the second MAC address on the list, and so on for the third, fourth, and further nodes. When assigning node names to a physical node, on a cluster that has no such assignment already, the first node that arrived on the list gets assigned the name `node001`, the second node that arrived on the list gets assigned the name `node002` and so on. The advanced options are shown in `device` mode by running the `help newnodes` command. The options can be introduced as being of three kinds: straightforward, grouping, and miscellaneous: * The straightforward options: * `-n|--nodes` * `-w|--write` * `-s|--save` Usually the most straightforward way to assign the nodes is to use the `-n` option, which accepts a list of nodes, together with a `-w` or `-s` option. The `-w` (`--write`) option sets the order of nodes to the corresponding order of listed MAC addresses, and is the same as setting an object in `cmsh`. The `-s` (`--save`) option is the same as setting and committing an object in `cmsh`, so `-s` implies a `-w` option is run at the same time. So, for example, if 8 new nodes are discovered by the node-installer on a cluster with no nodes so far, then: **Example** ```bash [basecm10->device]% newnodes -w -n node001..node008 ``` assigns (but does not commit) the sequence node001 to node008 the new MAC address according to the sequence of MAC addresses displaying on the list. * The grouping options: * `-g|--group` * `-c|--category` * `-h|--chassis` * `-r|--rack` The “`help newnodes`” command in `device` mode shows assignment options other than `-n` for a node range are possible. For example, the assignments can also be made for a group (`-g`), per category (`-c`), per chassis (`-h`), and per rack (`-r`). * The miscellaneous options: * `-f|--force` * `-o|--offset` By default, the `newnodes` command fails when it attempts to set a node name that is already taken. The `-f` (`--force`) option forces the new MAC address to be associated with the old node name. When used with an assignment grouping, (node range, group, category, chassis, or rack) all the nodes in the grouping lose their node-to-MAC assignments and get new assignments. The `-f` option should therefore be used with care. The `-o` (`--offset`) option takes a number \<*number*> and skips \<*number*> nodes in the list of detected unknown nodes, before setting or saving values from the assignment grouping. Examples of how to use the advanced options follow. **`newnodes` advanced use—range assignment behavior example:** For example, supposing there is a cluster with nodes assigned all the way up to node022. That is, CMDaemon knows what node is assigned to what MAC address. For the discussion that follows, the three nodes node020, node021, node022 can be imagined as being physically in a rack of their own. This is simply to help to visualize a layout in the discussion and tables that follow and has no other significance. An additional 3 new, that is unassigned, nodes are placed in the rack, and allowed to boot and get to the node-installer stage. The `newnodes` command discovers the new MAC addresses of the new nodes when they reach their node-installer stage, as before (the switch port column is omitted in the following text for convenience): **Example** ```bash [basecm10->device]% newnodes MAC First appeared ----------------- ----------------------------- 00:0C:29:EF:40:2A Tue, 01 Nov 2011 11:42:31 CET 00:0C:29:95:D3:5B Tue, 01 Nov 2011 11:46:25 CET 00:0C:29:65:9A:3C Tue, 01 Nov 2011 11:47:13 CET ``` The assignment of MAC to node address could be carried out as follows: **Example** ```bash [basecm10->device]% newnodes -s -n node023..node025 MAC First appeared Hostname -------------------- ----------------------------- -------- 00:0C:29:EF:40:2A Tue, 01 Nov 2011 11:42:31 CET node023 00:0C:29:95:D3:5B Tue, 01 Nov 2011 11:46:25 CET node024 00:0C:29:65:9A:3C Tue, 01 Nov 2011 11:47:13 CET node025 ``` Once this is done, the node-installer is able to stop looping, and to go ahead and install the new nodes with an image. The physical layout in the rack may then look as indicated by this:
**before** **after** MAC
node020 node020
node021 node021
node022 node022
node023 ...A
node024 ...B
node025 ...C
Here, `node023` is the node with the MAC address ending in `A`. If instead of the previous `newnodes` command, an offset of 1 is used to skip assigning the first new node: **Example** ```bash [basecm10->device]% newnodes -s -o 1 node024..node025 ``` then the rack layout looks like:
**before** **after** MAC
node020 node020
node021 node021
node022 node022
*unassigned* ...A
node024 ...B
node025 ...C
Here, *unassigned* is where `node023` of the previous example is physically located, that is, the node with the MAC address `...A`. The lack of assignment means there is actually no association of the name `node023` with that MAC address, due to the `newnodes` command having skipped over it with the `-o` option. If instead the assignment is done with: **Example** ```bash [basecm10->device]% newnodes -s 1 node024..node026 ``` then the `node023` name is unassigned, and the name `node024` is assigned instead to the node with the MAC address `...A`, so that the rack layout looks like:
**before** **after** MAC
node020 node020
node021 node021
node022 node022
node024 ...A
node025 ...B
node026 ...C
**`newnodes` advanced use—assignment grouping example:** Node range assignments are one way of using `newnodes`. However assignments can also be made to a category, a rack, or a chassis. For example, with Base View assigning node names to a rack can be done from the `Racks` option of the node. For example, to add a `node001` to a `rack1`, the navigation path would be: `Devices` > `Settings[node001]` > `Rack[rack1]`. In `cmsh`, the assignment of multiple node names to a rack can conveniently be done with a `foreach` loop from within `device` mode: **Example** ```bash [basecm10->device]% foreach -n node020..node029 (set rack rack02) [basecm10->device*]% commit [basecm10->device]% foreach -n node030..node039 (set rack rack03) [basecm10->device*]% commit ``` The assignment of node names with the physical node in the rack can then be arranged as follows: If the nodes are identical hardware, and are powered up in numerical sequence, from `node020` to `node039`, with a few seconds in between, then the list that the basic `newnodes` command (without options) displays is arranged in the same numerical sequence. Assigning the list in the rack order can then be done by running: **Example** ```bash [basecm10->device]% newnodes -s -r rack02..rack03 ``` If it turns out that the boot order was done very randomly and incorrectly for all of `rack02`, and that the assignment for `rack02` needs to be done again, then a simple way to deal with it is to bring down the nodes of `rack02`, then clear out all of the `rack02` current MAC associations, and redo them according to the correct boot order: **Example** ```bash [basecm10->device]% foreach -r rack02 ( clear mac ) ; commit ...removes MAC association with nodes from CMDaemon... ...now reboot nodes in rack02 in sequence (not with BCM)... [basecm10->device]% newnodes ...shows sequence as the nodes come up... [basecm10->device]% newnodes -s -r rack02 ...assigns sequence in boot order... ``` **`newnodes` advanced use—assignment forcing example:** The `--force` option can be used in the following case: Supposing that `node022` fails, and a new node hardware comes in to replace it. The new regular node has a new MAC address. So, as explained by scenario 3 (section [5.4.2](#542-deciding-or-selecting-node-configuration)), if there is no switch port assignment in operation for the nodes, then the node-installer loops around, waiting for intervention. (With switch port assignment in place, scenario 1 means the new node simply boots up by default and becomes the new `node022` without further intervention.) This situation can be dealt with from the command line by: * accepting the node configuration at the regular node console, via a sub-dialog * accepting the node configuration via `cmsh`, without needing to be at the regular node console: ```bash [basecm10->device]% newnodes -s -f -n node022 ``` #### 5.4.2.4 Node Identification The *node identification* resource can be accessed via the Base View navigation path: `Devices` > `Nodes Identification`. The node identification resource is roughly the Base View equivalent to the `newnodes` command of `cmsh`, and it opens up the `New Node list` window (figure [5.14](#fig-bright-view-provisioning-node-identification-wizard)). As is the case for `newnodes` in `cmsh`, the `New Node list` window of Base View lists the MAC address of any unassigned node that the head node detects, and shows the associated detected switch port for the node. Also, just as for `newnodes`, `New Node list` can help assign a node name to the node, assuming the node object exists. After assignment is done, the new status should be saved. ![Node Identification Resource](/base-command-manager/product-manuals/_fern-img/c84d2b742768b9ab88ba38ad08f91e8133964628f5f173f981bdc8b6005c7d7c.webp) *Figure 5.14: Node Identification Resource* The most useful way of using the node identification resource is for node assignment in large clusters. To do this, it is assumed that the node objects have already been created for the new nodes. The creation of the node objects means that the node names exist, and so assignment to the node names is able to take place. An easy way to create many nodes in Base View, set their provisioning interface, and set their IP addresses is described in the section on the *node creation wizard* (section [5.7.2](#572-adding-new-nodes-with-the-node-creation-wizard)). Node objects can also be created easily in large numbers by using `cmsh`’s `foreach` loop command on a node with the `--clone` option (section [2.5.5.9](/admin/clustermanagement#2559-looping-over-objects-with-foreach), [here](/admin/clustermanagement#2559-looping-over-objects-with-foreach)). The nodes are also assumed to be set for net booting, typically set from a BIOS setting. The physical nodes are then powered up in an arranged order. Because they are unknown new nodes, the node-installer keeps looping after a timeout. The head node in the meantime detects the new MAC addresses and switch ports in the sequence in which they first have come up and lists them in that order. By default, all these newly detected nodes are set to an install mode of `auto` (section [5.4.4](#544-determining-install-mode-type-and-execution-mode)), which means that their numbering goes up sequentially from whatever number is assigned to the preceding node in the list. Thus, if there are 10 new unassigned nodes that are brought into the cluster, and the first node in the list is assigned to the first available number, say `node327`; then clicking on assign automatically assigns the remaining nodes to the next 9 available numbers, say `node328--node336`. After the assignment, the node-installer looping process on the new nodes notices that the nodes are now known. The node-installer then breaks out of the loop, and installation goes ahead without any intervention needed at the node console. ### 5.4.3 Starting Up All Network Interfaces At the end of section [5.4.2](#542-deciding-or-selecting-node-configuration), the node-installer knows which node it is running on, and has decided what its node configuration is. #### 5.4.3.1 Starting Up All Provisioning Network Interfaces It now gets on with setting up the IP addresses on the provisioning interfaces required for the node-installer, while taking care of matters that come up on the way: **Avoiding duplicate IP addresses:** The node-installer brings up all the network interfaces configured for the node. Before starting each interface, the node-installer first checks if the IP address that is about to be used is not already in use by another device. If it is, then a warning and retry dialog is displayed until the IP address conflict is resolved. **Using `BOOTIF` to specify the boot interface:** `BOOTIF` is a special name for one of the possible interfaces. The node-installer automatically translates `BOOTIF` into the name of the device, such as `eth0` or `eth1`, used for network booting. This is useful for a machine with multiple network interfaces where it can be unclear whether to specify, for example, `eth0` or `eth1` for the interface that was used for booting. Using the name `BOOTIF` instead means that the underlying device, `eth0` or `eth1` in this example, does not need to be specified in the first place. **Halting on missing kernel modules for the interface:** For some interface types like VLAN and channel bonding, the node-installer halts if the required kernel modules are not loaded or are loaded with the wrong module options. In this case the kernel modules configuration for the relevant software image should be reviewed. Recreating the ramdisk and rebooting the node to get the interfaces up again may be necessary, as described in section [5.8.6](#586-ramdisk-cannot-start-network). #### 5.4.3.2 Bringing Up Non-Provisioning Network Interfaces Provisioning interfaces are by default automatically brought up during the init stage, as the node is fully booted up. The BMC and non-provisioning interfaces on the other hand have a different behavior: **Bringing Up And Initializing BMC Interfaces:** If a BMC interface is present and powered up, then it is expected to be running at least with layer 2 activity (ethernet). It can be initialized in the node configuration (section [3.7](/admin/configuring#37-configuring-bmc-ipmiilodraccimcredfish-interfaces)) with an IP address, netmask and user/password settings so that layer 3 (TCP/IP) networking works for it. BMC networking runs independently of node networking. **Bringing up non-BMC, non-provisioning network interfaces:** Non-provisioning interfaces are inactive unless they are explicitly brought up. BCM can configure how these non-provisioning interfaces are brought up by using the `bringupduringinstall` parameter, which can take the following values: * `yes`: Brings the interface up during the pre-init stage * `no`: Keeps the interface down during the pre-init stage. This is the default for non-provisioning interfaces. * `yesandkeep`: Brings the interface up during the pre-init stage, and keeps it up during the transition to the init stage. #### 5.4.3.3 Bringing Up And Keeping Up Provisioning Network Interfaces The preceding `bringupduringinstall` parameter is not generally supported for provisioning interfaces. However the `yesandkeep` value does work for provisioning interfaces too, under some conditions: * `yesandkeep`: Brings the interface up during the pre-init stage, and keeps it up during the transition to the init stage, for the following provisioning devices: * Ethernet device interfaces using a leased DHCP address * InfiniBand device interfaces running with distribution OFED stacks #### 5.4.3.4 Restarting The Network Interfaces At the end of this step (i.e. section [5.4.3.1](#5431-starting-up-all-provisioning-network-interfaces)) the network interfaces are up. When the node-installer has completed the remainder of its 13 steps (sections [5.4.4](#544-determining-install-mode-type-and-execution-mode)–[5.4.13](#5413-switching-to-the-local-init-process)), control is handed over to the local `init` process running on the local drive. During this handover, the node-installer brings down all network devices. These are then brought back up again by `init` by the distribution’s standard networking `init` scripts, which run from the local drive and expect networking devices to be down to begin with. ### 5.4.4 Determining Install-mode Type And Execution Mode Stored *install-mode* values decide whether synchronization is to be applied fully to the local drive of the node, only for some parts of its filesystem, not at all, or even whether to drop into a maintenance mode instead. Related to install-mode values are execution mode values ([here](#5443-install-modes-execution-modes)) that determine whether to apply the install-mode values to the next boot, to new nodes only, to individual nodes or to a category of nodes. Related to execution mode values is the confirmation requirement toggle value ([here](#5444-full-install-confirmation-via-datanode-setting)) in case a full installation is to take place. These values are merely determined at this stage; nothing is executed yet. #### 5.4.4.1 Install-mode Values The install-mode can have one of five values: `AUTO`, `FULL`, `MAIN`, `NOSYNC`, and `SKIP`. It should be understood that the term “install-mode” implies that these values operate only during the node-installer phase. **NOTE:** For example, `imageupdate` (section [5.6.2](#562-updating-running-nodes-with-cmsh-using-imageupdate)), which is run by CMDaemon, ignores these settings, which is as expected. This means that, for example, if `imageupdate` is run with `NOSYNC` set, then the head node image is still synchronized over as usual to the regular node while the node is up. It is only during node boot, during the installer stage, that setting NOSYNC prevents synchronization. * If the install-mode is set to `FULL`, then the node-installer re-partitions, creates new filesystems and synchronizes a full image onto the local drive according a *partition layout*. This process wipes out all pre-boot drive content. A partition layout (Appendix [D](/admin/partitioning)) includes defined values for the partitions, sizes, and filesystem types for the nodes being installed. An example of a partition layout is the default partition layout (Appendix [D.3](/admin/partitioning#d3-example-default-node-partitioning)). * If the install-mode is set to `AUTO`, then the node-installer checks the partition layout of the local drive against the node’s stored configuration. If these do not match because, for example, the node is new, or if they are corrupted, then the node-installer recreates the partitions and filesystems by carrying out a `FULL` install. If however the drive partitions and filesystems are healthy, the node-installer only does an incremental software image synchronization. Synchronization tends to be quick because the software image and the local drive usually do not differ much. Synchronization also removes any extra local files that do not exist on the image, for the files and directories considered. Section [5.4.7](#547-synchronizing-the-local-drive-with-the-software-image) gives details on how it is decided what files and directories are considered. * If the install-mode is set to `MAIN`, then the node-installer does not carry out a disk check, and goes on to maintenance mode, allowing manual investigation of specific problems. The local drive is untouched. * If the install-mode is set to `NOSYNC`, and the partition layout check matches the stored XML configuration, then the node-installer skips synchronizing the image to the node, so that contents on the local drive persist from the previous boot. An exception to this is the node certificate and key, that is the files `/cm/local/apps/cmd/etc/cert.{pem|key}`. These are updated from the head node if missing. If however the partition layout does not match the stored configuration, a `FULL` image sync is triggered. Thus, for example, a burn session ([Chapter 11 of the *Installation Manual*](/installation/burn)), with the default burn configuration which destroys the existing partition layout on a node, will trigger a `FULL` image sync on reboot after the burn session. The `NOSYNC` setting should therefore not be regarded as a way to protect data. Ways to preserve data across node reboots are discussed in the section that discusses the `FULL` install confirmation settings ([here](#5444-full-install-confirmation-via-datanode-setting)). `NOSYNC` is useful during mass planned node reboots when set with the `nextinstallmode` option of `device` mode. This sets the nodes to use the OS on the hard drive, during the next boot only, without an image sync: **Example** ```bash [basecm10]% device foreach -n node001..node999 (set nextinstallmode nosync) [basecm10]% device commit ``` * If the install-mode is set to `SKIP`, then the node-installer does not carry out a check of the partitions and filesystems, and it also does not carry out a software image synchronization. If a node runs into problems with its drive content during a normal start up attempt, then this mode can perhaps be used to attempt data recovery on the node. #### 5.4.4.2 Install-mode Logging The decision that is made is normally logged to the node-installer file, `/var/log/node-installer` on the head node. **Example** ```bash 08:40:58 node001 node-installer: Installmode is: AUTO 08:40:58 node001 node-installer: Fetching disks setup. 08:40:58 node001 node-installer: Setting up environment for initialize scripts. 08:40:58 node001 node-installer: Initialize script for category default is empty. 08:40:59 node001 node-installer: Checking partitions and filesystems. 08:40:59 node001 node-installer: Updating device status: checking disks 08:40:59 node001 node-installer: Detecting device '/dev/sda': found 08:41:00 node001 node-installer: Number of partitions on sda is ok. 08:41:00 node001 node-installer: Size for /dev/sda1 is ok. 08:41:00 node001 node-installer: Checking if /dev/sda1 contains ext3 filesystem. 08:41:01 node001 node-installer: fsck.ext3 -a /dev/sda1 08:41:01 node001 node-installer: /dev/sda1: recovering journal 08:41:02 node001 node-installer: /dev/sda1: clean, 129522/1250928 files, 886932/5000000 blocks 08:41:02 node001 node-installer: Filesystem check on /dev/sda1 is ok. 08:41:02 node001 node-installer: Size for /dev/sda2 is wrong. 08:41:02 node001 node-installer: Partitions and/or filesystems are missing/corrupt. (Exit code\ 18, signal 0) 08:41:03 node001 node-installer: Creating new disk layout. ``` In this case the node-installer detects that the size of `/dev/sda2` on the disk no longer matches the stored configuration, and triggers a full re-install. For further detail beyond that given by the node-installer log, the `disks` script at `/cm/node-installer/scripts/disks` on the head node can be examined. The node-installer checks the disk by calling the `disks` script. Exit codes, such as the `18` reported in the log example, are defined near the top of the `disks` script. #### 5.4.4.3 Install-mode’s Execution Modes Execution of an install-mode setting is possible in several ways, both permanently or just temporarily for the next boot. Execution can be set to apply to categories or individual nodes. The node-installer looks for install-mode execution settings in this order: 1. The “`New node installmode`” property of the node’s category. This decides the install mode for a node that is detected to be new. It can be set for the `default` category using a Base View navigation path such as: `Grouping` > `Node categories[default]` > `Edit` > `Settings` > `Install mode` or using `cmsh` with a one-liner such as: ```bash cmsh -c "category use default; set newnodeinstallmode FULL; commit" ``` By default, the “`New node installmode`” property is set to `FULL`. 2. The Install-mode setting as set by choosing a PXE menu option on the console of the node before it loads the kernel and ramdisk (figure [5.15](#fig-prov-pxe)). This only affects the current boot. By default the PXE menu install mode option is set to `AUTO`. ![PXE Menu With Install-mode Set To AUTO](/base-command-manager/product-manuals/_fern-img/d74520f84d13f8c4f8d46bb7d6a4839872329b8c849886c039da4dc72fb61bb8.webp) *Figure 5.15: PXE Menu With Install-mode Set To AUTO* 3. The “`Next boot install-mode`” property of the node configuration. This can be set for a node such as `node001` using a Base View navigation path such as:\ `Devices` > `Nodes[node001]` > `Edit` > `Settings` > `Install mode` It can also be set using `cmsh` with a one-liner: ```bash cmsh -c "device use node001; set nextinstallmode FULL; commit" ``` The property is cleared when the node starts up again, after the node-installer finishes its installation tasks. So it is empty unless specifically set by the administrator during the current uptime for the node. 4. The `install-mode` property can be set in the node configuration using Base View via `Devices` > `Nodes[node001]` > `Edit` > `Settings` > `Install mode` or using `cmsh` with a one-liner such as: ```bash cmsh -c "device use node001; set installmode FULL; commit" ``` By default, the install-mode property is auto-linked to the property set for `install-mode` for that category of node. Since the property for that node’s category defaults to `AUTO`, the property for the `install-mode` of the node configuration defaults to “`AUTO (Category)`”. 5. The `install-mode` property of the node’s category. This can be set using Base View with a navigation path such as:\ `Grouping` > `Node categories[default]` > `Edit` > `Settings` > `Install mode` or using `cmsh` with a one-liner such as: ```bash cmsh -c "category use default; set installmode FULL; commit" ``` As already mentioned in a previous point, the install-mode is set by default to `AUTO`. 6. A dialog on the console of the node (figure [5.16](#fig-prov-install-mode)) gives the user a last opportunity to overrule the install-mode value as determined by the node-installer. By default, it is set to `AUTO`: ![Install-mode Setting Option During Node-Installer Run](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/99fe0117b2dfbccb5ee9df5485e73fa9f1c9d6ddd2753eaa5583795688022662/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/prov-installmode.png) *Figure 5.16: Install-mode Setting Option During Node-Installer Run* #### 5.4.4.4 FULL Install Confirmation via `datanode` Setting Related to execution mode values is the ability to carry out a FULL install only after explicit confirmation, via the `datanode` property. This must be set in order to prompt for a confirmation, when a FULL installation is about to take place. If it is set, then the node-installer only goes ahead with the FULL install after the administrator has explicitly confirmed it. The `datanode` property can be set in the node configuration of, for example, `node001` with Base View via the navigation path: `Devices` > `Nodes[node001]` > `Edit` > `Settings` > `Data node[Yes]` Alternatively, the parameter `datanode` can be set using a `cmsh` one-liner as follows: ```bash [root@basecm10 ~]# cmsh -c "device use node001; set datanode yes; commit" ``` The property can also be set at a category level. Since `datanode` is a boolean value, the actual value that is used used for a node is the result of the `or` operation for that value across the levels. The level at which a value works due to its boolean or non-boolean type is explained more [here](/admin/clustermanagement#2132-corresponding-category-values-and-node-values). **Why the FULL install confirmation is useful:** The reason for such a setting is that a FULL installation can be triggered by disk or partition changes, or by a change in the MAC address. If that happens, then: * considering a drive, say, `/dev/sda` that fails, this means that any drive `/dev/sdb` would then normally become `/dev/sda` upon reboot. In that case an unwanted FULL install would not only be triggered by an install-mode settings of FULL, but also by the install-mode settings of AUTO or NOSYNC. Having the new, “accidental” `/dev/sda` have a FULL install is unlikely to be the intention, since it would probably contain useful data that the node-installer earlier left untouched. * considering a node with a new MAC address, but with local storage containing useful data from earlier. In this case, too, an unwanted FULL install would not only be triggered by an install-mode setting of FULL, but also by the install-mode settings AUTO or NOSYNC. Thus, in cases where nodes are used to store data, an explicit confirmation before overwriting local storage contents is a good idea. However, by default, no confirmation is asked for when a FULL installation is about to take place. **Carrying out the confirmation:** When the confirmation is required, then it can be carried out by the administrator as follows: * From the node console. A remote console launched from Base View or `cmsh` will also work if SOL connectivity has been configured. * From `cmsh`, within `device` mode, using the `installerinteractions` command (some output elided): **Example** ```bash [basecm10->device]% installerinteractions -w -n node001 --confirm Hostname Action --------- --------------------------------------------- node001 Requesting FULL Install (partition mismatch) [basecm10->device]% ...07:57:36 [notice] basecm10: node001 [ INSTALLER_CALLINGINIT ]... [basecm10->device]% ...07:58:20 [notice] basecm10: node001 [ UP ] ``` The `installerinteractions` command then sets the node to a `confirmed` state. The other possible states are `deny` and `pending`. Besides confirmation, the `installerinteractions` command has options that include letting it: * deny the installation, and put it into maintenance mode * carry out a dry-run * carry out its actions for node groupings such as: node lists, node categories, node groups, chassis, racks, as are possible in the grouping options ([here](/admin/clustermanagement#2559-looping-over-objects-with-foreach)). Further details on the command can be viewed by running `help installerinteractions`. **An alternative way to avoid overwriting node storage:** Besides the method of FULL install confirmation for `datanode`, there is a method based on XML assertions, that can also be used to prevent data loss on nodes. It uses XML assertions to confirm that the physical drive is recognized (Appendix [D.11](/admin/partitioning#d11-example-preventing-accidental-data-loss)). **A way to overwrite a specified block device:** A related method is that sometimes, for reasons of performance or convenience, it may be desirable to clear data on particular block devices for a node, and carry it out during the next boot only. This can done by setting the block device names to be cleared as values to the parameter `Block devices cleared on next boot`. The values can be set in `cmsh` as follows: ```bash [basecm10->device[node001]]% append blockdevicesclearedonnextboot /dev/sda /dev/sdb ; commit ``` The value of `blockdevicesclearedonnextboot` is automatically cleared after the node is rebooted. Clearing data in this way ignores any `datanode` or `nextinstallmode` settings, and should therefore be used with due care. ### 5.4.5 Running Initialize Scripts An *initialize script* is used when custom commands need to be executed before checking partitions and mounting devices (section [3.20.4](/admin/configuring#3204-adding-functionality-to-nodes-via-an-initialize-or-finalize-script)). For example, to initialize some not explicitly supported hardware, or to do a RAID configuration lookup for a particular node. In such cases the custom commands are added to an `initialize` script. How to edit an `initialize` script is described in Appendix [E.2](/admin/finalize#e2-accessing-from-base-view-and-cmsh). An `initialize` script can be added to both a node’s category and the node configuration. The node-installer first runs an `initialize` script, if it exists, from the node’s category, and then an `initialize` script, if it exists, from the node’s configuration. The node-installer sets several environment variables which can be used by the `initialize` script. Appendix [E](/admin/finalize) contains an example script documenting these variables. Related to the `initialize` script is the `finalize` script (section [5.4.11](#5411-running-finalize-scripts)). This may run after node provisioning is done, but just before the `init` process on the node runs. ### 5.4.6 Checking Partitions, RAID Configuration, Mounting Filesystems #### 5.4.6.1 Behavior As Decided By The Install-Mode Value In section [5.4.4](#544-determining-install-mode-type-and-execution-mode) the node-installer determines the install-mode value, along with when to apply it to a node. **AUTO:** The install-mode value is typically set to default to `AUTO`. If `AUTO` applies to the current node, it means the node-installer then checks the partitions of the local drive and its filesystems and recreates them in case of errors. Partitions are checked by comparing the partition layout of the local drive(s) against the drive layout as configured in the node’s category configuration and the node configuration. After the node-installer checks the drive(s) and, if required, recreates the layout, it mounts all filesystems to allow the drive contents to be synchronized with the contents of the software image. **FULL, MAIN, or SKIP:** If install-mode values of `FULL`, `MAIN`, or `SKIP` apply to the current node instead, then no partition checking or filesystem checking is done by the node-installer. **NOSYNC:** If the install-mode value of `NOSYNC` applies, then if the partition and filesystem checks both show no errors, the node starts up without getting an image synced to it from the provisioning node. If the partition or the filesystem check show errors, then the node partition is rewritten, and a known good image is synced across. #### 5.4.6.2 Behavior As Decided By XML Configuration Settings The node-installer is capable of creating advanced drive layouts, including LVM setups, and hardware and software RAID setups. Drive layout examples and relevant documentation are in Appendix [D](/admin/partitioning). The XML description used to set the drive layouts can be deployed for a single device or to a category of devices. **Hardware RAID:** BCM supports hardware RAID levels 0, 1, 5, 10, and 50, and supports the following options: * **stripe size:** | **Option** | | :--------- | | 64kB | | 128kB | | 256kB | | 512kB | | 1024kB | * **cache policy:** | **Option** | | :--------- | | Cached | | Direct | * **read policy:** | **Option** | **Description** | | :--------- | :-------------- | | NORA | No Read Ahead | | RA | Read Ahead | | ADRA | Adaptive Read | * **write policy:** | **Option** | **Description** | | :--------- | :-------------- | | WT | Write Through | | WB | Write Back | ### 5.4.7 Synchronizing The Local Drive With The Software Image After having mounted the local filesystems, these can be synchronized with the contents of the software image associated with the node (through its category). Synchronization is skipped if the install-mode values of `NOSYNC` or `SKIP` are set, and takes place `FULL` or `AUTO` are set. Synchronization is delegated by the node-installer to the CMDaemon provisioning system. The node-installer just sends a provisioning request to CMDaemon on the head node. For an install-mode of `FULL`, or for an install-mode of `AUTO` where the local filesystem is detected as being corrupted, full provisioning is done. For an install-mode of `AUTO` where the local filesystem is healthy and agrees with that of the software image, sync provisioning is done. #### 5.4.7.1 The `lock`, `unlock`, And `islocked` Commands For Software Images The software image that is requested is available to nodes by default. Its availability can be altered and checked with the following commands: * `lock`: this *locks* an image so that the image cannot be provisioned until the image is *unlocked*. * `unlock`: this *unlocks* a locked image, so that request for provisioning the image is no longer prevented by a lock * `islocked`: this lists the locked or unlocked states of images. Locking an image is sometimes useful, for example, to make changes to an image when nodes are booting: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% softwareimage list Name (key) Path Kernel version Nodes -------------------- ---------------------------------------- ---------------------------- -------- default-image /cm/images/default-image 3.10.0-1062.12.1.el7.x86_64 3 [basecm10]% softwareimage [basecm10->softwareimage]% lock default-image [basecm10->softwareimage]% islocked Name Locked -------------- -------- default-image yes [basecm10->softwareimage]% lock default-image [basecm10->softwareimage]% device [basecm10->device]% reboot node001 ...the cluster administrator makes changes to the node image during a boot, as it waits for the image to unlock... [basecm10->device]% softwareimage unlock default-image ``` For an unlocked image, on receiving the provisioning request, CMDaemon assigns the provisioning task to one of the provisioning nodes. The node-installer is notified when image synchronization starts, and also when the image synchronization task ends—whether it is completed successfully or not. #### 5.4.7.2 Exclude Lists: `excludelistsyncinstall` And `excludelistfullinstall` What files are synchronized is decided by an *exclude list*. An exclude list is a property of the node category, and is a list of directories and files that are excluded from consideration during synchronization. The excluded list that is used is decided by the type of synchronization chosen: `full` or `sync`: * A `full` type of synchronization rewrites the partition table of the node, then copies the filesystem from a software image to the node, using a list to specify files and directories to exclude from consideration when copying over the filesystem. The list of exclusions used is specified by the `excludelistfullinstall` property. The intention of `full` synchronization is to allow a complete working filesystem to be copied over from a known good software image to the node. By default the `excludelistfullinstall` list contains `/proc/`, `/sys/`, and `lost+found/`, which have no content in BCM’s default software image. The list can be modified to suit the requirements of a cluster, but it is recommended to have the list adhere to the principle of allowing a complete working node filesystem to be copied over from a known good software image. * A `sync` type of synchronization uses the property `excludelistsyncinstall` to specify what files and directories to exclude from consideration when copying parts of the filesystem from a known good software image to the node. The `excludelistsyncinstall` property is in the form of a list of exclusions, or more accurately in the form of two sub-lists. The contents of the sub-lists specify the parts of the filesystem that should be retained or not copied over from the software image during `sync` synchronization when the node is booting. The intention behind this is to have the node boot up quickly, updating only the files from the image to the node that need updating due to the reboot of the node, and otherwise keeping files that are already on the node hard disk unchanged. The contents of the sub-lists are thus items such as the node log files, or items such as the `/proc` and `/sys` pseudo-filesystems which are generated during node boot. The administrator should be aware that nothing on a node hard drive can be regarded as persistent because a FULL sync takes place if any error is noticed during a partition or filesystem check. Anything already on the node that matches the content of these sub-lists is not overwritten by image content during an `excludelistsyncinstall` sync. However, image content that is not on the node is copied over to the node only for items matching the first sub-list. The remaining files and directories on the node, that is, the ones that are not in the sub-lists, lose their original contents, and are copied over from the software image. A `cmsh` one-liner to get an exclude list for a category is: ```bash cmsh -c "category use default; get excludelistfullinstall" ``` Similarly, to set the list: ```bash cmsh -c "category use default; set excludelistfullinstall; commit" ``` where a text-editor opens up to allow changes to be made to the list. In Base View the navigation path is: `Grouping` > `Node Categories` > `Edit` > `Node Category` > `Settings` > `Exclude list full install` Image synchronization is done using `rsync`, and the syntax of the items in the exclude lists conforms to the “`INCLUDE/EXCLUDE PATTERN RULES`” section of the `rsync(1)` man page, which includes patterns such as “`**`”, “`?`”, and “`[[:alpha:]]`”. The `excludelistfullinstall` and `excludelistsyncinstall` properties decide how a node synchronizes to an image during boot. For a node that is already fully up, the related `excludelistupdate` property decides how a running node synchronizes to an image without a reboot event, and is discussed in section [5.6](#56-updating-running-nodes). #### 5.4.7.3 Interface Used To Receive Image Data: `provisioninginterface` For regular nodes with multiple interfaces, one interface may be faster than the others. If so, it can be convenient to receive the image data via the fastest interface. Setting the value of `provisioninginterface`, which is a property of the node configuration, allows this. By default it is set to `BOOTIF` for regular nodes. Using `BOOTIF` is not recommended for node configurations with multiple interfaces. When listing the network interfaces in `cmsh`, the provisioning interface has a `[prov]` flag appended to its name. **Example** ```bash [basecm10->device[node001]->interfaces]% list Type Network device name IP Network ------------ -------------------- ---------------- ---------------- physical BOOTIF [prov] 10.141.0.1 internalnet physical eth1 10.141.1.1 internalnet physical eth2 10.141.2.1 internalnet ``` **Head nodes and `provisioninginterface`:** A head node in a single-head cluster does not use the `provisioninginterface` setting. Head nodes in a failover configuration (Chapter [17](/admin/ha)), however, do have a value set for `provisioninginterface`, corresponding to the interface on the head that is being provisioned over `internalnet` by the other head (`eth0` in figure [17.1](/admin/ha#1714-failover-network-topology)). #### 5.4.7.4 Transport Protocol Used For Image Data: `provisioningtransport` The `provisioningtransport` property of the node sets whether the image data is sent encrypted or unencrypted to the node from the provisioner. The property value is set via the `device` mode for the receiving node to one of these values: * `rsyncdaemon`, which sends the data unencrypted * `rsyncssh`, which sends the data encrypted The `provisioningtransport` value can be set for all nodes, including provisioning nodes, head nodes, and cloud-director ([section 3.2 of the *Cloudbursting Manual*](/cloudbursting/cexcloud#32-cluster-extension-with-aws-cloud-director-startup-from-scratch)) nodes. Because encryption severely increases the load on the provisioning node, using `rsyncssh` is only suggested if the users on the network cannot be trusted. By default, `provisioningtransport` is set to `rsyncdaemon`. If high availability (Chapter [17](/admin/ha)) is set up with the head nodes exposed to the outside world on the external network, the administrator should consider setting up `rsyncssh` for the head nodes. The `rsyncssh` transport requires passwordless root access via `ssh` from the provisioner to the node being provisioned. This is configured by default in the default BCM nodes. However, if a new image is created with the `--exclude` options for `cm-create-image` as explained in (section [11.6.2](/admin/softwareimagemanagement#1162-creating-the-software-image-with-cm-create-image)), the keys must be copied over from `/root/.ssh/` on the existing nodes. #### 5.4.7.5 Tracking The Status Of Image Data Provisioning: `provisioningstatus` The `provisioningstatus` command within the `softwareimage` mode of `cmsh` displays an updated state of the provisioning system. As a one-liner, it can be run as: ```bash basecm10:~ # cmsh -c "softwareimage provisioningstatus" Provisioning subsystem status: idle, accepting requests Update of provisioning nodes requested: no Maximum number of nodes provisioning: 10000 Nodes currently provisioning: 0 Nodes waiting to be provisioned: Provisioning node basecm10: Max number of provisioning nodes: 10 Nodes provisioning: 0 Nodes currently being provisioned: ``` The `provisioningstatus` command has several options that allow the requests to be tracked. The `-r` option displays the basic status information on provisioning requests, while the `-a` option displays all status information on provisioning requests. Both of these options display the request IDs. The Base View equivalent to `provisioningstatus` is accessed via the navigation path: `Provisioning` > `Provisioning nodes` By default, it displays basic status information on provisioning requests. #### 5.4.7.6 Tracking The Provisioning Log Changes: `synclog` For a closer look into the image file changes carried out during provisioning requests, the `synclog` command from `device` mode can be used (lines elided in the following output): **Example** ```bash [basecm10->device]% synclog node001 Tue, 11 Jan 2011 13:27:17 CET - Starting rsync daemon based provisioning. Mode is SYNC. sending incremental file list ./ ... deleting var/lib/ntp/etc/localtime var/lib/ntp/var/run/ntp/ ... sent 2258383 bytes received 6989 bytes 156232.55 bytes/sec total size is 1797091769 speedup is 793.29 Tue, 11 Jan 2011 13:27:31 CET - Rsync completed. ``` #### 5.4.7.7 Path Of The Provisioning Log File The path of the log file can be found with the `-p` option: **Example** ```bash [basecm10->device]% synclog -p node001 /var/spool/cmd/node001-\.rsync [basecm10->device]% ``` #### 5.4.7.8 Statistical Analysis Of Provisioning Sessions: `syncinfo` A provisioning session takes place between a provisioning image and a filesystem partition on a node. Statistics can be presented for the sessions using the `syncinfo` command. The statistical information presented is for number of files considered for transfer, the number of files that were actually transfered, how long the transfer took, which image and node were involved, and so on. The `syncinfo` command is run in device mode (output ellipsized and truncated): **Example** ```bash [head->device]% syncinfo Node Path Provisioner Age Duration Total files Transfered files ... ------- ------------------------ ----------- ---- -------- ----------- ---------------- ... node001 /cm/images/default-image head 34s 21s 171,504 328 ... node002 /cm/images/default-image head 34s 22s 171,504 328 ... ... ``` The `syncinfo` command has options to run it per node, category, rack, and so on. Details on the options can be seen by running the `help` command (`help syncinfo`). #### 5.4.7.9 Aborting Provisioning With `cancelprovisioningrequest` The `cancelprovisioningrequest` command cancels provisioning. Its usage is: `cancelprovisioningrequest [OPTIONS] [ ...]` To cancel all provisioning requests, it can be run as: ```bash basecm10:~ # cmsh -c "softwareimage cancelprovisioningrequest -a" ``` The `provisioningstatus` command of `cmsh`, can be used to find request IDs. Individual request IDs, for example `10` and `13`, can then be specified in the `cancelprovisioningrequest` command, as: ```bash basecm10:~ # cmsh -c "softwareimage cancelprovisioningrequest 10 13" ``` The help page for `cancelprovisioningrequest` shows how to run the command on node ranges, groups, categories, racks, chassis, and so on. The Base View equivalents to the `cmsh` versions for managing provisioning requests can be accessed via the navigation path `Provisioning` > `Provisioning Requests` ### 5.4.8 Writing Network Configuration Files In the previous section, the local drive of the node is synchronized according to install-mode settings with the software image from the provisioning node. The node-installer now sets up configuration files for each configured network interface. These are files like:\ `/etc/sysconfig/network-scripts/ifcfg-eth0`\ for Red Hat, Scientific Linux, CentOS, and Rocky Linux, while SUSE would use:\ `/etc/sysconfig/network/ifcfg-eth0`\ These files are placed on the local drive. When the node-installer finishes its remaining tasks (sections [5.4.9](#549-creating-a-local-etcfstab-file)–[5.4.13](#5413-switching-to-the-local-init-process)) it brings down all network devices and hands over control to the local `/sbin/init` process. Eventually a local `init` script uses the network configuration files to bring the interfaces back up. ### 5.4.9 Creating A Local `/etc/fstab` File The `/etc/fstab` file on the local drive contains local partitions on which filesystems are mounted as the `init` process runs. The actual drive layout is configured in the category configuration or the node configuration, so the node-installer is able to generate and place a valid local `/etc/fstab` file. In addition to all the mount points defined in the drive layout, several extra mount points can be added. These extra mount points, such as NFS imports, `/proc`, `/sys` and `/dev/shm`, can be defined and managed in the node’s category and in the specific configuration of the node configuration, using Base View or `cmsh` (section [3.13.2](/admin/configuring#3132-mounting-a-filesystem-using-base-view-and-cmsh)). ### 5.4.10 Booting From The Local Hard Drive By default, a node-installer boots from the software image on the head node via the network. The node-installer can, optionally, during image synchronization, install a local drive boot record on the local hard drive if the `installbootrecord` boolean property of the node configuration or node category is set to `on`. Setting the local drive boot record means that the node tries to use a local hard drive boot installer during the next boot. This is a step toward having it become a standalone node that does not boot from the network. This step, and the other steps needed to allow booting from the local hard drive are covered next. #### 5.4.10.1 Setting The Boot Record To Allow The Node To Be Standalone The local drive boot record is installed in the MBR of the local drive, overwriting the default iPXE boot record (section [5.1.2](#512-ipxe-booting-from-a-disk-drive)). With a working custom software image, the boot record can be installed with `cmsh` commands for a node `node001` with: ```bash cmsh -c "device use node001; set installbootrecord yes; commit" ``` or for a category `default` with: ```bash cmsh -c "category use default; set installbootrecord yes; commit" ``` Since `installbootrecord` is a boolean property, it means that if the node or if the node category have the value set, then the node uses that value. In Base View, the equivalent is the `Install boot record` option. This can similarly be enabled and saved in the Base View node configuration or node category. Setting the local drive boot record allows the next boot to be from the local hard drive, if the node is set up right to boot from the local hard drive. Booting from the local hard drive often requires some further changes, as explained next. #### 5.4.10.2 Managing Boot Sequence And Bootloader To Ensure The Node Can Be Standalone For a local hard drive boot to work: 1. hard drive booting must be set to have a higher priority than network booting in the BIOS of the node. Otherwise regular PXE booting is attempted, despite whatever value `installbootrecord` has. 2. A working bootloader must be present. By default, the node image for BCM has nodes set to use a SYSLINUX bootloader. If the administrator is not using the default software image, but is using a custom software image (section [11.6.1](/admin/softwareimagemanagement#1161-creating-a-base-distribution-archive-from-a-base-host)), and if the image is based on a running node filessystem that has not been built directly from a parent distribution, then the GRUB boot configuration may not be appropriate for a standalone GRUB boot to work. This is because the parent distribution installers often use special logic for setting up the GRUB boot configuration. Carrying out this same special logic for all distributions using the custom software image creation tool `cm-create-image` (section [11.6.2](/admin/softwareimagemanagement#1162-creating-the-software-image-with-cm-create-image)) is impractical. Providing a custom working image from a standalone node that has been customized after direct installation from the parent distribution, ensures the GRUB boot configuration layout of the custom image is as expected by the parent distribution. This then allows a standalone GRUB boot on the node to run properly. Nodes can be set to use a GRUB bootloader from within `device` mode, or from within `category` mode, by changing the `bootloader` parameter within the mode. For example, for a node `node001`: **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% device use node001 [basecm10->device[node001]]% get bootloader syslinux (default) [basecm10->device[node001]]% set bootloader grub [basecm10->device*[node001*]]% commit ``` or, for the `default` category: ```bash [root@basecm10 ~]# cmsh [basecm10]% category use default [basecm10->category[default]]% get bootloader syslinux [basecm10->category[default]]% set bootloader grub [basecm10->category*[default*]]% commit ``` Arranging for the two items in the preceding list ensures that the next boot is from GRUB on the hard drive. However, the `BOOTIF` also needs to be changed for booting to be successful. How it can be changed, and why it needs to be changed, is described next. #### 5.4.10.3 Changing `BOOTIF` To Ensure The Node Can Be Standalone If the BIOS is set to boot from the hard drive, and if there is a working boot loader, and if the boot record has been installed, then the node boots via the boot record on the hard drive. `BOOTIF` is the default value for the network interface for a node that is configured as a BCM software image. However, the `BOOTIF` interface is undefined during hard drive booting, because it depends on the network provisioning setup, which is not running. This means that the networking interface would fail during hard drive boot for a standard image. To remedy this, the interface should be set to a defined network device name, such as `eth0`, or the modern equivalents such as `en01` (section [5.8.1.1](#5811-interface-naming-conventions-post-rhel7-recommended)). The defined network device name, as the kernel sees it, can be found by logging into the node and taking a look at the output of `ip link`: **Example** ```bash [root@basecm10 ~]# ssh node001 ip link 1: lo: mtu 65536 qdisc noqueue... 2: eth0: mtu 1500 qdisc ... ``` In BCM the provisioning interface is mandatory, even if it is not provisioning. So it is set to the value of kernel-defined network device name instead of `BOOTIF`: **Example** ```bash [basecm10]% device interfaces node001 [basecm10->device[node001]->interfaces]% list Type Network device name IP Network ------------ -------------------- ---------------- ---------------- physical BOOTIF [prov] 10.141.0.1 internalnet [basecm10->device[node001]->interfaces]% set bootif networkdevicename eth0 [basecm10->device*[node001*]->interfaces*]% commit [basecm10->device[node001]->interfaces]% list Type Network device name IP Network ------------ -------------------- ---------------- ---------------- physical eth0 [prov] 10.141.0.1 internalnet Tue Mar 31 13:46:44 2020 [notice] basecm10: node001 [ UP ], restart required (eth0) ``` In the preceding example, the kernel-defined network device name is assumed to be `eth0`. It should be modified as required. In addition, the new IP address is assumed to be on the same internal network. If the administrator intends the node to be standalone on another network, then the network and the IP address can be set to appropriate values. When interface changes are carried out to make the node standalone, warnings show up saying that a reboot is required. A reboot of the node should be done when the interface configuration is complete. During reboot, the node then boots from the hard drive as a standalone, with a non-`BOOTIF` network interface. #### 5.4.10.4 Bringing A Node That Boots From Its Hard Drive Back Into A Cluster If the node is to be brought back into the cluster, then simply unsetting “Install boot record” and rebooting the node does not restore its iPXE boot record and hence its ability to iPXE boot. To restore the iPXE boot record, the node can be booted from the default image copy on the head node via a network boot again. Typically this is done by manual intervention during node boot to select network booting from the BIOS of the node. Setting the value of `provisioninginterface` in `cmsh` for the node to `BOOTIF` is also recommended. As suggested by the BCM iPXE boot prompt, setting network booting to work from the BIOS (regular “PXE” booting) is preferred to the relatively roundabout way of iPXE booting from the disk. #### 5.4.10.5 SELinux Initialization For Hard Drive Boot And PXE Boot If configured, SELinux ([Chapter 12 of the *Installation Manual*](/installation/selinux)) is initialized at this point. For a boot from the hard drive, the initialization occurs if an SELinux filesystem has been saved to disk previously. For a PXE boot, the initialization takes place if the `SELinuxInitialize` directive is set to `true` in the `node-installer.conf` file. ### 5.4.11 Running Finalize Scripts A *finalize script* is similar to an `initialize` script (section [5.4.5](#545-running-initialize-scripts)), only it runs a few stages later in the node-provisioning process. In the context of configuration (section [3.20.4](/admin/configuring#3204-adding-functionality-to-nodes-via-an-initialize-or-finalize-script)) it is used when custom commands need to be executed after the preceding mounting, provisioning, and housekeeping steps, but before handing over control to the node’s local `init` process. For example, custom commands may be needed to: * initialize some not explicitly supported hardware before `init` takes over * supply a configuration file for the software image that cannot simply be added to the software image and used by `init` because it needs node-specific settings * load a slightly altered standard software image on particular nodes, typically with the change depending on automatically detecting the hardware of the node it is being loaded onto. While this could also be done by creating a full new software image and loading it on to the nodes according to the hardware, it usually turns out to be better for simplicity’s sake (future maintainability) to minimize the number of software images for the cluster. The custom commands used to implement such changes are then added to the `finalize` script. How to edit a `finalize` script is described in Appendix [E.2](/admin/finalize#e2-accessing-from-base-view-and-cmsh). A `finalize` script can be added to both a node’s category and the node configuration. The node-installer first runs a `finalize` script, if it exists, from the node’s category, and then a `finalize` script, if it exists, from the node’s configuration. The node-installer sets several environment variables which can be used by the `finalize` script. Appendix [E](/admin/finalize) contains an example script which documents these variables. ### 5.4.12 Unloading Specific Drivers Many kernel drivers are only required during the installation of the node. After installation they are not needed and can degrade node performance. Baseboard Management Controllers (BMCs, section [3.7](/admin/configuring#37-configuring-bmc-ipmiilodraccimcredfish-interfaces)) that use IPMI drivers are an egregious example of this. The IPMI drivers are required to have the node-installer configure the IP address of any IPMI cards. Once the node is configured, these drivers are no longer needed, but they continue to consume significant CPU cycles and power if they stay loaded, which can affect job performance. To solve this, the node-installer can be configured to unload a specified set of drivers just before it hands over control to the local init process. This is done by editing the `removeModulesBeforeInit` setting in the node-installer configuration file\ `/cm/node-installer/scripts/node-installer.conf`, For the `node-installer.conf` file in multidistro and multiarch (section [11.7](/admin/softwareimagemanagement#117-creating-images-for-other-distributions-and-architectures-multidistro-and-multiarch)) configurations, the directory path `/cm/node-installer` takes the form:\ `/cm/node-installer-`*\-\*\ The values for \<*distribution*> and \<*architecture*> can take the values outlined [here](/admin/softwareimagemanagement#1171-the-cm-image-tool). By default, the IPMI drivers are placed in the `removeModulesBeforeInit` setting. To pick up IPMI-related data values, IPMI access is then carried out over the network without the drivers. ### 5.4.13 Switching To The Local `init` Process At this point the node-installer is done. The node’s local drive now contains a complete Linux installation and is ready to be started. The node-installer hands over control to the local `/sbin/init` process, which continues the boot process and starts all runlevel services. From here on the boot process continues as if the machine was started from the drive just like any other regular Linux machine. ## 5.5 Node States During the boot process, several state change messages are sent to the head node CMDaemon or detected by polling from the head node CMDaemon. The most important node states for a cluster after boot up are introduced in section [2.1.1](/admin/clustermanagement#211-devices). These states are described again, along with some less common ones to give a more complete picture of node states. ### 5.5.1 Node States Icons In Base View In the node icons used by Base View: * Nodes in the `UP` state are indicated by an up-arrow. * If all health checks (section [12.2.4](/admin/monitoring#1224-health-check)) for the node are successful, the up-arrow is green. * If there is a health check that fails or if the node requires a reboot, the up-arrow is red. * Nodes in the `DOWN` state are indicated by a blue down-arrow. * There are some other states, including: * Nodes in a `CLOSED` state are indicated by an X * Nodes in a `DOWN` state that are installing are indicated by a underscored down-arrow icon: ↓\_ ### 5.5.2 Node States Shown In `cmsh` In `cmsh`, the node state can be found using the `status` command from `device` mode for a node: **Example** ```bash [basecm10->device]% status -n node001..node002 node001 ............. [ UP ] restart-required, health check failed node002 ............. [ DOWN ] (hostname changed) restart-required ``` Devices in general can have their states conveniently listed with the `list -f` ([here](/admin/clustermanagement#2536-working-with-objects-list-format-sort)) command: **Example** ```bash [basecm10->device]% list -f "hostname:10, status:48" hostname ( status ---------- ------------------------------------------------ apc01 [ UP ] basecm10 [ UP ] devhp [ UP ] node001 [ UP ] restart-required, health check failed node002 [ DOWN ] (hostname changed) restart-required ``` The reason for a red icon as shown in section [5.5.1](#551-node-states-icons-in-base-view) can be found within the parentheses. In this example it is `(hostname changed)`. ### 5.5.3 Node States Indicating Regular Start Up During a successful boot process the node goes through the following states: * `BOOTING`. This is the state while the kernel and initrd are being downloaded by the node during network booting. To allow the `BOOTING` state to be detected for a node: * `BOOTIF` must be defined as an interface, or * if there is no interface with the value `BOOTIF`, but a particular network device, such as `eth0`, is the boot interface, then a special revision tag of `bootif` can be used for no more than one interface: **Example** ```bash [basecm10->device[node001]->interfaces[eth0]]% set revision bootif ``` * `INSTALLING`. This state is normally entered as soon as the node-installer has determined on which node the node-installer is running. Within this state, information messages display indicating what is being done while the node is in the `INSTALLING` state. Possible messages under the status column for the node within `cmsh` and Base View are normally, in sequence: 1. `node-installer started` 2. Optionally, the following two messages: 1. `waiting for user input` 2. `installation was resumed` 3. `checking disks` 4. `recreating partitions and filesystems` 5. `mounting disks` 6. One of these following two messages: 1. `waiting for FULL provisioning to start` 2. `waiting for SYNC provisioning to start` 7. `provisioning started, waiting for completion` 8. `provisioning complete` 9. `initializing SELinux` Between steps 1 and 3 in the preceding, these optional messages can also show up: * If `burn` mode is entered or left: * `running burn-in tests` * `burn-in test completed successfully` * If `maintenance` mode is entered: * `entered maintenance mode` * `INSTALLER_CALLINGINIT`. This state is entered as soon as the node-installer has handed over control to the local `init` process. The associated message normally seen with it in `cmsh` and Base View is: * `switching to local root` * `UP`. This state is entered as soon as the CMDaemon of the node connects to the head node CMDaemon. ### 5.5.4 Node States That May Indicate Problems Other node states are often associated with problems in the boot process: * `DOWN`. This state is registered as soon as the CMDaemon on the regular node is no longer detected by CMDaemon on the head node. In this state, the state of the regular node is still tracked, so that CMDaemon is aware if the node state changes. * `CLOSED`. This state is appended to the `UP` or `DOWN` state of the regular node by the administrator, and causes most CMDaemon monitoring actions for the node to cease. The state of the node is however still tracked by default, so that CMDaemon is aware if the node state changes. The `CLOSED` state can be set from the `device` mode of `cmsh` using the `close` command. The `help` text for the command gives details on how it can be applied to categories, groups and so on. The `-m` option sets a message by the administrator for the closed node or nodes. **Example** ```bash root@headymcheadface ~]# cmsh [headymcheadface]% device [headymcheadface->device]% close -m "fan dead" -n node001,node009,node020 Mon May 2 16:32:01 [notice] headymcheadface: node001 ...[ DOWN/CLOSED ] (fan dead) Mon May 2 16:32:01 [notice] headymcheadface: node009 ...[ DOWN/CLOSED ] (fan dead) Mon May 2 16:32:01 [notice] headymcheadface: node020 ...[ DOWN/CLOSED ] (fan dead) ``` The `CLOSED` state can also be set from Base View via the navigation path `Devices` > `Nodes` > `Edit` > `Monitored state` > `Open`/`Close`. When the `CLOSED` state is set for a device, CMDaemon commands can still attempt to act upon it. For example, in the `device` mode of `cmsh`: * `open`: This is the converse to the `close` command. It has the same options, including the `-m` option that logs a message. It also has the following extra options: * `--reset`: Resets whatever the status is of the `devicestatus` check. However, this reset by itself does not solve any underlying issue. The issue may still require a fix, despite the status having been reset. For example, the `--reset` option can be used to reset the restart-required flag (section [5.5.2](#552-node-states-shown-in-cmsh)). However, the reason that set the restart-required flag is not solved by the reset. Restarts are required for regular nodes if there have been changes in the following: network settings, disk setup, software image, or category. * `-f|--failbeforedown` \<*count*>: Specifies the number of failed pings before a device is marked as down (default is 1). * `drain` and `undrain` (Appendix [G.4.1](/admin/metricshealthchecksactions#g41-actions)) * For nodes that have power control: * `power -f on` * `power -f off` * `power -f reset` Power control mechanisms such as PDUs, custom power scripts, and BMCs using IPMI/HP iLO/DRAC/CIMC/Redfish, are described in Chapter [4](/admin/power). In Base View, the equivalents for a node `node001` for example, are via the navigation paths: * `Devices` > `Nodes[node001]` > `Edit` > `Monitored state` > `Open`/`Close` * `Devices` > `Nodes[node001]` > `Edit` > `Workload` > `Drain`/`Undrain` * `Devices` > `Nodes[node001]` > `Edit` > `Power` > `On/Off/Reset`. CMDaemon on the head node only maintains device monitoring logs for a device that is in the `UP` state. If the device is in a state other than `UP`, then CMDaemon only tracks its state, and can display the state if queried. For example: if a node displays the state `UP` when queried about its state, and is given a ‘close’ command, it then goes into a `CLOSED` state. Querying the node state then displays the state `UP/CLOSED`. It remains in that `CLOSED` state when the node is powered down. Querying the node state after being powered down displays `DOWN/CLOSED`. Next, powering up the node from that state, and having it go through the boot process, has the node displaying various `CLOSED` states during queries. Typically the query responses show it transitioning from `DOWN/CLOSED`, to `INSTALLING/CLOSED`, to `INSTALLER_CALLINGINIT/CLOSED`, and ending up displaying the `UP/CLOSED` state. Thus, a node set to a `CLOSED` state remains in a `CLOSED` state regardless of whether the node is in an `UP` or `DOWN` state. The only way out of a `CLOSED` state is for the administrator to tell the node to open via the `cmsh` “open” option discussed earlier. The node, as far as CMDaemon is concerned, then switches from the `CLOSED` state to the `OPEN` state. Whether the node listens or not does not matter—the head node records it as being in an `OPENING` state for a short time, and during this time the next `OPEN` state (`UP/OPEN`, `DOWN/OPEN`, etc.) is agreed upon by the head node and the node. When querying the state of a node, an `OPEN` tag is not displayed in the response, because it is the “standard” state. For example, `UP` is displayed rather than `UP/OPEN`. In contrast, a `CLOSED` tag is displayed when it is active, because it is a “special” state. The `CLOSED` state is normally set to take a node that is unhealthy out of the cluster management system. The node can then still be in the `UP` state, displaying `UP/CLOSED`. It can even continue running workload jobs in this state, since workload managers run independent of CMDaemon. So, if the workload manager is still running, the jobs themselves are still handled by the workload manager, even if CMDaemon is no longer aware of the node state until the node is re-opened. For this reason, draining a node is often done before closing a node, although it is not obligatory. * `OPENING`. This transitional state is entered as soon as the CMDaemon of the node rescinds the `CLOSED` state with an “open” command from `cmsh`. The state usually lasts no more than about 5 seconds, and never more than 30 seconds in the default configuration settings of BCM. The `help` text for the `open` command of `cmsh` gives details on its options. * `INSTALLER_FAILED`. This state is entered from the `INSTALLING` state when the node-installer has detected an unrecoverable problem during the boot process. For instance, it cannot find the local drive, or a network interface cannot be started. This state can also be entered from the `INSTALLER_CALLINGINIT` state when the node takes too long to enter the `UP` state. This could indicate that handing over control to the local `init` process failed, or the local `init` process was not able to start the CMDaemon on the node. Lastly, this state can be entered when the previous state was `INSTALLER_REBOOTING` and the reboot takes too long. * `INSTALLER_UNREACHABLE`. This state is entered from the `INSTALLING` state when the head node CMDaemon can no longer ping the node. It could indicate the node has crashed while running the node-installer. * `INSTALLER_REBOOTING`. In some cases the node-installer has to reboot the node to load the correct kernel. Before rebooting it sets this state. If the subsequent reboot takes too long, the head node CMDaemon sets the state to `INSTALLER_FAILED`. ## 5.6 Updating Running Nodes #### 5.6.0.1 Updating Running Nodes From A Stored Image By Rebooting Changes made to the contents of the software image for nodes, kept on the head node, become a part of any other provisioning nodes according to the housekeeping system on the head node (section [5.2.4](#524-provisioning-nodes-housekeeping)). Thus, when a regular node reboots, the latest image is installed from the provisioning system onto the regular node via a provisioning request (section [5.4.7](#547-synchronizing-the-local-drive-with-the-software-image)). #### 5.6.0.2 Updating Running Nodes From A Stored Image Without Rebooting However, updating a running node with the latest software image changes is also possible without rebooting it. Such an update can be requested using `cmsh` or Base View, and is queued and delegated to a provisioning node, just like a regular provisioning request. The properties that apply to the regular provisioning of an image also apply to such an update. For example, the value of the `provisioninginterface` setting (section [5.4.7.3](#5473-interface-used-to-receive-image-data-provisioninginterface)) on the node being updated determines which interface is used to receive the image. * In `cmsh` the request is submitted with the `imageupdate` option (section [5.6.2](#562-updating-running-nodes-with-cmsh-using-imageupdate)). * In Base View, it is submitted, for a node `node001` for example, using the navigation path: `Devices` > `Nodes[node001]` > `Edit` > `Software image` > `Update node` (section [5.6.3](#563-updating-running-nodes-with-base-view-using-the-update-node-option)). The `imageupdate` command and “`Update node`” menu option use a configuration file called `excludelistupdate`, which is, as its name suggests, a list of exclusions to the update. The running node is thus updated from a stored image with the help of that configuration file when `imageupdate` or “`Update node`” are run. More details are given in the rest of this section (section [5.6](#56-updating-running-nodes)). #### 5.6.0.3 Updating A Stored Image From A Running Node The converse, that is, to update a stored image from what is on a running node, can be also be carried out. This converse can be viewed as grabbing from a node, and synchronizing what is grabbed, to an image. It can be done using `grabimage` (`cmsh`), or `Grab to image` (Base View), and involves further exclude lists `excludelistgrab` or `excludelistgrabnew`. The `grabimage` command and `Grab to image` option are covered in detail in section [11.5.2](/admin/softwareimagemanagement#1152-managing-the-software-directly-on-a-node-then-syncing-node-to-image). ### 5.6.1 Updating Running Nodes: Configuration With `excludelistupdate` The exclude list `excludelistupdate` used by the `imageupdate` command is defined as a property of the node’s category. It has the same structure and `rsync` patterns syntax as that used by the exclude lists for provisioning the nodes during installation (section [5.4.7](#547-synchronizing-the-local-drive-with-the-software-image)). #### 5.6.1.1 Distinguishing Between The Intention Behind The Various Exclude Lists The administrator should note that it is the `excludelist`*update* list that is being discussed here, in contrast with the `excludelistsync`*install*/`excludelistfull`*install* lists which are discussed in section [5.4.7](#547-synchronizing-the-local-drive-with-the-software-image), and also in contrast with the `excludelist`*grab*/`excludelist`*grabnew* lists of section [11.5.2](/admin/softwareimagemanagement#1152-managing-the-software-directly-on-a-node-then-syncing-node-to-image). So, for the `imageupdate` command the `excludelistupdate` list concerns an *update* to a running system, while for installation `sync` or `full` provisioning, the corresponding exclude lists (`excludelistsyncinstall` and `excludelistfullinstall`) from section [5.4.7](#547-synchronizing-the-local-drive-with-the-software-image) are about an *install* during node start-up. Because the copying intention during updates is to be speedy, the `imageupdate` command synchronizes files rather than unnecessarily overwriting unchanged files. Thus, the `excludelistupdate` exclusion list it uses is actually analogous to the `excludelistsyncinstall` exclusion list used in the `sync` case of section [5.4.7](#547-synchronizing-the-local-drive-with-the-software-image), rather than being analogous to the `excludelistfullinstall` list. Similarly, the `excludelistgrab`/`excludelistgrabnew` lists of section [11.5.2](/admin/softwareimagemanagement#1152-managing-the-software-directly-on-a-node-then-syncing-node-to-image) are about a *grab* from the running node to the image. * The `excludelistgrab` list here is intended for the case of synchronizing the existing image with the running node, and is thus analogous to the `excludelistsyncinstall` exclusion list. * The `excludelistgrabnew` list here is intended for the case of copying a full image from the running node, and is thus analogous to the `excludelistfullinstall` list. The following table summarizes this:
**During:** **Exclude list used is:** **Copy intention:**
update `excludelistupdate` sync, image to running node
install `excludelistfullinstall` full, image to starting node
`excludelistsyncinstall` sync, image to starting node
grab `excludelistgrabnew` full, running node to image
`excludelistgrab` sync, running node to image
The preceding table is rather terse. It may help to understand it if is expanded with some in-place footnotes, where the footnotes indicate what actions can cause the use of the exclude lists:
**During:** **Exclude list used is:** **Copy intention:**
update `excludelistupdate` sync, image to running node
eg: imageupdate
install `excludelistfullinstall` full, image to starting node
eg: node-provisioning eg: node provisioning
process during pre- with installmode FULL
init stage depending
on installmode decision `excludelistsyncinstall` sync, image to starting node
eg: node provisioning AUTO
with healthy partition
grab `excludelistgrabnew` full, running node to image
eg: `grabimage` ( `cmsh` ), `grabimage -i` / `Grab to image` for a new image
`Grab to image`
(Base View) `excludelistgrab` sync, running node to image
`grabimage` / `Grab to image` for the original image
#### 5.6.1.2 The Exclude List Logic For `excludelistupdate` During an `imageupdate` command, the synchronization process uses the `excludelistupdate` list, which is a list of files and directories. One of the cross checking actions that may run during the synchronization is that the items on the list are excluded when copying parts of the filesystem from a known good software image to the node. The detailed behavior is as follows: The `exludelistupdate` list is in the form of two sublists. Both sublists are lists of paths, except that the second sublist is prefixed with the text “`no-new-files: `” (without the double quotes). For the node being updated, all of its files are looked at during an `imageupdate` synchronization run. During such a run, the logic that is followed is: * if an excluded path from `excludelistupdate` exists on the node, then nothing from that path is copied over from the software image to the node * if an excluded path from `excludelistupdate` does not exist on the node, then * if the path is on the first, non-prefixed list, then the path is copied over from the software image to the node. * if the path is on the second, prefixed list, then the path is not copied over from the software image to the node. That is, no new files are copied over, like the prefix text implies. This is illustrated by figure [5.17](#fig-cmgui-provisioning-synchronization-exclude-list-update-flowchart). ![Exclude list logic](/base-command-manager/product-manuals/_fern-img/080344ed55b5d132ae4928f4a7ad7e152d7a479590ebe37b4df1f3b1f5398f9b.webp) *Figure 5.17: Exclude list logic* The files and directories on the node that are not in the sub-lists lose their original contents, and are copied over from the software image. So, content not covered by the sub-lists at that time is normally not protected from deletion. Thus, the provisioning system excludes paths described according to the `excludelistupdate` property. The provisioning system also excludes a statically-imported filesystem on a node if the filesystem is a member of the following special list: NFS, Lustre, FUSE, CephFS, CIFS, PanFS, FhGFS, BeeGFS, GlusterFS, or GPFS. If this exclusion were not done, then all data on these imported filesystems would be wiped, since they are not part of the software image. The automatic exclusion for these imported filesystems does not rely on the excludelist values maintained by CMDaemon—instead, CMDaemon carries out the check on-the-fly when provisioning starts. Statically-imported filesystems that have their mounts managed by BCM via the `fsmounts` mode can be excluded from being mounted on the nodes in the first place, by removing them from the listed mounts within the `fsmounts` mode. Imported filesystems not on the special list can have their data wiped out during provisioning or sync updates, if the statically-imported filesystems are placed in the image manually—that is, if the filesystems are mounted manually into the image on the head node via `/etc/fstab` without using `cmsh` or Base View. **Filesystems mounted dynamically cannot have their appearance or disappearance detected reliably:** Any filesystem that may be imported via an auto-mount operation must therefore explicitly be excluded by the administrator manually adding the filesystem to the exclude list. This is to prevent an incorrect execution of `imageupdate`. Neglecting to do this may wipe out the filesystem, if it happens to be mounted in the middle of an `imageupdate` operation. **The `fstab` system is a statically mounting system, and not an auto-mounter:** While `fstab` mounts filesystems automatically, system administrators should not confuse that with auto-mounting. Auto-mounting as provided by `autofs` is designed for the dynamic mounting of filesystems on demand by regular users. The `fstab` table is designed for mounting as carried out by the system, as occurs during boot, which is why it is regarded as a static, non-auto-mounting system. #### 5.6.1.3 Editing An Exclude List A sample `cmsh` one-liner which opens up a text editor in a category so that the exclude list for updates can be edited is: ```bash cmsh -c "category use default; set excludelistupdate; commit" ``` Similarly, the exclude list for updates can also be edited in Base View via the navigation path: `Grouping` > `Node categories` > `Edit` > `Settings` > `Exclude list update` #### 5.6.1.4 Provisioning Modifications Via `excludelistmanipulatescript` Sometimes the administrator has a need to slightly modify the execution of exclude lists during provisioning. The `excludelistmanipulatescript` file takes as an input the exclude list inherited from a category, modifies it in some way, and then produces a new exclude list. Conceptually it is a bit like how an administrator might use `sed` if it worked without a pipe. As usual, [setting it for node level overrides the category level](/admin/clustermanagement#213-node-categories). A script that manipulates the exclude lists of a node can be specified as follows within `cmsh`: ```bash [basecm10]% device use node001 [basecm10->device[node001]]% set excludelistmanipulatescript (a vi session will start. A script is edited and saved) [basecm10->device[node001*]]% commit ``` The script can be as simple as: **Example** ```bash #!/bin/bash echo "- *" echo 'no-new-files: - *' ``` If provisioning a node from the head node, then the script modifies the node-provisioning exclude lists—`excludelistfullinstall`, `excludelistsyncinstall`, and `excludelistupdate`—so that they appear to contain these items only: ```bash - * no-new-files: - * ``` The provisioning node then excludes everything during provisioning. Careful editing and testing of the script is advised. Saving a script with just a single whitespace, for example, usually has undesirable results. A more useful script template is the following: **Example** ```bash #!/bin/bash while read; do echo "$REPLY" done echo "# This and next line added by category excludelistmanipulatescript." echo "# The command line arguments were: $@" ``` The provisioning exclude lists are simply read in, then sent out again without any change, except that the last two lines add comment lines to the exclude lists that are to be used. Internally, the arguments taken by the `excludelistmanipulatescript` are the destination path and the sync mode (one of `install`|`update`|`full`|`grab`|`grabnew`). This can be seen in the output of `$@`, if running an `imageupdate` command to execute a dry run with the preceding example: ```bash [basecm10]% device use node001 [basecm10->device[node001]]% get excludelistmanipulatescript (the script is put in) [basecm10->device[node001*]]% commit; imageupdate Performing dry run (use synclog command to review result, then pass -w to perform real update)... Wed Apr 15 04:55:46 2015 [notice] basecm10: Provisioning started: sendi\ ng basecm10:/cm/images/default-image to node001:/, mode UPDATE, dry run\ = yes, no data changes! [basecm10->device[node001]]% Wed Apr 15 04:55:51 2015 [notice] basecm10: Provisioning completed: sen\ t basecm10:/cm/images/default-image to node001:/, mode UPDATE, dry run \ = yes, no data changes! imageupdate [ COMPLETED ] ``` An excerpt from the sync log, after running the `synclog` command, then shows output similar to (some output elided): ```bash ... - /cm/shared/* - /cm/shared/ - /home/* - /home/ - /cm/shared/apps/slurm/* - /cm/shared/apps/slurm/ # This and next line added by category excludelistmanipulatescript. # The command line arguments were: update / Rsync output: sending incremental file list cm/local/apps/cmd/scripts/healthchecks/configfiles/ ... ``` Here, the sync mode is `update` and the destination path is “`/`”. Which of the exclude lists is being modified can be determined by the `excludelistmanipulatescript` by parsing the sync mode. The bash variable that accepts the exclude list text is set to a safely-marked form using curly braces. This is done to avoid expansion surprises, due to wild card characters in the exclude lists. For example, if `$REPLY` were used instead of `${REPLY}`, and the script were to accept an exclude list line containing “`- /proc/*`”, then it would give quite confusing output. #### 5.6.1.5 Other Exclude List Handling Options **The `excludelistfailover` and `excludelistnormal` files:** are two further exclude list files that modify standard provisioning behavior. These are discussed in section [17.4.8.1](/admin/ha#17481-exclude-lists-and-cloning). **The `excludelistsnippets` tool:** When synchronizing to a cloud director, or to an edge director, it is sometimes useful to exclude unneeded files and paths from the synchronization, in order to speed it up. The `excludelistsmanipulatescript` tool is powerful enough to do it, but it has some issues due to its power. For example, it is a script, which means that it is called whenever it is used, and so uses up some extra resources. Also, it is a bit tricky to set up. An easier way to manipulate exclude lists for the unneeded files and paths is via [the `excludelistsnippets` tool, described in section 4.4.1 of the *Cloudbursting Manual*](/cloudbursting/clicloud#441-setting-exclude-lists-with-excludelistsnippets). This tool allows additional exclusion to be specified in a simpler way. #### 5.6.1.6 Exclude List State At Node Level **Exclude lists at category level and node level:** An exclude list can be set at node level, as well as at category level. Roles and overlays can add implied exclude lists too. At category level, an exclude list such as `excludelistfullinstall` can be set up explicitly with: **Example** ```bash [basecm10->category[default]]% set excludelistfullinstall ...a text editor such as vi opens up and the list can be edited... [basecm10->category*[default*]]% commit ``` At node level, an exclude list can be set in the same way: **Example** ```bash [basecm10->device[node001]]% set excludelistfullinstall ...a text editor such as vi opens up and the list can be edited... [basecm10->device*[node001*]]% commit ``` An exclude list that is not empty at node level overrules its corresponding category list. Exclude lists brought in via roles are however simply included in the exclude list. **The `excludelist` command:** At node level it can be unclear what the resulting exclude list (“operational exclude list”) actually is. The exclude list state at node level can therefore be viewed using the `excludelist` command options. The `excludelist` command becomes active if a software image has been set at the node level. * The `list` option to `excludelist` lists the source and destination paths: **Example** ```bash [basecm10->device[node001]]% excludelist list Source path (on the head node) Destination path (on the node) -------------------------------- -------------------------------- /cm/images/default-image / ``` * The `get` option to the `excludelist` command has synchronization mode and destination suboptions for a node. Earlier on ([here](#5611-distinguishing-between-the-intention-behind-the-various-exclude-lists)), the intention behind the various exclude lists, according to the type of update or synchronization, were distinguished. The `excludelist get` command can have a destination path specified, and have the type of update or synchronization specified according to those distinguishing concepts. The output to the `excludelist get` command then shows the operational exclude list as seen by a node for that path and for that update or synchronization. Thus, for example: * The `full` install operational exclude list for the path `/` on node `node001`, intended for a full installation to a node that is starting up, can be found as follows: **Example** ```bash [basecm10->device[node001]]% excludelist get full / # For details on the exclude patterns defined here please refer to # the FILTER RULES section of the rsync man page. # # Files that match these patterns will not be installed onto the node. - lost+found/ - /proc/* - /sys/* - /boot/efi # extra defaults - /proc/* - /sys/* ``` * Similarly, the `sync` install operational exclude list for the path `/` on node `node001`, intended for a sync installation to a node that is starting up, can be found as follows: ```bash [basecm10->device[node001]]% excludelist get sync / # For details on the exclude patterns defined here please refer to # the FILTER RULES section of the rsync man page. # # Files that exist on a node and match one of these patterns will not be # modified or deleted. Any files that match one of these patterns and that # exist in the image but are absent on the node, will be copied to the node. - /.autofsck - /boot/grub*/grub.cfg - /cm/local/apps/openldap/etc/certs/ldap.key - /cm/local/apps/openldap/etc/certs/ldap.pem - /data/* - /home/* ... ``` * Other `excludelist get` options, besides `full` and `sync`, are: * `grab` (a grab from a running node for a sync back to an existing image) * `grabnew` (a grab from a running node for a full install to a new image) * `update` (a sync update of a running node from an image). All `excludelist get` options correspond to the intentions of the associated exclude list types as distinguished [here](#5611-distinguishing-between-the-intention-behind-the-various-exclude-lists). ### 5.6.2 Updating Running Nodes: With `cmsh` Using `imageupdate` Using a defined `excludelistupdate` property (section [5.6.1](#561-updating-running-nodes-configuration-with-excludelistupdate)), the `imageupdate` command of `cmsh` is used to start an update on a running node: **Example** ```bash [basecm10->device]% imageupdate -n node001 Performing dry run (use synclog command to review result, then pass -w to perform real update)... Tue Jan 11 12:13:33 2011 basecm10: Provisioning started on node node001 [basecm10->device]% imageupdate -n node001: image update in progress ... [basecm10->device]% Tue Jan 11 12:13:44 2011 basecm10: Provisioning completed on node node001 ``` By default the `imageupdate` command performs a dry run, which means no data on the node is actually written. Before passing the “`-w`” switch, it is recommended to analyze the rsync output using the `synclog` command (section [5.4.7.6](#5476-tracking-the-provisioning-log-changes-synclog)). If the user is now satisfied with the changes that are to be made, the `imageupdate` command is invoked again with the “`-w`” switch to implement them: **Example** ```bash [basecm10->device]% imageupdate -n node001 -w Provisioning started on node node001 node001: image update in progress ... [basecm10->device]% Provisioning completed on node node001 ``` ### 5.6.3 Updating Running Nodes: With Base View Using the `Update node` Option In Base View, an image update can be carried out by selecting the specific node or category, for example `node001`, and updating it via the navigation path: `Devices` > `Nodes[node001]` > `Edit` > `Software image` > `Update node` ### 5.6.4 Updating Running Nodes: Considerations An attempt to update the image on a running node can run into some issues: * Updating an image via `cmsh` or Base View automatically updates the provisioners first via the `updateprovisioners` command (section [5.2.4.4](#5244-provisioning-role-change-notification-with-updateprovisioners)) if the provisioners have not been updated in the last 5 minutes. The conditional update period can be set with the `dirtyautoupdatetimeout` parameter (section [5.2.4.6](#5246-provisioning-node-update-safeguards-and-dirtyautoupdatetimeout)). So, with the default setting of 5 minutes, if there has been a new image created within the last 5 minutes, then provisioners do not get the updated image when doing the updates, which means that nodes in turn do not get those updates. Running the `updateprovisioners` command just before running the `imageupdate` command therefore usually makes sense. * By default, BCM does not allow provisioning if automount ([here](/admin/cmdconf#c107-allowimageupdatewithautomount-directive)) is running. * Also, when updating services, the services on the nodes may not restart since the `init` process may not notice the replacement. For these reasons, especially for more extensive changes, it can be safer for the administrator to simply reboot the nodes instead of using `imageupdate` to provision the images to the nodes. A reboot by default ensures that a node places the latest image with an `AUTO` install (section [5.4.7](#547-synchronizing-the-local-drive-with-the-software-image)), and restarts all services. The `Reinstall node` option, which can be run, for example, on a node `node001`, using a navigation path of `Devices` > `Nodes[node001``]` > `Edit` > `Software image` > `Reinstall node` also does the same as a reboot with default settings, except for that it unconditionally places the latest image with a `FULL` install, and so may take longer to complete. ## 5.7 Adding New Nodes How the administrator can add a single node to a cluster is described in [section 1.3 of the *Installation Manual*](/installation/quickstart#13-booting-regular-nodes). This section explains how nodes can be added in ways that are more convenient for larger numbers of nodes. ### 5.7.1 Adding New Nodes With `cmsh` And Base View Add Functions Node objects can be added from within the `device` mode of `cmsh` by running the `add` command: **Example** ```bash [basecm10->device]% add physicalnode node002 10.141.0.2 [basecm10->device*[node002*]% commit ``` The Base View equivalent of this is following the navigation path: `Devices` > `Nodes` > `ADD` > `PhysicalNode`\[Settings] > `Hostname` then adding the value `node002` to `Hostname`, and saving it. When adding the node objects in `cmsh` and Base View, some values (the MAC addresses for example) may need to be filled in before the object validates. For regular nodes, there should be an interface and an IP address for the network that it boots from, as well as for the network that manages the nodes. A regular node typically has only one interface, which means that the same interface provides boot and management services. This interface is then the boot interface, BOOTIF, during the pre-init stage, but is also the management interface, typically `eth0` or whatever the device is called, after the pre-init stage. The IP address for BOOTIF is normally provided via DHCP, while the IP address for the management interface can be set to a static IP address via `cmsh` or Base View by the administrator. Adding new node objects as “placeholders” can also be done from `cmsh` or Base View. By placeholders, here it is meant that an incomplete node object is set. For example, sometimes it is useful to create a node object with the MAC address setting unfilled because it is still unknown. Why this can be useful is covered shortly. ### 5.7.2 Adding New Nodes With The Node Creation Wizard Besides adding nodes using the `add` command of `cmsh` or the `ADD` button of Base View as in the preceding text, there is also a Base View wizard that guides the administrator through the process—the *node creation wizard*. This is useful when adding many nodes at a time. It is available via the navigation path: `Devices` > `Nodes` > `CREATE NODES` This wizard should not be confused with the closely-related node *identification* resource described in section [5.4.2.4](#5424-node-identification), which identifies unassigned MAC addresses and switch ports, and helps assign them node names. * The node *creation* wizard creates an object for nodes, assigns them node names, but it leaves the MAC address field for these nodes unfilled, keeping the node object as a “placeholder”. * The node *identification* resource assigns MAC addresses so that node names are associated with a MAC address. If a node is left with an unassigned MAC address—that is, in a “placeholder” state—then it means that when the node starts up, the provisioning system lets the administrator associate a MAC address and switch port number at the node console for the node. This occurs when the node-installer reaches the node configuration stage during node boot as described in section [5.4.2](#542-deciding-or-selecting-node-configuration). This is sometimes preferable to associating the node name with a MAC address remotely with the node identification resource. In the first screen of the node creation wizard, IP address range suggestions are displayed for the new placeholder nodes. The administrator can override the range. The same screen also allows a category to be selected for the nodes (figure [5.18](#fig-bright-view-provisioning-node-creation-wizard-interface)). ![Node Creation Wizard: Setting Interfaces](/base-command-manager/product-manuals/_fern-img/45cbff12fcefe799907908923d4a5942b8eb5e1a9361847b493e98957a3984e2.webp) *Figure 5.18: Node Creation Wizard: Setting Interfaces* The remaining screens of the wizard configure the interface assignment and executes the object creation. Once the object has been created, node identification (section [5.4.2.4](#5424-node-identification)) can be carried out. The `cmsh` equivalent of the node creation wizard is running `foreach --clone` on a node that is to be cloned over a node range (section [2.5.5.9](/admin/clustermanagement#2559-looping-over-objects-with-foreach), [here](/admin/clustermanagement#2559-looping-over-objects-with-foreach)). ## 5.8 Troubleshooting The Node Boot Process During the node boot process there are several common issues that can lead to an unsuccessful boot. This section describes these issues and their solutions. It also provides general hints on how to analyze boot problems. Before looking at the various stages in detail, the administrator may find that simply updating software or firmware may fix the issue. In general, it is recommended that all available updates are deployed on a cluster. * Updating software is covered in Chapter [11](/admin/softwareimagemanagement). * On the head node, the most relevant software can be updated with `yum`, `zypper`, or `apt`, as explained in section [11.2](/admin/softwareimagemanagement#112-managing-packages-on-the-head-node). For example, with `yum`: **Example** ```bash yum update cmdaemon node-installer ``` * Similarly for the software image, the most relevant software can be updated too. This is done via a procedure involving a chroot installation., as described in section [11.4](/admin/softwareimagemanagement#114-managing-a-package-in-a-software-image-and-running-it-on-nodes). If using `yum`, then the update can be carried out within the image, \<*`software image`*>, with: **Example** ```bash yum update --installroot=/cm/images/ cmdaemon node-installer-slave ``` * UEFI or BIOS firmware should be updated as per the vendor recommendation The various stages that may fail during node boot are now examined. ### 5.8.1 Node Fails To PXE Boot Possible reasons to consider if a node is not even starting to network boot (PXE boot for x86 nodes) in the first place: * DHCP may not be running. A check can be done to confirm that DHCP is running on the internal network interface (usually eth0): ```bash [root@basecm10 ~]# ps u -C dhcpd USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 2448 0.0 0.0 11208 436 ? Ss Jan22 0:05 /usr/sbin/dhcpd eth0 ``` This may indicate that `Node booting` is disabled in Base View (figure [3.5](/admin/configuring#321-configuring-networks), [here](/admin/configuring#fig-bright-view-network-settings)) and needs to be enabled. The equivalent in `cmsh` is to check if the response to: ```bash cmsh -c "network use internalnet; get nodebooting" ``` needs to be set to `yes`. * The DHCP daemon may be “locked down” (section [3.2.1](/admin/configuring#321-configuring-networks): figure [3.5](/admin/configuring#321-configuring-networks) and table [3.1](/admin/configuring#tablenetwork-configuration-settings)). New nodes are granted leases only after `lockdowndhcpd` is set to `no` in `cmsh`, or `Lock down dhcpd` is disabled in Base View for the network. * A rogue DHCP server may be running. If there are all sorts of other machines on the network the nodes are on, then it is possible that there is a rogue DHCP server active on it, perhaps on an IP address that the administrator has forgotten, and interfering with the expected PXE booting. Such stray DHCP servers should be eliminated. * One way to identify the problem is to remove all the connections and switches and just connect the head node directly to a problem node, NIC-to-NIC. This should allow a normal network boot to happen. If a normal network boot then does happen, it indicates the problem is indeed due to a rogue DHCP server on the more-connected network. * For a more cerebral approach, which avoids recabling, the `nmap` utility may be useful. The `nmap` utility since version 7.90 can discover and list multiple DHCP servers using its `broadcast-dhcp-discover` script. The following session output shows the configuration and installation of the utility on to node002 on the internal network. It then runs it for the internal network interface `ens3`. If it finds a second DHCP server on the network (in this test case on node001 at 10.141.0.1), then it may show responses in the output similar to the following (some output ellipsized): **Example** ```bash [root@node002 ~]# wget https://nmap.org/dist/nmap-7.90.tgz ... [root@node002 ~]# tar xvzf nmap-7.90 ... [root@node002 ~]# cd nmap-7.90 [root@node002 nmap-7.90]# make distclean && ./configure --disable-rdma && make ... [root@node002 nmap-7.90]# ./nmap --script broadcast-dhcp-discover -e ens3 Starting Nmap 7.90 ( https://nmap.org ) at 2022-09-08 16:55 CEST Pre-scan script results: | broadcast-dhcp-discover: | Response 1 of 2: | IP Offered: 10.141.163.254 | DHCP Message Type: DHCPOFFER | Server Identifier: 10.141.0.1 ... | Response 2 of 2: | IP Offered: 10.141.167.255 | DHCP Message Type: DHCPOFFER | Server Identifier: 10.141.255.254 ... [root@node002 nmap-7.90]# ``` * The boot sequence may be set wrongly in the BIOS. The boot interface should normally be set to be the first boot item in the BIOS. * The node may be set to boot from UEFI mode. If UEFI mode has a buggy network boot implementation, then it may fail to network boot. For x86 nodes, setting the node to PXE boot using the legacy BIOS mode can be tried instead, or perhaps the UEFI firmware can be updated. * There may a bad cable connection. This can be due to moving the machine, or heat creep, or another physical connection problem. Firmly inserting the cable into its slot may help. Replacing the cable or interface as appropriate may be required. * There may a problem with the switch. Removing the switch and connecting a head node and a regular node directly with a cable can help troubleshoot this. Disabling the Spanning Tree Protocol (STP) functions of a managed switch is recommended. With STP on, nodes may randomly fail to network boot. * The cable may be connected to the wrong interface. By default, on the head node, for a type 1 network, the first consistent network device name, for example `eno1`, is normally assigned the internal network interface, and the second one, for example `en02`, is assigned the external network interface. However, the following possibilities should be considered during troubleshooting: * The two interfaces can be confused when physically viewing them and a connection to the wrong interface can therefore be made. * It is also possible that the administrator has changed the default assignment. * The interface may have been set by the administrator to follow the network device naming scheme that has been used prior to RHEL7. Interfaces with names such as `eth0` and `eth1` on the head node are suggestive of this. The problem with the pre-RHEL7 scheme is that it can sometimes lead to network interfaces swapping after reboot, which is why the scheme is no longer recommended. The workaround for this issue in pre-RHEL7 schemes was to define a persistent name in the udev ruleset for network interfaces. From NVIDIA Base Command Manager version 9.0 onward, the default scheme is the consistent network device naming scheme, and it is recommended. #### 5.8.1.1 Interface Naming Conventions Post-RHEL7 (Recommended) [https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Networking_Guide/ch-Consistent_Network_Device_Naming.html](https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Networking_Guide/ch-Consistent_Network_Device_Naming.html) describes the consistent network device scheme for interfaces post-RHEL7. This scheme sets an interface assignment on iPXE boot for multiple interfaces that is also valid by default during the very first iPXE boot. This means that an administrator can know which interface is used for provisioning and can connect the provisioning cable accordingly. Some care may need to be taken in unusual naming assignments, in order to avoid exceeding the 16-character limit that Linux has for the naming of network interfaces. #### 5.8.1.2 Reverting To The Pre-RHEL7 Interface Naming Conventions (Not Recommended) To revert to the pre-RHEL7 behavior, the text: ```bash net.ifnames=0 biosdevname=0 ``` can be appended to the line starting with `GRUB_CMDLINE_LINUX` in `/etc/default/grub` within the head node. For this: * The `biosdevname` parameter only works if the dev helper is installed. The dev helper is available from the `biosdevname` RPM package. The parameter also requires that the system supports SMBIOS 2.6 or ACPI DSM. * The `net.ifnames` parameter is needed if `biosdevname` is not installed. **Example** ```bash GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/swap vconsole.keymap=us \ crashkernel=auto rd.lvm.lv=centos/root vconsole.font=latarcyr\ heb-sun16 rhgb quiet net.ifnames=0 biosdevname=0" ``` A cautious system administrator may back up the original `grub.cfg` file: ```bash [root@basecm10 ~]# cp --preserve /boot/grub2/grub.cfg /boot/grub2/grub.cfg.orig ``` The GRUB configuration should be generated with: ```bash [root@basecm10 ~]# grub2-mkconfig -o /boot/grub2/grub.cfg ``` If for some reason the administrator would like to carry out the pre-RHEL7 naming convention on a regular node, then the text `net.ifnames=0 biosdevname=0` can be appended to the `kernelparameters` property, for an image selected from `softwareimage` mode. **Example** ```bash [basecm10->softwareimage]% list Name (key) Path -------------------- ------------------------------ default-image /cm/images/default-image openstack-image /cm/images/openstack-image [basecm10->softwareimage]% use default-image [basecm10->softwareimage[default-image]]% append kernelparame\ ters " net.ifnames=0 biosdevname=0" [basecm10->softwareimage*[default-image*]]% commit ``` The `append` command requires a space at the start of the quote, in order to separate the kernel parameters from any pre-existing ones. * The TFTP server that sends out the image may have hung. During a normal run, an output similar to this appears when an image is in the process of being served: ```bash [root@basecm10 ~]# ps ax | grep [t]ftp 7512 ? Ss 0:03 in.tftpd --maxthread 500 /tftpboot ``` If the TFTP server is in a zombie state, the head node should be rebooted. If the TFTP service hangs regularly, there is likely a networking hardware issue that requires resolution. Incidentally, grepping the process list for a TFTP service returns nothing when the head node is listening for TFTP requests, but not actively serving a TFTP image. This is because the TFTP service runs under xinet.d and is called on demand. Running ```bash [root@basecm10 ~]# chkconfig --list ``` should include in its output the line: ```bash tftp: on ``` if TFTP is running under xinet.d. * The switchover process from TFTP to HTTP may have hung. During a normal provisioning run, assuming that CMDaemon uses the default `bootloaderprotocol` setting of `HTTP`, then TFTP is used to load the initial boot loader, but the kernel and ramdisk are loaded up via HTTP for speed. Some hardware has problems with switching over to using HTTP. In that case, setting `bootloaderprotocol` to `TFTP` keeps the node using TFTP for loading the kernel and ramdisk, and should work. Another possible way to solve this is to upgrade the PXE boot BIOS to a version that does not have this problem. ARMv8 hardware can boot only via `TFTP`. Setting `bootloaderprotocol` to `HTTPS` only works for some special hardware. * VLAN tagging may have been set up incorrectly in the BIOS of the node. VLAN provisioning (section [5.3.4](#534-vlan-provisioning)) requires several changes in VLAN configuration for it to work. * Sometimes a manufacturer releases hardware with buggy drivers that have a variety of problems. For instance: Ethernet frames may be detected at the interface (for example, by `ethtool`), but TCP/IP packets may not be detected (for example, by `wireshark`). In that case, the manufacturer should be contacted to upgrade their driver. * The interface may have a hardware failure. In that case, the interface should be replaced. ### 5.8.2 Node-installer Logging If the node manages to get beyond the net booting stage to the node-installer stage, then the first place to look for hints on node boot failure is usually the node-installer log file. The node-installer runs on the node that is being provisioned, and sends logging output to the syslog daemon running on that node. This forwards all log data to the IP address from which the node received its DHCP lease, which is typically the IP address of the head node or failover node. In a default BCM setup, the `local5` facility of the `syslog` daemon is used on the node that is being provisioned to forward all node-installer messages to the log file `/var/log/node-installer` on the head node. After the node-installer has finished running, its log is also stored in `/var/log/node-installer` on the regular nodes. If there is no node-installer log file anywhere yet, then it is possible that the node-installer is not yet deployed on the node. Sometimes this is due to a system administrator having forgotten to change a provisioning-related configuration setting. One possibility is that the `nodegroups` setting (section [5.2.1](#521-provisioning-nodes-configuration-settings)), if used, may be misconfigured. Another possibility is that the image was set to a `locked` state (section [5.4.7](#547-synchronizing-the-local-drive-with-the-software-image)). The `provisioningstatus -a` command can indicate this: **Example** ```bash [basecm10->softwareimage]% provisioningstatus -a | grep locked Scheduler info: requested software image is locked, request deferred ``` To get the image to install properly, the `locked` state should be removed for a locked image. **Example** ```bash [root@basecm10 ~]# cmsh -c "softwareimage islocked" Name Locked -------------- -------- default-image yes [root@basecm10 ~]# cmsh -c "softwareimage unlock default-image" [root@basecm10 ~]# cmsh -c "softwareimage islocked" Name Locked -------------- -------- default-image no ``` The node automatically picks up the image after it is unlocked. Optionally, extra log information can be written by enabling debug logging, which sets the syslog importance level at `LOG_DEBUG`. To enable debug logging, the debug field is changed in `/cm/node-installer/scripts/node-installer.conf`. For the `node-installer.conf` file in multidistro and multiarch (section [11.7](/admin/softwareimagemanagement#117-creating-images-for-other-distributions-and-architectures-multidistro-and-multiarch)) configurations, the directory path `/cm/node-installer` takes the form:\ `/cm/node-installer-`*\-\*\ The values for \<*distribution*> and \<*architecture*> can take the values outlined [here](/admin/softwareimagemanagement#1171-the-cm-image-tool). From the console of the booting node the log file is generally accessible by pressing `Alt+F7` on the keyboard. Debug logging is however excluded from being viewed in this way, due to the output volume making this impractical. A booting node console can be accessed remotely if Serial Over LAN (SOL) is enabled (section [16.7](/admin/administration#167-serial-over-lan-console-access)), to allow the viewing of console messages directly. A further depth in logging can be achieved by setting the kernel option `loglevel=N`, where `N` is a number from `0` (`KERN_EMERG`) to `7` (`KERN_DEBUG`). One possible point at which the node-installer can fail on some hardware is if SOL (section [16.7](/admin/administration#167-serial-over-lan-console-access)) is enabled in the BIOS, but the hardware is unable to cope with the flow. The installation can freeze completely at that point. This should not be confused with the viewing quirk described in section [16.7.4](/admin/administration#1674-the-conman-serial-console-logger-and-viewer), even though the freeze typically appears to take place at the same point, that point being when the console shows “`freeing unused kernel memory`” as the last text. One workaround to the freeze would be to disable SOL. ### 5.8.3 Provisioning Logging The provisioning system sends log information to the CMDaemon log file. By default this is in `/var/log/cmdaemon` on the local host, that is, the provisioning host. The host this log runs on can be configured with the CMDaemon directive `SyslogHost` (Appendix [C.73](/admin/cmdconf#c73-sysloghost-directive)). The image synchronization log file can be retrieved with the `synclog` command ([here](#5476-tracking-the-provisioning-log-changes-synclog)) running from device mode in `cmsh`. Hints on provisioning problems are often found by looking at the tail end of the log. If the tail end of the log shows an rsync exit code of 23, then it suggests a transfer error. Sometimes the cause of the error can be determined by examining the file or filesystem for which the error occurs. For example, the following `synclog` output: **Example** ```bash [...] sending incremental file list rsync: [generator] delete_file: rmdir(lscratch) failed: Device or resource busy (16) [...] ``` could be due to a missing `lscratch` mountpoint in the software image. For the rsync transport, logs for node installation are kept under `/var/spool/cmd/`, with a log written for each node during provisioning. The name of the node is set as the prefix to the log name. For example `node002` generates the log: ```bash /var/spool/cmd/node002-\.rsync ``` ### 5.8.4 Ramdisk Fails During Loading Or Sometime Later One issue that may come up after a software image update via `yum`, `zypper`, or `apt` (section [11.4](/admin/softwareimagemanagement#114-managing-a-package-in-a-software-image-and-running-it-on-nodes)), is that the ramdisk stage may fail during loading or sometime later, for a node that is rebooted after the update. This occurs if there are instructions to modify the ramdisk by the update. In a normal machine the ramdisk would be regenerated. In a cluster, the extended ramdisk that is used requires an update, but BCM is not aware of this. Running the `createramdisk` command from `cmsh` or the `Recreate Initrd` command via the Base View navigation paths: * `Devices` > `Nodes` > `Edit` > `Kernel` > `Recreate Initrd` * `Grouping` > `Node Categories` > `Edit` > `Kernel` > `Recreate Initrd` * `Provisioning` > `Software Images` > `Edit` > `Recreate Initrd` (section [5.3.2](#532-selecting-kernel-driver-modules-to-load-onto-nodes)) generates an updated ramdisk for the cluster, and solves the failure for this case. Another, somewhat related possible cause of a halt at this stage, is that the kernel modules that are to be loaded may have been specified incorrectly by the administrator in the hierarchy of software image, category, or node ([here](#532-selecting-kernel-driver-modules-to-load-onto-nodes)). A check of the kernel modules specified in `softwareimage` mode, `category` mode, or `device` mode (for the particular node) may reveal a misconfiguration. ### 5.8.5 Ramdisk Fails With `No space left on device` The `createramdisk` log for a software image may show a message: `... abort: write error on (No space left on device)` This error can be caused due to a large ramdisk being generated that exceeds the default tmpfs size configured by CMDaemon. To solve this, a larger default tmpfs size value, in MB, can be specified with an `AdvancedConfig` ([here](/admin/cmdconf#c80-advancedconfig-directive)) directive in `cmd.conf`, on all the head nodes in the cluster: **Example** ```bash AdvancedConfig = { "RamdiskCreationTmpfsSize=512" } ``` After increasing the value with an `AdvancedConfig` directive, a restart of CMDaemon service is required on all head nodes for the change to take effect. The default tmpfs size was 256MB for BCM 10 versions before 10.31.0. Due to kernel growth, this was sometimes inadequate. The default size was therefore set to 512MB for BCM 10 versions from 10.31.0 onward. ### 5.8.6 Ramdisk Cannot Start Network The ramdisk must activate the node’s network interface in order to fetch the node-installer. To activate the network device, the correct kernel module needs to be loaded. If this does not happen, booting fails, and the console of the node displays something similar to figure [5.19](#fig-prov-no-network). ![No Network Interface](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/4f5cac181c4a693b74c6eb27a9c486ea63159520e9e7d65b659b98a665ad4969/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/prov-no-network.png) *Figure 5.19: No Network Interface* To solve this issue the correct kernel module should be added to the software image’s kernel module configuration (section [5.3.2](#532-selecting-kernel-driver-modules-to-load-onto-nodes)). For example, to add the e1000 module to the default image using `cmsh`: **Example** ```bash [mc]% softwareimage use default-image [mc->softwareimage[default-image]]% kernelmodules [mc->softwareimage[default-image]->kernelmodules]% add e1000 [mc->softwareimage[default-image]->kernelmodules[e1000]]% commit Initial ramdisk for image default-image was regenerated successfully [mc->softwareimage[default-image]->kernelmodules[e1000]]% ``` After committing the change it typically takes about a minute before the initial ramdisk creation is completed via a `mkinitrd` run by CMDaemon. ### 5.8.7 Node-Installer Cannot Create Disk Layout When the node-installer is not able to create a drive layout it displays a message similar to figure [5.20](#fig-prov-no-disk). The node-installer log file (section [5.8.2](#582-node-installer-logging)) contains something like: ```bash Mar 24 13:55:31 10.141.0.1 node-installer: Installmode is: AUTO Mar 24 13:55:31 10.141.0.1 node-installer: Fetching disks setup. Mar 24 13:55:31 10.141.0.1 node-installer: Checking partitions and filesystems. Mar 24 13:55:32 10.141.0.1 node-installer: Detecting device '/dev/sda': not found Mar 24 13:55:32 10.141.0.1 node-installer: Detecting device '/dev/hda': not found Mar 24 13:55:32 10.141.0.1 node-installer: Can not find device(s) (/dev/sda /dev/hda). Mar 24 13:55:32 10.141.0.1 node-installer: Partitions and/or filesystems are missing/corrupt. (Exit code 4, signal 0) Mar 24 13:55:32 10.141.0.1 node-installer: Creating new disk layout. Mar 24 13:55:32 10.141.0.1 node-installer: Detecting device '/dev/sda': not found Mar 24 13:55:32 10.141.0.1 node-installer: Detecting device '/dev/hda': not found Mar 24 13:55:32 10.141.0.1 node-installer: Can not find device(s) (/dev/sda /dev/hda). Mar 24 13:55:32 10.141.0.1 node-installer: Failed to create disk layout. (Exit code 4, signal 0) Mar 24 13:55:32 10.141.0.1 node-installer: There was a fatal problem. This node can not be\ installed until the problem is corrected. ``` ![No Disk](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/db8e87b98eac8e395b471156a9ee61994a44bd068551de069cf268c50243de4f/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/prov-no-disk.png) *Figure 5.20: No Disk* Disk layout failures can have several reasons. #### 5.8.7.1 BIOS And Order Issues One reason may be that the drive may be disabled in the BIOS. It should be enabled. Another reason may be that the drive order was changed. This could happen if, for example, a defective motherboard has been replaced. The drive order should be kept the same as it was before a motherboard change. #### 5.8.7.2 Read-only Mode Issues Another reason may be due to SSDs that have a hardware jumper or toggle switch that sets a drive to read-only mode. A read-only mode drive will typically fail at this point. The drive should be made writeable. #### 5.8.7.3 Hardware Issues If the node-installer log for the node shows lines with the text `Input/output error`, then it generally indicates a hardware issue. Possible hardware issues include: * a drive failure * a faulty cable between storage and controller * a faulty storage controller * a faulty backplane in the server If the node has enough RAM, then it is possible to boot up the node up as a diskless node, to carry out further diagnosis with disk tools such as `smartmontools`. #### 5.8.7.4 Software Driver Issues One of the most common software issues is that the correct storage driver is not being loaded. To solve this issue, the correct kernel module should be added to the software image’s kernel module configuration (section [5.3.2](#532-selecting-kernel-driver-modules-to-load-onto-nodes)). Experienced system administrators work out what drivers may be missing by checking the results of hardware probes. For example, going into the node-installer shell using `Alt-F2`, and then looking at the output of `lspci`, shows a list of hardware detected in the PCI slots and gives the chipset name of the storage controller hardware in this case: **Example** ```bash [ root@node001 ~]# lspci | grep SCSI 00:10.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2\ 008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03) ``` The next step is to Google with likely search strings based on that output. The Linux Kernel Driver DataBase (LKDDb) is a hardware database built from kernel sources that lists driver availability for Linux. It is available at [http://cateee.net/lkddb/](http://cateee.net/lkddb/). Using the Google search engine’s “`site`” operator to restrict results to the `cateee.net` web site only, a likely string to try might be: **Example** ```bash SAS2008 site:cateee.net ``` The search result indicates that the `mpt2sas` kernel module needs to be added to the node kernels. A look in the modules directory of the software image shows if it is available: **Example** ```bash find /cm/images/default-image/lib/modules/ -name "*mpt2sas*" ``` If it is not available, the driver module must then be obtained. If it is a source file, it will need to be compiled. By default, nodes run on standard distribution kernels, so that only standard procedures need to be followed to compile modules. If the module is available, it can be added to the default image, by using `cmsh` in `softwareimage` mode to create the associated object. The object is given the same name as the module, i.e. `mp2sas` in this case: **Example** ```bash [basecm10]% softwareimage use default-image [basecm10->softwareimage[default-image]]% kernelmodules [basecm10->softwareimage[default-image]->kernelmodules]% add mpt2sas [basecm10->softwareimage[default-image]->kernelmodules*[mpt2sas*]]% commit [basecm10->softwareimage[default-image]->kernelmodules[mpt2sas]]% Thu May 19 16:54:52 2011 [notice] basecm10: Initial ramdisk for image de\ fault-image is being generated [basecm10->softwareimage[default-image]->kernelmodules[mpt2sas]]% Thu May 19 16:55:43 2011 [notice] basecm10: Initial ramdisk for image de\ fault-image was regenerated successfully. [basecm10->softwareimage[default-image]->kernelmodules[mpt2sas]]% ``` After committing the change it can take some time before ramdisk creation is completed—typically about a minute, as the example shows. Once the ramdisk is created, the module can be seen in the list displayed from `kernelmodules` mode. On rebooting the node, it should now continue past the disk layout stage. ### 5.8.8 Node-Installer Cannot Start BMC (IPMI/iLO) Interface In some cases the node-installer is not able to configure a node’s BMC interface, and displays an error message similar to figure [5.21](#fig-prov-no-ipmi). ![No BMC Interface](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/5d95bd3b66a23ab9948f5a0db44e52b8d175fb7fd91a085f45ba7b6532f13c17/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/prov-no-ipmi.png) *Figure 5.21: No BMC Interface* Troubleshooting may involve trying out node identification and port queries with a BMC (section [3.7.4](/admin/configuring#374-identification-with-a-bmc)), as well as other steps that are described in the text that follows. Often the issue can be solved by adding the correct BMC (IPMI/iLO) kernel modules to the software image’s kernel module configuration. However, in some cases the node-installer is still not able to configure the BMC interface. If this is the case then the BMC probably does not support one of the commands the node-installer uses to set specific settings, or there may be a hardware glitch in the BMC. #### 5.8.8.1 The `setupBmc` Node-Installer Configuration Setting To solve this issue, setting up BMC interfaces can be disabled globally by setting the `setupBmc` field to `false` in the node-installer configuration file `/cm/node-installer/scripts/node-installer.conf` (for multiarch/multidistro configurations the path takes the form: `/cm/node-installer-`*\-\*`/scripts/node-installer.conf`). Doing this disables configuration of all BMC interfaces by the node-installer. A custom `finalize` script (Appendix [E](/admin/finalize)) can then be used to run the required commands instead. The `setupBmc` field in the node-installer should not be confused with the `SetupBMC` directive in `cmd.conf` ([here](/admin/cmdconf#c58-setupbmc-directive), Appendix [C.58](/admin/cmdconf#c58-setupbmc-directive)). The former is about enabling the BMC interface, while the latter is about enabling automated passwords to the BMC interface (an interface that must of course be enabled in the first place to work). #### 5.8.8.2 The `failOnMissingBmc` Node-Installer Configuration Setting If the kernel modules for the BMC are loaded up correctly, and the BMC is configured, but it is not detected by the node-installer, then the node-installer halts by default. This corresponds to the setting `failOnMissingBmc = true` in the node-installer configuration file `/cm/node-installer/scripts/node-installer.conf`. Toggling this to `false` skips BMC network device detection, and lets the node-installer continue past the BMC detection and configuration stage. This can be convenient, for example, if the BMC is not yet configured and the aim is to get on with setting up the rest of the cluster. #### 5.8.8.3 The `failOnFailedBmcCommand` Node-Installer Configuration Setting If a BMC command fails, then the node-installer by default terminates node installation. The idea behind this is to allow the administrator to fix the problem. Sometimes, however, hardware can wrongly signal a failure. That is, it can signal a false failure, as opposed to a true failure. A common case is the case of `ipmitool`. `ipmitool` is used by BCM to configure the BMC. With most hardware vendors it works as expected, signaling success and failure correctly. As per the default behavior: with success, node installation proceeds, while with failure, it terminates. With certain hardware vendors however `ipmitool` fails with an exit code 1, even though the BMC is properly configured. Again, as per the default behavior: success has node installation proceed, while failure has node installation terminate. Only this time, because the failure signal is incorrect, the termination on failure is also incorrect behavior. To get around the default behavior for false failure cases, the administrator can force the node-installer to set the value of `failOnFailedBmcCommand` to `false` in the node-installer configuration file `/cm/node-installer/scripts/node-installer.conf` (for multiarch/multidistro configurations the path takes the form: `/cm/node-installer-`*\-\*`/scripts/node-installer.conf`). The installation then skips past the false failure. #### 5.8.8.4 BMC Hardware Glitch And Cold Reset Sometimes, typically due to a hardware glitch, a BMC can get into a state where it is not providing services, but the BMC is still up (responding to pings). Contrariwise, a BMC may not respond to pings, but still respond to IPMI commands. A fix for such glitchy states is usually to power cycle the BMC. This is typically done, either physically, or by using a BMC management tool such as `ipmitool`. Physically resetting the power supply to the BMC is done typically by pulling the power cable out and then pushing it in again. For typical rack-based servers the server can just be pulled out and in again. Just doing a shutdown of the server with the power cable still in place normally does not power down the BMC. BMC management does allow the BMC to power down and be reset from software, without having to physically handle the server. This software-based *cold reset* is a BIOS-manufacturer-dependent feature. A popular tool used for managing BMCs that can do such a cold reset is `ipmitool`. This can be run remotely, but also on the node console if the node cannot be reached remotely. With `ipmitool`, a cold reset is typically carried out with a command such as: ```bash [root@basecm10 ~]# module load ipmitool [root@basecm10 ~]# ipmitool -U -P -H -I lanplus mc reset cold ``` The values for \<*bmcusername*> and \<*bmcpassword*> can be obtained as shown in section [3.7.2](/admin/configuring#372-bmc-authentication). #### 5.8.8.5 BMC Troubleshooting With The System Event Log The System Event Log (SEL) can be read with: ```bash [root@basecm10 ~]# module load ipmitool [root@basecm10 ~]# ipmitool -U -P -H -I lanplus sel list ``` The timestamped output can be inspected for errors related to the CPU, ECC, or memory. #### 5.8.8.6 Other BMC Troubleshooting Some more specific commands for handling IPMI might be via the `service ipmi` \<*option*> commands, which can show the IPMI service has failed to start up: **Example** ```bash [root@basecm10 ~]# service ipmi status Redirecting to /bin/systemctl status ipmi.service ipmi.service - IPMI Driver Loaded: loaded (/usr/lib/systemd/system/ipmi.service; disabled; vendor preset: enabled) Active: inactive (dead) ``` In the preceding session the driver has simply not been started up. It can be started up with the `start` option: **Example** ```bash [root@basecm10 ~]# service ipmi start Redirecting to /bin/systemctl start ipmi.service Job for ipmi.service failed because the control process exited with error code. See "systemctl\ status ipmi.service" and "journalctl -xe" for details. ``` In the preceding session, the start up failed. The service status output shows: **Example** ```bash [root@basecm10 ~]# service ipmi status -l Redirecting to /bin/systemctl status -l ipmi.service ipmi.service - IPMI Driver Loaded: loaded (/usr/lib/systemd/system/ipmi.service; disabled; vendor preset: enabled) Active: failed (Result: exit-code) since Mon 2016-12-19 14:34:27 CET; 2min 3s ago Process: 8930 ExecStart=/usr/libexec/openipmi-helper start (code=exited, status=1/FAILURE) Main PID: 8930 (code=exited, status=1/FAILURE) Dec 19 14:34:27 basecm10 systemd[1]: Starting IPMI Driver... Dec 19 14:34:27 basecm10 openipmi-helper[8930]: Startup failed. Dec 19 14:34:27 basecm10 systemd[1]: ipmi.service: main process exited, code=exited, status=1/\ FAILURE Dec 19 14:34:27 basecm10 systemd[1]: Failed to start IPMI Driver. Dec 19 14:34:27 basecm10 systemd[1]: Unit ipmi.service entered failed state. Dec 19 14:34:27 basecm10 systemd[1]: ipmi.service failed. ``` Further details can be found in the journal: **Example** ```bash [root@basecm10 ~]# journalctl -xe | grep -i ipmi ... -- Unit ipmi.service has begun starting up. Dec 19 14:34:27 basecm10 kernel: ipmi message handler version 39.2 Dec 19 14:34:27 basecm10 kernel: IPMI System Interface driver. Dec 19 14:34:27 basecm10 kernel: ipmi_si: Unable to find any System Interface(s) Dec 19 14:34:27 basecm10 openipmi-helper[8930]: Startup failed. ... ``` In the preceding session, the failure is due to a missing BMC interface (`Unable to find any System Interface(s)`). A configured BMC interface should show an output status similar to: **Example** ```bash [root@basecm10 ~]# service ipmi status Redirecting to /bin/systemctl status ipmi.service ipmi.service - IPMI Driver Loaded: loaded (/usr/lib/systemd/system/ipmi.service; disabled; vendor preset: enabled) Active: active (exited) since Mon 2016-12-19 14:37:10 CET; 2min 0s ago Process: 61019 ExecStart=/usr/libexec/openipmi-helper start (code=exited, status=0/SUCCESS) Main PID: 61019 (code=exited, status=0/SUCCESS) Dec 19 14:37:10 basecm10 systemd[1]: Starting IPMI Driver... Dec 19 14:37:10 basecm10 systemd[1]: Started IPMI Driver. ``` Sometimes the issue may be an incorrect networking specification for the BMC interfaces. MAC and IP details that have been set for the BMC interface can be viewed with the `lan print` option to `ipmitool` if the service has been started: **Example** ```bash [root@basecm10 ~]# module load ipmitool [root@basecm10 ~]# ipmitool lan print Set in Progress : Set Complete Auth Type Support : MD5 PASSWORD Auth Type Enable : Callback : MD5 PASSWORD : User : MD5 PASSWORD : Operator : MD5 PASSWORD : Admin : MD5 PASSWORD : OEM : IP Address Source : Static Address IP Address : 93.184.216.34 Subnet Mask : 255.255.255.0 MAC Address : aa:bb:01:02:cd:ef SNMP Community String : public IP Header : TTL=0x00 Flags=0x00 Precedence=0x00 TOS=0x00 BMC ARP Control : ARP Responses Enabled, Gratuitous ARP Disabled Gratituous ARP Intrvl : 0.0 seconds Default Gateway IP : 93.184.216.1 Default Gateway MAC : 00:00:00:00:00:00 Backup Gateway IP : 0.0.0.0 Backup Gateway MAC : 00:00:00:00:00:00 802.1q VLAN ID : Disabled 802.1q VLAN Priority : 0 RMCP+ Cipher Suites : 0,1,2,3,4,6,7,8,9,11,12,13,15,16,17,18 Cipher Suite Priv Max : caaaaaaaaaaaaaa : X=Cipher Suite Unused : c=CALLBACK : u=USER : o=OPERATOR : a=ADMIN : O=OEM ``` During normal operation the metrics (Appendix [G](/admin/metricshealthchecksactions)) displayed by BCM are useful. However, if those are not available for some reason, then the direct output from BMC sensor metrics may be helpful for troubleshooting: **Example** ```bash [root@basecm10 ~]# module load ipmitool [root@basecm10 ~]# ipmitool sensor list all # ipmitool sensor list Ambient Temp | 22.000 | degrees C | ok | na | na | na | 38.000 | 41.000 | 45.000 AVG Power | 300.000 | Watts | ok | na | na | na | na | na | na Fan 1 Tach | 4125.000 | RPM | ok | na | 750.000 | na | na | na | na ... ```