Cluster Management With NVIDIA Base Command Manager

This chapter introduces cluster management with NVIDIA Base Command Manager. A cluster running BCM exports a cluster management interface to the outside world, which can be used by any application designed to communicate with the cluster.

Section 2.1 introduces a number of concepts which are key to cluster management using BCM.

Section 2.2 gives a short introduction on how the modules environment can be used by administrators. The modules environment provides facilities to control aspects of a users’ interactive sessions and also the environment used by compute jobs.

Section 2.3 introduces how authentication to the cluster management infrastructure works and how it is used. Section 2.4 and section 2.5 introduce the cluster management GUI (Base View) and cluster management shell (cmsh) respectively. These are the primary applications that interact with the cluster management daemon.

Section 2.6 describes the basics of the cluster management daemon, CMDaemon, running on all nodes of the cluster.

2.1 Concepts

In this section some concepts central to cluster management with BCM are introduced.

2.1.1 Devices

A device in BCM infrastructure represents components of a cluster. A device can be any of the following types:

• Head Node

• Physical Node

• Virtual Node

• Cloud Node

• GPU Unit

• Chassis

• Switch (ethernet, InfiniBand, Myrinet)

• Lite Node

• Power Distribution Unit

• Rack Sensor Kit

• Generic Device

A device can have a number of properties (e.g. rack position, hostname, switch port) which can be set in order to configure the device. Using BCM, operations (e.g. power on) may be performed on a device. The property changes and operations that can be performed on a device depend on the type of device. For example, it is possible to mount a new filesystem to a node, but not to an Ethernet switch.

Every device that is managed by BCM has a device state associated with it. The table below describes the most important states for devices:

These, and other states are described in more detail in section 5.5.

[ DOWN ] and [ CLOSED ] (DOWN) states have an important difference. In the case of [ DOWN ], the device is down, but is typically intended to be available, and thus typically indicates a failure. In the case of [ CLOSED ] (DOWN), the device is down, but is intended to be unavailable, and typically indicates that the administrator deliberately brought the device down, and would like the device to be ignored.

2.1.2 Software Images

A software image is a blueprint for the contents of the local filesystems on a regular node. In practice, a software image is a directory on the head node containing a full Linux filesystem.

The software image in a standard BCM installation is based on the same parent distribution that the head node uses. A different distribution can also be chosen after installation, from the distributions listed in section 2.1 of the Installation Manual for the software image. That is, the head node and the regular nodes can run different parent distributions. However, such a “mixed” cluster can be harder to manage and it is easier for problems to arise in such mixtures. Such mixtures, while supported, are therefore not recommended, and should only be administered by system administrators that understand the differences between Linux distributions.

RHEL 8 and Rocky Linux 8 mixtures are completely compatible with each other on the head and regular nodes. The same applies to RHEL9 and Rocky Linux 9. That is because Rocky Linux is designed to be a binary-compatible derivative of its RHEL parents. On the other hand, SLES and Ubuntu need quite some effort to work in a mixture.

When a regular node boots, the node provisioning system (Chapter 5) sets up the node with a copy of the software image, which by default is called default-image.

Once the node is fully booted, it is possible to instruct the node to re-synchronize its local filesystems with the software image. This procedure can be used to distribute changes to the software image without rebooting nodes (section 5.6.2).

It is also possible to “lock” a software image so that no node is able to pick up the image until the software image is unlocked. (section 5.4.7).

Software images can be changed using regular Linux tools and commands (such as rpm and chroot). More details on making changes to software images and doing image package management can be found in Chapter 9.

2.1.3 Node Categories

2.1.3.1 Reasons For Categories

The collection of settings in BCM that can apply to a node is called the configuration of the node. The administrator usually configures nodes using the Base View (section 2.4) or cmsh (section 2.5) front end tools, and the configurations are managed internally with a database.

A node category is a group of regular nodes that share the same configuration. Node categories allow efficiency, allowing an administrator to:

• configure a large group of nodes at once. For example, to set up a group of nodes with a particular disk layout.

• operate on a large group of nodes at once. For example, to carry out a reboot on an entire category.

A regular node is in exactly one category at all times, which is default by default. The default category can be changed by accessing the base object of partition mode (here), and setting the value of defaultcategory to another, existing, category.

Nodes are typically divided into node categories based on the hardware specifications of a node or based on the task that a node is to perform. Whether or not a number of nodes should be placed in a separate category depends mainly on whether the configuration—for example: monitoring setup, disk layout, role assignment—for these nodes differs from the rest of the nodes.

2.1.3.2 Corresponding Category Values And Node Values

• For non-Boolean values, a node inherits values from the category it is in. Each value is treated as the default property value for a node, and can be overruled by specifying the node property value for a particular node.

• For Boolean values, such as datanode (here) and installbootrecord (here), a node does not inherit the value from the category it is in. Instead the category Boolean value has the Boolean or operation applied to the node Boolean value, and the result is the Boolean value that is used for the node. Since by default the values for these Booleans are zero, the result is initially similar to the non-Boolean values behavior.

2.1.3.3 Category And Software Image Do Not Necessarily Map One-To-One

One configuration property value of a node category is its software image (section 2.1.2). However, there is no requirement for a one-to-one correspondence between node categories and software images. Therefore multiple node categories may use the same software image, and conversely, one variable image—it is variable because it can be changed by the node setting—may be used in the same node category.

Software images can have their parameters overruled by the category settings. By default, however, the category settings that can overrule the software image parameters are unset.

By default, all nodes are placed in the default category. Alternative categories can be created and used at will, such as:

Example

2.1.4 Node Groups

A node group consists of nodes that have been grouped together for convenience. The group can consist of any mix of all kinds of nodes, irrespective of whether they are head nodes or regular nodes, and irrespective of what category they are in. A node may be in 0 or more node groups at one time. I.e.: a node may belong to many node groups.

Node groups are used mainly for carrying out operations on an entire group of nodes at a time. Since the nodes inside a node group do not necessarily share the same configuration, configuration changes cannot be carried out using node groups.

Example

One important use for node groups is in the nodegroups property of the provisioning role configuration (section 5.2.1), where a list of node groups that provisioning nodes provision is specified.

2.1.5 Roles

A role is a task that can be performed by a node. By assigning a certain role to a node, an administrator activates the functionality that the role represents on this node. For example, a node can be turned into provisioning node, or can be turned into a storage node, by assigning the corresponding roles to the node. Roles typically have services (section 3.15) that are associated with them. For example, the dhcpd service is associated with the boot role, which means that nodes assigned a boot role can then provide netbooting using dhcpd to other nodes.

Roles can be assigned to individual nodes or to node categories. When a role has been assigned to a node category, it is implicitly assigned to all nodes inside the category.

A configuration overlay (section 2.1.6) is a group of roles that can be assigned to designated groups of nodes within a cluster. This allows configuration of a large number of configuration parameters in various combinations of nodes.

Some roles allow parameters to be set that influence the behavior of the role. For example, the Slurm Client Role (which turns a node into a Slurm client) uses parameters to control how the node is configured within Slurm in terms of queues and the number of GPUs.

When a role has been assigned to a node category with a certain set of parameters, it is possible to override the parameters for a node inside the category. This can be done by assigning the role again to the individual node with a different set of parameters. Roles that have been assigned to nodes override roles that have been assigned to a node category.

Roles have a priority setting associated with them. Roles assigned at category level have a fixed priority of 250, while roles assigned at node level have a fixed priority of 750. The configuration overlay priority is variable, but is set to 500 by default. Thus, for example, roles assigned at the node level override roles assigned at the category level. Roles assigned at the node level also override roles assigned by the default configuration overlay.

A role can be imported from another entity, such as a node, a category, or a configuration overlay.

Roles are assigned only from existing roles. A custom role cannot be created by the cluster administrator, which means that the add command (here) does not work in role mode, by design. Instead, only the assign command (here) can be used for roles.

Examples of role assignment are given in sections 5.2.2 and 5.2.3.

2.1.6 Configuration Overlay

A configuration overlay assigns roles (section 2.1.5) for groups of nodes. The number of roles can be quite large, and priorities can be set for these.

Multiple configuration overlays can be set for a node. A priority can be set for each configuration overlay, so that a configuration overlay with a higher priority is applied to its associated node instead of a configuration overlay with a lower priority. The configuration overlay with the highest priority then determines the actual assigned role.

A configuration overlay assigns a group of roles to an instance. This means that roles are assigned to nodes according to the instance configuration, along with a priority. Whether the configuration overlay assignment is used, or whether the original role assignment is used, depends upon the configured priorities.

Configuration overlays can take on priorities in the range 0-1000, except for 250 and 750, which are forbidden. Setting a priority of -1 means that the configuration overlay is ignored.

The priorities of 250, 500, and 750 are also special, as indicated by the following table:

A custom configuration overlay can be created by the cluster administrator in configurationoverlay mode, which means that the add command (here) does work for it. This can be used to achieve a functional equivalent to creating custom roles. The creation of custom roles in role mode is a function that is missing by design.

2.2 Modules Environment

The modules environment is the shell environment that is set up by a third-party software (section 7.1 of the Installation Manual) called Environment Modules. The software allows users to modify their shell environment using pre-defined modules. A module may, for example, configure the user’s shell to run a certain version of an application.

Details of the modules environment from a user perspective are discussed in section 2.3 of the User Manual. However some aspects of it are relevant for administrators and are therefore discussed here.

2.2.1 Adding And Removing Modules

Modules may be loaded and unloaded, and also be combined for greater flexibility.

Modules currently installed are listed with:

module list

The modules available for loading are listed with:

module avail

Loading and removing specific modules is done with module load and module remove, using this format:

module load <module name 1> [<module name 2> ...]

For example, loading the shared module (section 2.2.2), the gcc compiler, the openmpi parallel library, and the openblas library, allows an MPI application myapp.c to be compiled with OpenBLAS optimizations:

Example

module add shared
module add gcc/13.1.0
module add openmpi/gcc/64/4.1.5
module add openblas
module add openblas/dynamic/0.3.18
mpicc -o myapp myapp.c

The exact versions used can be selected using tab-completion. In most cases, specifying version numbers explicitly is typically only necessary when multiple versions of an application are installed and available. When there is no ambiguity, module names without a further path specification may be used.

2.2.2 Using Local And Shared Modules

Applications and their associated modules are divided into local and shared groups. Local applications are installed on the local filesystem, whereas shared applications reside on a shared (i.e. imported) filesystem.

It is recommended that the shared module be loaded by default for ordinary users. Loading it gives access to the modules belonging to shared applications, and allows the module avail command to show these extra modules.

Loading the shared module automatically for root is not recommended on a cluster where shared storage is not on the head node itself. This is because root logins could be obstructed if this storage is not available, and if the root user relies on files in the shared storage.

On clusters without external shared storage, root can safely load the shared module automatically at login. This can be done by running the following command as root:

module initadd shared

Other modules can also be set to load automatically by the user at login by using “module initadd” with the full path specification. With the initadd option, individual users can customize their own default modules environment.

Modules can be combined in meta-modules. By default, the default-environment meta-module exists, which allows the loading of several modules at once by a user. Cluster administrators are encouraged to customize the default-environment meta-module to set up a recommended environment for their users. The default-environment meta-module is empty by default.

The administrator and users have the flexibility of deciding the modules that should be loaded in undecided cases via module dependencies. Dependencies can be defined using the prereq and conflict commands. The man page for modulefile gives details on configuring the loading of modules with these commands.

2.2.3 Setting Up A Default Environment For All Users

How users can set up particular modules to load automatically for their own use with the module initadd command is discussed in section 2.2.2.

How the administrator can set up particular modules to load automatically for all users by default is discussed in this section (section 2.2.3). In this example it is assumed that all users have just the following modules as a default:

Example

[fred@basecm11 ~]$ module list
Currently Loaded Modulefiles:
1) shared

The slurm and gdb modules can then be set up by the administrator as a default for all users in the following 2 ways:

1. Creating and defining part of a .profile to be executed for login shells. For example, a file userdefaultmodules.sh created by the administrator:

   [root@basecm11 ~]# cat /etc/profile.d/userdefaultmodules.sh
   module load shared
   module load slurm
   module load gdb

   Whenever users now carry out a bash login, these modules are loaded.

2. Instead of placing the modules directly in a script under profile.d like in the preceding item, a slightly more sophisticated way is to set the modules in the meta-module /cm/shared/modulefiles/default-environment. For example:

   [root@basecm11 ~]# cat /cm/shared/modulefiles/default-environment
   #%Module1.0######################################################
   ## default modulefile
   ##
   proc ModulesHelp { } {
           puts stderr "\tLoads default environment modules for this cluster"
   }
   module-whatis   "adds default environment modules"
   # Add any modules here that should be added by when a user loads the 'default-enviro\
   nment' module
   module add shared slurm gdb

   The script userdefaultmodules.sh script under profile.d then only needs to have the default-environment module loaded in it:

   [root@basecm11 ~]# cat /etc/profile.d/userdefaultmodules.sh
   module load -s default-environment

   The -s option is used to load it silently, because otherwise a message is displayed on the terminal informing the person logging in that the default-environment module has been loaded.

   Now, whenever the administrator changes the default-environment module, users get these changes too during login.

The lexicographical order of the scripts in the /etc/profile directory is important. For example, naming the file defaultusermodules.sh instead of userdefaultmodules.sh means that the modules.sh script is run after the file is run, instead of before, which would cause an error.

2.2.4 Creating A Modules Environment Module

All module files are located in the /cm/local/modulefiles and /cm/shared/modulefiles directories. A module file is a Tcl or Lua script in which special commands are used to define functionality. The modulefile(1) man page has more on this.

Cluster administrators can use the existing modules files as a guide to creating and installing their own modules for module environments, and can copy and modify a file for their own software if there is no environment provided for it already by BCM.

2.2.5 Lua Modules Environment (Lmod)

By default, BCM uses traditional Tcl scripts for its environment modules files, or Tmod in BCM shorthand. Lua modules, or Lmod, provide alternative environment modules files, where the files are typically written in Lua. Lmod can be used as a replacement for Tmod.

Conceptually Lmod works in the same way as Tmod, but provides some extra features and commands.

For Lmod, the module files are typically written in Lua, but Lmod is also capable of reading Tcl module files. It is therefore not necessary to convert all existing Tcl modules manually to the Lua language.

On a BCM cluster, both Lmod and Tmod are installed by default. However only one of them is active, depending on which one is enabled. Switching between Lmod and Tmod for a node can be done by setting an environment variable, $ENABLE_LMOD in the cm-lmod-init.sh shell script.

2.2.5.1 Switching For The Head Node

For example, for the head node:

Example

[root@basecm11 ~]# cat /etc/sysconfig/modules/lmod/cm-lmod-init.sh
export ENABLE_LMOD=1

In the preceding example, Lmod is enabled, and Tmod is disabled because $ENABLE_LMOD is set to 1.

Example

[root@basecm11 ~]# cat /etc/sysconfig/modules/lmod/cm-lmod-init.sh
export ENABLE_LMOD=0

In the preceding example, Lmod is disabled, and Tmod is enabled because $ENABLE_LMOD is set to 0.

A change in the file on the node is effective after having logged out, then logged into the shell again.

2.2.5.2 Switching For The Regular Nodes

A node image is a directory and contents of that directory. It is used as the template for a regular node when the node is provisioned (Chapter 5). For a node image with the name <image name>, the cm-lmod-init.sh file is located at /cm/images/<image name>/etc/sysconfig/modules/lmod/cm-lmod-init.sh. For switching between Lmod and Tmod on a regular node, the file is changed on the image, and the file on the image is then updated to the node. The update from the image to the node is typically carried out with the imageupdate command in cmsh (section 5.6.2) or the Update node command in Base View (section 5.6.3).

2.3 Authentication

2.3.1 Changing Administrative Passwords On The Cluster

How to set up or change regular user passwords is not discussed here, but in Chapter 6 on user management.

Amongst the administrative passwords associated with the cluster are:

1. The root password of the head node: This allows a root login to the head node.

2. The root passwords of the software images: These allow a root login to a regular node running with that image, and is stored in the image file.

3. The root password of the node-installer: This allows a root login to the node when the node-installer, a stripped-down operating system, is running. The node-installer stage prepares the node for the final operating system when the node is booting up. Section 5.4 discusses the node-installer in more detail.

4. The root password of MySQL: This allows a root login to the MySQL server.

To avoid having to remember the disparate ways in which to change these 4 kinds of passwords, the cm-change-passwd command runs a dialog prompting the administrator on which of them, if any, should be changed, as in the following example:

[root@basecm11 ~]# cm-change-passwd
With this utility you can easily change the following passwords:
  * root password of head node
  * root password of slave images
  * root password of node-installer
  * root password of mysql
Note: if this cluster has a high-availability setup with 2 head
      nodes, be sure to run this script on both head nodes.
Change password for root on head node? [y/N]: y
Changing password for root on head node.
Changing password for user root.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
Change password for root in default-image [y/N]: y
Changing password for root in default-image.
Changing password for user root.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
Change password for root in node-installer? [y/N]: y
Changing password for root in node-installer.
Changing password for user root.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
Change password for MYSQL root user? [y/N]: y
Changing password for MYSQL root user.
Old password:
New password:
Re-enter new password:

For a high-availability—also called a failover—configuration, the passwords are copied over automatically to the other head node when a change is made in the software image root password (case 2 here).

For the remaining password cases (head root password, MySQL root password, and node-installer root password), the passwords are best “copied” over to the other head node by simply rerunning the script on the other head node.

Also, in the case of the password for software images used by the regular nodes: the new password that is set for a regular node only works on the node after the image on the node itself has been updated, with, for example, the imageupdate command (section 5.6.2). Alternatively, the new password can be made to work on the node by simply rebooting the node to pick up the new image.

The LDAP root password is a random string set during installation. Changing this is not done using cm-change-password. It can be changed as explained in Appendix I.

If the administrator has stored the password to the cluster in the Base View front-end, then the password should be modified there too (figure 2.2).

2.3.2 Logins Using ssh

The standard system login root password of the head node, the software image, and the node-installer, can be set using the cm-change-passwd command (section 2.3.1).

In contrast, ssh logins from the head node to the regular nodes are set by default to be passwordless:

• For non-root users, an ssh passwordless login works if the /home directory that contains the authorized keys for these users is mounted. The /home directory is mounted by default on the head node as well as on the regular node, so that by default a passwordless login works from the head node to the regular nodes, as well as from the regular nodes to the head node.

• For the root user, an ssh passwordless login should always work from the head node to the regular nodes since the authorized keys are stored in /root. Logins from the regular node to the head node are configured by default to request a password, as a security consideration.

Users can be restricted from ssh logins

• on regular nodes using the cmsh usernodelogin option (section 7.2.1) or the Base View User node login (section 7.2.2) settings

• on the head node by modifying the sshd configuration on the head node. For example, to allow only root logins, the value of AllowUsers can be set in /etc/ssh/sshd_config to root. The man page for sshd_config has details on this.

2.3.3 Certificates

2.3.3.1 PEM Certificates And CMDaemon Front-end Authentication

While nodes in the cluster accept ordinary ssh-based logins, the cluster manager accepts public key authentication using X509v3 certificates. Public key authentication using X509v3 certificates means in practice that the person authenticating to the cluster manager must present their public certificate, and in addition must have access to the private key that corresponds to the certificate.

BCM uses the PEM format for certificates. In this format, the certificate and private key are stored as plain text in two separate PEM-encoded files, ending in .pem and .key.

Using cmsh and authenticating to BCM: By default, one administrator certificate is created for root for the cmsh front end to interact with the cluster manager. The certificate and corresponding private key are thus found on a newly-installed BCM cluster on the head node at:

/root/.cm/admin.pem
/root/.cm/admin.key

The cmsh front end, when accessing the certificate and key pair as user root, uses this pair by default, so that prompting for authentication is then not a security requirement. The logic that is followed to access the certificate and key by default is explained in detail in item 2 here.

Using Base View and authenticating to BCM: When an administrator uses the Base View front end, a login to the cluster is carried out with username password authentication (figure 2.2), unless the authentication has already been stored in the browser, or unless certificate-based authentication is used.

• Certificate-based authentication can be carried out using a PKCS#12 certificate file. This can be generated from the PEM format certificates. For example, for the root user, an openssl command that can be used to generate the admin.pfx file is:

  openssl pkcs12 -export -in ~/.cm/admin.pem -inkey ~/.cm/admin.key -out ~/.cm/admin.pfx

  • In Chrome, the IMPORT wizard at chrome://settings/certificates can be used to save the file into the browser.

  • For Firefox, the equivalent navigation path is:

    about:preferences#privacy > Certificates > View Certificates > Your Certificates > Import

  The browser can then access the Base View front end without a username/password combination.

If the administrator certificate and key are replaced, then any other certificates signed by the original administrator certificate must be generated again using the replacement, because otherwise they will no longer function.

Certificate generation in general, including the generation and use of non-administrator certificates, is described in greater detail in section 6.5.

2.3.3.2 Replacing A Temporary Or Evaluation License

In the preceding section, if a license is replaced, then regular user certificates need to be generated again. Similarly, if a temporary or evaluation license is replaced, regular user certificates need to be generated again. This is because the old user certificates are signed by a key that is no longer valid. The generation of non-administrator certificates and how they function is described in section 6.5.

2.3.3.3 Checking Certificates Validity With cm-check-certificates.sh

A BCM script that checks whether certificates are current or expired is /cm/local/apps/cmd/scripts/cm-check-certificates.sh:

Example

root@basecm11:~# /cm/local/apps/cmd/scripts/cm-check-certificates.sh
/cm/local/apps/cmd/etc/cluster.pem: OK
/cm/local/apps/cmd/etc/cert.pem: OK
/cm/local/apps/cmd/etc/cluster.key : matches
All /cm/local/apps/cmd/etc/cluster.pem files are up to date (82070dedb489df6c19ffa3ace1bf354e)
/root/.cm/admin.pem: OK
     ... output truncated ...
root@basecm11:~#

2.3.4 Profiles

Certificates that authenticate to CMDaemon contain a profile.

A profile determines which cluster management operations the certificate holder may perform. The administrator certificate is created with the admin profile, which is a built-in profile that allows all cluster management operations to be performed. In this sense it is similar to the root account on unix systems. Other certificates may be created with different profiles giving certificate owners access to a pre-defined subset of the cluster management functionality (section 6.5).

2.4 Base View GUI

This section introduces the basics of the cluster management GUI (Base View). Base View is the web application front end to cluster management in BCM.

is supported to run on the last 2 versions of Firefox, Google Chrome, Edge, and Safari. “Last 2 versions” means the last two publicly released versions (https://browserl.ist/?q=last+2+versions) at the time of release of . For example, at the time of writing of this section, July 2026, the last 2 versions were:

should run on more up-to-date versions of the browsers in the table without issues.

should run on other recent browsers without issues too, but this is not supported. Browsers that run on mobile devices are also not supported.

2.4.1 Installing The Cluster Management GUI Service

In a default installation, accessing the head node hostname or IP address with a browser leads to the landing page (figure 2.1).

Figure 2.1: Head node hostname or IP address landing page at https://<host name or IP address>

The landing page is served by the Apache web server from the distribution, and can be served over the HTTP (unencrypted) or HTTPS (encrypted) protocols.

The certificates used to ensure an encrypted connection are set within:

• /etc/httpd/conf.d/ssl.conf for the RHEL family of distributions. The PEM-encoded certificate at /etc/pki/tls/certs/localhost.crt is set by default.

• /etc/apache2/sites-available/default-ssl.conf for Ubuntu. The PEM-encoded certificate at /etc/ssl/certs/ssl-cert-snakeoil.pem is set by default.

The system administrator may wish to consider the security aspects of using the default distribution certificates, and may wish to replace them.

Within the landing page are several blocks, one of which is the Base View block. Base View is the BCM GUI. Within the Base View block is a clickable link, which is a circle with a chain-link symbol inside it.

Base View connects by default to the encrypted web service on port 8081. This is served from the head node cluster manager, rather than from Apache, to the browser. The direct URL for this is of the form:

https://<host name or IP address>:8081/base-view

The BCM package that provides the service is base-view and it is installed by default with BCM. The service can be disabled by removing the package with, for example, yum remove base-view.

2.4.1.1 NVIDIA Base Command Manager Base View Login Window

Figure 2.2 shows the login dialog window for Base View.

Figure 2.2: Base View Login via https://<host name or IP address>:8081/base-view

2.4.1.2 NVIDIA Base Command Manager Base View Default Display On Connection

Clicking on the Login button logs the administrator into the Base View service on the cluster. By default an overview window is displayed, corresponding to the navigation path Cluster > Overview (figure 2.3).

Figure 2.3: Cluster Overview

2.4.2 Navigating The Cluster With Base View

Aspects of the cluster can be managed by administrators using Base View (figure 2.3).

2.4.2.1 Navigating The Cluster With Base View: Getting Around

The resource tree, displayed on the left side of the window, consists of representations of cluster usage resources that are viewable or configurable, such as Networking, Provisioning, Grouping, and Devices, Cloud, and Containers.

Selecting a resource opens a window that allows parameters related to the resource to be viewed and managed.

As an example, the Cluster resource can be selected. On the first session this opens up the Overview window, which allows the cluster administrator to see the main characteristics of the cluster quickly. The characteristics can be explored further by clicking through on widgets that open up new windows, or clicking on tabs. Clicking on the BCM NVIDIA logo in the top left corner of the page conveniently brings the user back to Cluster > Overview.

Within the main content window, there are a variety of widgets that are used. A helpful nuance of widget behavior to note is that

• the ↗ widget is for opening up an existing object, and

• the + widget is for opening up a new object.

Both widgets open up a subwindow in which values can be viewed, and perhaps modified.

Also useful to know is the following: going back to a previous window is possible by scrolling back up to the top of the subwindow and clicking on the appropriate item in the breadcrumbs trail. For example: A user has navigated down the path: Cluster > Settings > Time zone > , and has reached a configuration screen to set the time zone, as in figure 2.4:

Figure 2.4: Base View Time Zone Breadcrumb

If the user wishes to return to the earlier Cluster > Settings screen without changing any values in the current time zone screen, then it can be done by clicking on the appropriate breadcrumb. In this case it is the breadcrumb associated with the navigation path to that earlier screen, which in figure 2.4 is bcm11:base - Settings.

2.4.2.2 Navigating The Cluster With Base View: A Tree Map Of Locations From The Cluster Resource

The tabs within the Cluster resource window are mapped out in the tree figure displayed next, and the tree is followed by a brief description of its items:

Cluster
|
|-- Overview
|     |
|     |-- Cluster Utilization
|     |     `-- displays a chart of cluster utilization against time for:
|     |         OccupationRate
|     |         TotalGPUUtilization
|     |         TotalCPUUtilization
|     |         TotalMemoryUtilization
|     |
|     |-- Job Slots Utilization
|     |     `-- displays charts of any job slots utilization
|     |
|     |-- Device Status
|     |     |-- Devices{Closed|Down|Total|Up}
|     |     |-- ManagedSwitches{Closed|Down|Total|Up}
|     |     |-- NVLinkSwitches{Closed|Down|Total|Up}
|     |     |-- Nodes{Closed|Down|Total|Up}
|     |     `-- PowerShelves{Closed|Down|Total|Up}
|     |
|     |-- Resource Status
|     |     |-- Cores{Down|Total|Up}
|     |     |-- DPUNodes{Down|Total|Up}
|     |     `-- GPUs{Down|Total|Up}
|     |
|     |-- Health Checks
|     |     |-- Entity
|     |     |-- Name
|     |     |-- Value
|     |     `-- Last Check
|     |
|     |-- GPUs
|     |     |-- Node
|     |     |-- GPU
|     |     |-- Utilization
|     |     |-- Temperature
|     |     `-- Power Usage
|     |
|     |-- Workload
|     |     |-- Name
|     |     |-- Scheduler
|     |     |-- Running
|     |     `-- Queued
|     |
|     |-- Cluster Overview
|     |     |-- Uptime
|     |     `-- Phase load
|     |
|     |-- Disks
|     |     |-- Mountpoint
|     |     |-- Used
|     |     `-- Free
|     |
|     `-- License Information
|           |-- Nodes
|           |-- Accelerators
|           |-- Building Management System Integration
|           |-- Leak Detection
|           |-- NMX Manager
|           |-- Inter-node Memory Exchange (IMEX) Management
|           |-- GB200 Rack Management
|           |-- GB200 Firmware Management
|           |-- Power Reservation Steering
|           |-- Run:ai
|           |-- GPU Workload Power Profiles
|           |-- Autonomous Hardware Recovery
|           `-- Autonomous Job Recovery
|
|-- Settings
|     |-- Cluster name
|     |     |-- Cluster name
|     |     |-- Cluster reference architecture
|     |     |-- Administrator email
|     |     |-- Name
|     |     `-- Headnode
|     |
|     |-- Name servers
|     |     |-- Name servers
|     |     |-- Name servers from DHCP
|     |     `-- Time servers
|     |
|     |-- Search domains
|     |     |-- Search domains
|     |     `-- Relay Host
|     |
|     |-- Externally visible IP
|     |     `-- Externally visible IP
|     |
|     |-- Time zone
|     |     `-- Time zone
|     |
|     |-- BMC settings
|     |     |-- BMC Settings
|     |     |-- SNMP Settings
|     |     |-- DPU Settings
|     |     |-- SELinux Settings
|     |     |-- Access Settings
|     |     |-- Provisioning Settings
|     |     |-- ZTP settings
|     |     |-- ZTP new switch settings
|     |     |-- NetQ settings
|     |     |-- UFM settings
|     |     `-- NMX Manager settings
|     |
|     |-- Default burn configuration
|     |     |-- Default burn config
|     |     `-- Available Burn configs
|     |
|     |-- External Network
|     |     |-- External network
|     |     |-- Management network
|     |     `-- No zero conf
|     |
|     |-- Default category
|     |     |-- Default category
|     |     `-- ArchOS
|     |
|     |-- Sign installer certificates
|     |     `-- Sign installer certificates
|     |
|     |-- Failover
|     |     `-- Failover
|     |
|     |-- Failover groups
|     |     `-- Failover groups
|     |
|     |-- Notes
|     |     `-- Notes
|     |
|     |-- Wlm job power usage settings
|     |     `-- Wlm job power usage settings
|     |
|     |-- Leak action policies
|     |     |-- Active leak action policies
|     |     `-- Leak action policies
|     |
|     |-- Building Management System
|     |     |-- BMS
|     |     |-- BMS path
|     |     |-- BMS Certificate
|     |     `-- BMS private key
|     |
|     `-- Prometheus metric forwarders
|           `-- Prometheus metric forwarders
|
|-- Run Command
|     |-- Nodes
|     `-- Command
|
|-- License Information
|     |-- License Information
|     |     |-- Edition
|     |     |-- Duration
|     |     |-- Version
|     |     |-- MAC Address
|     |     |-- Nodes
|     |     |-- Accelerators
|     |     |-- Licensee
|     |     `-- Serial
|     |
|     `-- Mission Control
|           |-- Manage Mission Control (hyperlink)
|           |-- Building Management System Integration
|           |-- Leak Detection
|           |-- NMX Manager
|           |-- Inter-node Memory Exchange (IMEX) Management
|           |-- GB200 Rack Management
|           |-- GB200 Firmware Management
|           |-- Power Reservation Steering
|           |-- Run:ai
|           |-- Autonomous Hardware Recovery
|           `-- Autonomous Job Recovery
|
|-- System Information
|     |-- Type
|     |-- Memory Total
|     |-- Disk count
|     |-- Hostname
|     |-- BIOS Date
|     |-- OS Version
|     |-- System Name
|     |-- Number of CPUs
|     `-- Number of Cores
|
|-- Hardware Inventory
|     |-- Device
|     |-- Name
|     |-- Manufacturer
|     |-- Model
|     |-- Serial Number
|     `-- Part Number
|
|-- Node Arch OS Information
|     |-- Hostname
|     |-- Reported Arch
|     |-- Reported OS
|     |-- Reported Timestamp
|     |-- Configured Arch
|     `-- Configured OS
|
|-- Version Information
|     |-- Hostname
|     |-- BCM Version
|     |-- CMDaemon Version
|     |-- Build Index
|     |-- Build Hash
|     `-- Database Version
|
|-- Port Forwarding
|     |-- Device
|     |-- IP
|     |-- Port
|     |-- Head Port
|     |-- Sessions
|     `-- Actions
|
|-- IMEX Configuration
|     |-- Node
|     |-- IP
|     |-- Status
|     |-- Version
|     |-- Connection Node
|     |-- Connection IP
|     `-- Connection Status
|
`-- Workload Utilization
      `-- displays a chart of jobs against nodes

2.4.2.3 Overview

The Overview tab window in the tree map has sections that show:

• Cluster Utilization graphs, which includes Occupation rate (here).

• The number of devices in various states (UP, DOWN …) and their total number. Devices states are covered in sections 5.5.3 and 5.5.4.

• The number of resources (Cores, DPUs, and GPUs) in various states (UP, DOWN, …) and their total number.

• Health checks (sections 10.2.1 and 10.2.4).

• GPU usage information (includes temperature, power).

• Workload scheduling. Workload management is described in Chapter 7.

• Uptime.

• Disks status.

• The license information. This shows the number of nodes and accelerators (such as GPUs) that have been used, out of the number that are licensed to be used. In addition, the license information section shows whether particular license features are enabled, such as for features available in NVIDIA Mission Control (section 1.2 of the NVIDIA Mission Control Manual).

2.4.2.4 Settings

The Settings window has a number of global cluster properties and property groups. These are loosely grouped as follows:

• Cluster name: This section can also manage the cluster administrator email, partition name, and head node name.

• Name servers: This section also manages the time servers.

• Search domains: A relay host can also be set in this section.

• Externally visible IP

• Time zone

• BMC settings: This section is also used to manage SNMP, DPU, SELinux, Access, Provisioning, ZTP, NetQ, UFM, and NMX Manager settings.

• Default burn configuration: Sets the default burn configuration, and values in the burn configuration settings.

• External network: The management network can be defined from this section.

• Default category: Sets the default category

• Sign installer certificates: This section manages the manual or automatic signing of the node-installer certificates (section 5.4.1).

• Failover: Opens up a window to manage failover properties.

• Failover groups: Opens up a window to manage failover groups properties.

• Notes

• Wlm job power usage settings: Opens up a new window to add settings

• Leak action policies: Opens up a new window to manage settings.

• Building Management System

• Prometheus metric forwarders: Opens up a new window to manage settings.

2.4.2.5 Run Command

The Run command tab window allows a specified command to be run on a selected node of the cluster.

2.4.2.6 License Information

The License Information section of the License Information tab window shows some further information to do with cluster licensing; it is the Base View version of verify-license info (section 4.2.2 of the Installation Manual).

A slightly obscure property within this pane is Version, which refers to the version type of the license, rather than the BCM version. The license for NVIDIA Base Command Manager, with a value for version of 10.0 and above, is of a type that is incompatible with versions before 10.0. This means that an upgrade from before 9.2, to 10.0 or beyond, requires a license upgrade. The BCM support team must be contacted to arrange the license upgrade.

The Mission Control section of the License Information window lets NVIDIA Mission Control features be managed with the Manage Mission Control URL. Other parts of the section show which features of the NVIDIA Mission Control license are currently active.

2.4.2.7 System information

The System information tab window shows the main hardware specifications of the devices in the cluster (CPU, memory, BIOS), along with the operating system version that each device runs.

2.4.2.8 Hardware Inventory

The Hardware Inventory tab window displays a table of hardware identifiers. It is the Base View version of the hardwareinventoryinfo command in the device mode of cmsh.

2.4.2.9 Version Information

The Version Information tab window shows version information for important cluster software components, such as the CMDaemon database version, BCM version and builds.

2.4.2.10 Port Forwarding

The Port Forwarding tab window shows ports that have been forwarded from the head node to the regular nodes. It is the Base View version of the portforward list command (section 3.2.3.7) in the device mode of cmsh.

2.4.2.11 IMEX Configuration

The IMEX configurations tab window lists the state of IMEX mappings per node.

2.4.2.12 Workload Utilization

The Workload Utilization tab window displays a chart of nodes running a job or not.

2.5 Cluster Management Shell

This section introduces the basics of the cluster management shell, cmsh. This is the command-line interface to cluster management in BCM. Since cmsh and Base View give access to the same cluster management functionality, an administrator need not become familiar with both interfaces. Administrators intending to manage a cluster with only Base View may therefore safely skip this section.

The cmsh front end allows commands to be run with it, and can be used in batch mode. Although cmsh commands often use constructs familiar to programmers, it is designed mainly for managing the cluster efficiently rather than for trying to be a good or complete programming language. For programming cluster management, the use of Python bindings (Chapter 1 of the Developer Manual) is generally recommended instead of using cmsh in batch mode.

Usually cmsh is invoked from an interactive session (e.g. through ssh) on the head node, but it can also be used to manage the cluster from outside.

2.5.1 Invoking cmsh

From the head node, cmsh can be invoked as follows:

Example

[root@mycluster ~]# cmsh
[mycluster]%

By default it connects to the IP address of the local management network interface, using the default BCM port. If it fails to connect as in the preceding example, but a connection takes place using cmsh localhost, then the management interface is most probably not up. In that case, bringing the management interface up allows cmsh to connect to CMDaemon.

Running cmsh without arguments starts an interactive cluster management session. To go back to the unix shell, a user enters quit or ctrl-d:

[mycluster]% quit
[root@mycluster ~]#

2.5.1.1 Batch Mode And Piping In cmsh

The -c flag allows cmsh to be used in batch mode. Commands may be separated using semi-colons:

[root@mycluster ~]# cmsh -c "main showprofile; device status apc01"
admin
apc01 ............... [   UP   ]
[root@mycluster ~]#

Alternatively, commands can be piped to cmsh:

[root@mycluster ~]# echo device status | cmsh
device status
apc01 ............... [   UP   ]
mycluster ........... [   UP   ]
node001 ............. [   UP   ]
node002 ............. [   UP   ]
switch01 ............ [   UP   ]
[root@mycluster ~]#

2.5.1.2 Dotfiles And /etc/cmshrc File For cmsh

In a similar way to unix shells, cmsh sources an rc file from the /etc directory, and also dotfiles, if they exist. The sourcing is done upon start-up in both batch and interactive mode.

If /etc/cmshrc exists, then its settings are used, but the values can be overridden by user dotfiles. This is standard Unix behavior, analogous to how bash works with /etc/bashrc and .bashrc files.

In the following list of cmsh dotfiles, a setting in the file that is in the shorter path overrides a setting in the file with the longer path (i.e.: “shortest path overrides”):

• ~/.cm/cmsh/.cmshrc

• ~/.cm/.cmshrc

• ~/.cmshrc

2.5.1.3 Defining Command Aliases In cmsh

Sourcing settings is convenient when defining command aliases. Command aliases can be used to abbreviate longer commands. For example, putting the following in .cmshrc would allow lv to be used as an alias for device list virtualnode:

Example

alias lv device list virtualnode

Besides defining aliases in dotfiles, aliases in cmsh can also be created with the alias command. The preceding example can be run within cmsh to create the lv alias. Running the alias command within cmsh lists the existing aliases.

Aliases can be exported from within cmsh together with other cmsh dot settings with the help of the export command:

Example

[mycluster]% export > /root/mydotsettings

The dot settings can be taken into cmsh by running the run command from within cmsh:

Example

[mycluster]% run /root/mydotsettings

2.5.1.4 Built-in Aliases In cmsh

The following aliases are built-ins, and are not defined in any .cmshrc or cmshrc files:

[basecm11]% alias
alias - goto -
alias .. exit
alias / home
alias ? help
alias ds device status
alias ls list

The meanings are:

• goto -: go to previous directory level of cmsh

• exit: go up a directory level, or leave cmsh if already at top level.

• home: go to the top level directory

• help: show help text for current level

• device status: show status of devices that can be accessed in device mode

2.5.1.5 Automatic Aliases In cmsh

A cmsh script is a file that has a sequence of cmsh commands that run within a cmsh session.

The directory .cm/cmsh/ can have placed in it a cmsh script with a .cmsh suffix and an arbitrary prefix. The prefix then automatically becomes an alias in cmsh.

In the following example

• the file tablelist.cmsh provides the alias tablelist, to list devices using the | symbol as a delimiter, and

• the file dfh.cmsh provides the alias dfh to carry out the Linux shell command df -h

Example

[root@mycluster ~]# cat /root/.cm/cmsh/tablelist.cmsh
list -d "|"
[root@mycluster ~]# cat /root/.cm/cmsh/dfh.cmsh
!df -h
[root@mycluster ~]# cmsh
[mycluster]% device
[mycluster->device]% alias | egrep '(tablelist|dfh)'
alias dfh run /root/.cm/cmsh/dfh.cmsh
alias tablelist run /root/.cm/cmsh/tablelist.cmsh
[mycluster->device]% list
Type                   Hostname (key)   MAC                Category         Ip
---------------------- ---------------- ------------------ ---------------- ---------------
HeadNode               mycluster        FA:16:3E:B4:39:DB                   10.141.255.254
PhysicalNode           node001          FA:16:3E:D5:87:71  default          10.141.0.1
PhysicalNode           node002          FA:16:3E:BE:05:FE  default          10.141.0.2
[mycluster->device]% tablelist
Type                  |Hostname (key)  |MAC               |Category        |Ip
----------------------|----------------|------------------|----------------|---------------
HeadNode              |mycluster       |FA:16:3E:B4:39:DB |                |10.141.255.254
PhysicalNode          |node001         |FA:16:3E:D5:87:71 |default         |10.141.0.1
PhysicalNode          |node002         |FA:16:3E:BE:05:FE |default         |10.141.0.2
[mycluster->device]% dfh
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        1.8G     0  1.8G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G   33M  1.8G   2% /run
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/vdb1        25G   17G  8.7G  66% /
tmpfs           374M     0  374M   0% /run/user/0

The cmsh session in NVIDIA Base Command Manager does not need restarting for the alias to become active.

2.5.1.6 Default Arguments In cmsh Scripts

In a cmsh script, the parameters $1, $2 and so on can be used to pass arguments. If the argument being passed is blank, then the values the parameters take also remain blank. However, if the parameter format has a suffix of the form - <value>, then <value> is the default value that the parameter takes if the argument being passed is blank.

Example

[root@mycluster ~]# cat .cm/cmsh/encrypt-node-disk.cmsh
home
device use ${1-node001}
set disksetup /root/my-encrypted-node-disk.xml
set revision ${2-test}
commit

The script can be run without an argument (a blank value for the argument), in which case it takes on the default value of node001 for the parameter:

[root@mycluster ~]# cmsh
[mycluster]% encrypt-node-disk
[mycluster->device[node001]]%

The script can be run with an argument (node002 here), in which case it takes on the passed value of node002 for the parameter:

[root@mycluster ~]# cmsh
[mycluster]% encrypt-node-disk node002
[mycluster->device[node002]]%

2.5.1.7 Options Usage For cmsh

The options usage information for cmsh is obtainable with cmsh -h:

Usage:
          cmsh [options] [hostname[:port]]
          cmsh [options] -c <command>
          cmsh [options] -f <filename>
Options:
          --help|-h
               Display this help
          --noconnect|-u
               Start unconnected
          --controlflag|-z
               ETX in non-interactive mode
          --color <yes/no>
               Define usage of colors
          --spool <directory>
               Alternative /var/spool/cmd
          --tty|-t
               Pretend a TTY is available
          --noredirect|-r
               Do not follow redirects
          --norc|-n
               Do not load cmshrc file on start-up
          --noquitconfirmation|-Q
               Do not ask for quit confirmation
          --echo|-x
               Echo all commands
          --quit|-q
               Exit immediately after error
          --disablemultiline|-m
               Disable multiline support
          --hide-events
               Hide all events by default
          --disable-events
              Disable all events by default
          --certificate|-i
               Specify alternative certificate
          --key|-k
               Specify alternative private key
Arguments:
          hostname
              The hostname or IP to connect to
          command
              A list of cmsh commands to execute
          filename
              A file which contains a list of cmsh commands to execute
Examples:
          cmsh                                    run in interactive mode
          cmsh -c 'device status'                 run the device status command and exit
          cmsh --hide-events -c 'device status'   run the device status command and exit, without
                                                  showing any events that arrive during this time
          cmsh -f some.file -q -x                 run and echo the commands from some.file, exit

Events can be suppressed in cmsh:

• The --hide-events flag suppresses the display of events in an interactive cmsh session.

• The --disable-events flag disables event processing entirely and is intended for non-interactive scripted commands. It is compatible with --wait options for commands within a cmsh session, allowing a script to suppress event output while still waiting for an operation to complete.

Example

[basecm11 ~]# cmsh --disable-events -c "softwareimage use dgx-image; createramdisk --wait"

2.5.1.8 Man Page For cmsh

There is also a man page for cmsh(8), which is a bit more extensive than the help text. It does not however cover the modes and interactive behavior.

2.5.2 Levels, Modes, Help, And Commands Syntax In cmsh

The top-level of cmsh is the level that cmsh is in when entered without any options.

To avoid overloading a user with commands, cluster management functionality has been grouped and placed in separate cmsh mode levels. Mode levels and associated objects for a level make up a hierarchy available below the top-level.

There is an object-oriented terminology associated with managing via this hierarchy. To perform cluster management functions, the administrator descends via cmsh into the appropriate mode and object, and carries out actions relevant to the mode or object.

For example, within user mode, an object representing a user instance, fred, might be added or removed. Within the object fred, the administrator can manage its properties. The properties can be data such as a password fred123, or a home directory /home/fred.

Figure 2.5 shows the top-level commands available in cmsh. These commands are displayed when help is typed in at the top-level of cmsh:

alias ......................... Set aliases
category ...................... Enter category mode
cert .......................... Enter cert mode
cloud ......................... Enter cloud mode
color ......................... Manage console text color settings
configurationoverlay .......... Enter configurationoverlay mode
connect ....................... Connect to cluster
delimiter ..................... Display/set delimiter
device ........................ Enter device mode
disconnect .................... Disconnect from cluster
edgesite....................... Enter edgesite mode
etcd .......................... Enter etcd mode
events ........................ Manage events
exit .......................... Exit from current object or mode
export ........................ Display list of aliases current list formats
fspart .......................  Enter fspart mode
group ......................... Enter group mode
groupingsyntax ................ Manage the default grouping syntax
help .......................... Display this help
hierarchy ..................... Enter hierarchy mode
history ....................... Display command history
kubernetes..................... Enter kubernetes mode
list .......................... List state for all modes
main .......................... Enter main mode
modified ...................... List modified objects
monitoring .................... Enter monitoring mode
network ....................... Enter network mode
nodegroup ..................... Enter nodegroup mode
partition ..................... Enter partition mode
powercircuit .................. Enter powercircuit mode
process ....................... Enter process mode
profile ....................... Enter profile mode
quit .......................... Quit shell
quitconfirmation .............. Manage the status of quit confirmation
rack .......................... Enter rack mode
refresh ....................... Refresh all modes
run ........................... Execute cmsh commands from specified file
session ....................... Enter session mode
softwareimage ................. Enter softwareimage mode
task .......................... Enter task mode
time .......................... Measure time of executing command
unalias ....................... Unset aliases
user .......................... Enter user mode
watch ......................... Execute a command periodically, showing output
wlm ........................... Enter wlm mode

Figure 2.5: Top level commands in cmsh

All levels inside cmsh provide these top-level commands.

Passing a command as an argument to help gets details for it:

Example

[myheadnode]% help run
Name:     run - Execute all commands in the given file(s)
Usage:    run [OPTIONS] <filename> [<filename2> ...]
Options:  -x, --echo
               Echo all commands
          -q, --quit
               Exit immediately after error
[myheadnode]%

In the general case, invoking help at any mode level or within an object, without an argument, provides two lists:

• Firstly, under the title of Top: a list of top-level commands.

• Secondly, under the title of the level it was invoked at: a list of commands that may be used at that level.

For example, entering session mode and then typing in help displays, firstly, output with a title of Top, and secondly, output with a title of session (some output ellipsized):

Example

[myheadnode]% session
[myheadnode->session]% help
============================ Top =============================
alias ......................... Set aliases
category ...................... Enter category mode
cert .......................... Enter cert mode
cloud ......................... Enter cloud mode
...
========================== session ===========================
id ....................... Display current session id
killsession .............. Kill a session
list ..................... Provide overview of active sessions
[myheadnode->session]%

2.5.2.1 Navigation Through Modes And Objects In cmsh

The major modes tree is shown in Appendix M.1.

The following notes can help the cluster administrator in navigating the cmsh shell:

• To enter a mode, a user enters the mode name at the cmsh prompt. The prompt changes to indicate that cmsh is in the requested mode, and commands for that mode can then be run.

• To use an object within a mode, the use command is used with the object name. In other words, a mode is entered, and an object within that mode is used. When an object is used, the prompt changes to indicate that that object within the mode is now being used, and that commands are applied for that particular object.

• To leave a mode, and go back up a level, the exit command is used. Similarly, if an object is in use, the exit command exits the object. At the top level, exit has the same effect as the quit command, that is, the user leaves cmsh and returns to the unix shell. The string .. is an alias for exit.

• The home command, which is aliased to /, takes the user from any mode depth to the top level.

• The path command at any mode depth displays a string that can be used as a path to the current mode and object, in a form that is convenient for copying and pasting into cmsh. The string can be used in various ways. For example, it can be useful to define an alias in .cmshrc (here).

  In the following example, the path command is used to print out a string. This string makes it easy to construct a bash shell command to run a list from the correct place within cmsh:

  Example

  [basecm11->configurationoverlay[slurm-client]->roles[slurmclient]]% list
  Name (key)
  ----------------------------
  slurmclient
  [basecm11->configurationoverlay[slurm-client]->roles[slurmclient]]% path
  home;configurationoverlay;use "slurm-client";roles;use slurmclient;

  Pasting the string into a bash shell, using the cmsh command with the -c option, and appending the list command to the string, replicates the session output of the list command:

  [basecm11 ~]# cmsh -c 'configurationoverlay;use "slurm-client";roles;use slurmclient; list'
  Name (key)
  ----------------------------
  slurmclient

  The following example shows the path command can also be used inside the cmsh session itself for convenience:

  Example

  [basecm11]% device
  [basecm11->device]% list
  Type             Hostname (key)  MAC                Category  Ip             Network     Status
  ---------------- --------------- ------------------ --------- -------------- ----------- ------
  EthernetSwitch   switch01        00:00:00:00:00:00            10.141.0.50    internalnet [ UP ]
  HeadNode         basecm11        00:0C:29:5D:55:46            10.141.255.254 internalnet [ UP ]
  PhysicalNode     node001         00:0C:29:7A:41:78  default   10.141.0.1     internalnet [ UP ]
  PhysicalNode     node002         00:0C:29:CC:4F:79  default   10.141.0.2     internalnet [ UP ]
  [basecm11->device]% exit
  [basecm11]% device
  [basecm11->device]% use node001
  [basecm11->device[node001]]% path
  home;device;use node001;
  [basecm11->device[node001]]% home
  [basecm11]% home;device;use node001    #copy-pasted from path output earlier
  [basecm11->device[node001]]%

A command can also be executed in a mode without staying within that mode. This is done by specifying the mode before the command that is to be executed within that node. Most commands also accept arguments after the command. Multiple commands can be executed in one line by separating commands with semi-colons.

A cmsh input line has the following syntax:

<mode> <cmd> <arg> …<arg>; …; <mode> <cmd> <arg> …<arg>

where <mode> and <arg> are optional. More precisely, using standard command-synopsis notation, the synopsis is:
[<mode>] <cmd> [<arg> ... ] [; ... ; [<mode>] <cmd> [<arg> ... ]]

Example

[basecm11->network]% device status basecm11; list
basecm11 ............ [   UP   ]
Name (key)    Type      Netmask bits  Base address  Domain name          Ipv6
------------- --------- ------------- ------------- -------------------- ----
externalnet   External  16            192.168.1.0   brightcomputing.com  no
globalnet     Global    0             0.0.0.0       cm.cluster
internalnet   Internal  16            10.141.0.0    eth.cluster
[basecm11->network]%

In the preceding example, while in network mode, the status command is executed in device mode on the host name of the head node, making it display the status of the head node. The list command on the same line after the semi-colon still runs in network mode, as expected, and not in device mode, and so displays a list of networks.

Inserting a semi-colon makes a difference, in that the mode is actually entered, so that the list displays a list of nodes (some output truncated here for convenience):

Example

[basecm11->network]% device; status basecm11; list
basecm11 ................ [   UP   ]
Type          Hostname (key)  MAC                Category   Ip             Network     Status
------------- --------------- ------------------ ---------  -------------- ----------- ------
HeadNode      basecm11        FA:16:3E:C8:06:D1             10.141.255.254 internalnet [ UP ]
PhysicalNode  node001         FA:16:3E:A2:9C:87  default    10.141.0.1     internalnet [ UP ]
[basecm11->device]%

2.5.3 Working With Objects

Modes in cmsh work with associated groupings of data called objects. For instance, device mode works with device objects, and network mode works with network objects.

The commands used to deal with objects have similar behavior in all modes. Not all of the commands exist in every mode, and not all of the commands function with an explicit object:

Working with objects with these commands is demonstrated with several examples in this section.

2.5.3.1 Working With Objects: use, exit

Example

[mycluster->device]% use node001
[mycluster->device[node001]]% status
node001 ............. [   UP   ]
[mycluster->device[node001]]% exit
[mycluster->device]%

In the preceding example, use node001 issued from within device mode makes node001 the current object. The prompt changes accordingly. The status command, without an argument, then returns status information just for node001, because making an object the current object makes subsequent commands within that mode level apply only to that object. Finally, the exit command exits the current object level.

2.5.3.2 Working With Objects: add, commit, remove

The commands introduced in this section have many implicit concepts associated with them. So an illustrative session is first presented as an example. What happens in the session is then explained in order to familiarize the reader with the commands and associated concepts.

Example

[mycluster->device]% add physicalnode node100 10.141.0.100
[mycluster->device*[node100*]]% commit
[mycluster->device[node100]]% category add test-category
[mycluster->category*[test-category*]]% commit
[mycluster->category[test-category]]% remove test-category
[mycluster->category*]% commit
Successfully removed 1 Categories
Successfully committed 0 Categories
[mycluster->category]% device remove node100
[mycluster->category]% device
[mycluster->device*]% commit
Successfully removed 1 Devices
Successfully committed 0 Devices
[mycluster->device]%

add: The add command creates an object within its associated mode, and in cmsh the prompt drops into the object level just created. Thus, at the start in the preceding example, within device mode, a new object, named node100, is added. For this particular object properties can also be set, such as the type (physicalnode), and IP address (10.141.0.100). The node object level ([node100*]) is automatically dropped into from device mode when the add command is executed. After execution, the state achieved is that the object has been created with some properties. However, it is still in a temporary, modified state, and not yet persistent.

Asterisk tags in the prompt are a useful reminder of a modified state, with each asterisk indicating a tagged object that has an unsaved, modified property. In this case, the unsaved properties are the IP address setting, the node name, and the node type.

• The add command—syntax notes:

• In most modes the add command takes only one argument, namely the name of the object that is to be created. However, in device mode an extra object-type, in this case physicalnode, is also required as argument, and an optional extra IP argument may also be specified. The response to “help add” while in device mode gives details:

  [myheadnode->device]% help add
  Name:
            add - Create a new device of the given type with specified hostname
  Usage:
            add <type> <hostname>
            add cloudnode <hostname> [provider]
            add physicalnode <hostname> [ip] [interface]
  Arguments:
            type          chassis, fabricresourcebox, fabricswitch, genericdevice, litenode,
                          cloudnode, dpu, physicalnode, headnode, powerdistributionunit,
                          racksensor, switch, unmanagednode
            interface     eg. ens3, bond0=ens3+ens4

commit: The commit command is a further step that actually saves any changes made after executing a command. In this case, in the second line, it saves the node100 object with its properties. The asterisk tag disappears for the prompt if settings for that mode level and below have been saved.

Conveniently, the top level modes, such as the category mode, can be accessed directly from within this level if the mode is stated before the command. So, stating the mode category before running the add command allows the specified category test-category to be added. Again, the test-category object level within category mode is automatically dropped into when the add command is executed.

• The -w|--wait option to commit:

• The commit command by default does not wait for a state change to complete. This means that the prompt becomes available right away. This means that it is not obvious that the change has taken place, which could be awkward if scripting with cmsh for cloning (discussed shortly) a software image (section 2.1.2). The -w|--wait option to the commit command works around this issue by waiting for any associated background task, such as the cloning of a software image, to be completed before making the prompt available.

remove: The remove command removes a specified object within its associated mode. On successful execution, if the prompt is at the object level, then the prompt moves one level up. The removal is not actually carried out fully yet; it is only a proposed removal. This is indicated by the asterisk tag, which remains visible, until the commit command is executed, and the test-category removal is saved. The remove command can also remove a object in a non-local mode, if the non-local mode is associated with the command. This is illustrated in the example where, from within category mode, the device mode is declared before running the remove command for node100. The proposed removal is configured without being made permanent, but in this case no asterisk tag shows up in the category mode, because the change is in device mode. To drop into device mode, the mode command “device” is executed. An asterisk tag then does appear, to remind the administrator that there is still an uncommitted change (the node that is to be removed) for the mode. The commit command would remove the object whichever mode it is in—the non-existence of the asterisk tag does not change the effectiveness of commit.

• The -d|--data option to remove:

• The remove command by default removes an object, and not the represented data. An example is if, in softwareimage mode, a software image is removed with the remove (without options) command. As far as the cluster manager is concerned, the image is removed after running commit. However the data in the directory for that software image is not removed. The -d|--data option to the remove command arranges removal of the data in the directory for the specified image, as well as removal of its associated object.

• The -a|--all option to remove:

• The remove command by default does not remove software image revisions. The -a|--all option to the remove command also removes all software image revisions.

2.5.3.3 Working With Objects: clone, modified, swap

Continuing on with the node object node100 that was created in the previous example, it can be cloned to node101 as follows:

Example

[mycluster->device]% clone node100 node101
Warning: The Ethernet switch settings were not cloned, and have to be set manually
[mycluster->device*[node101*]]% exit
[mycluster->device*]% modified
State  Type                     Name
------ ------------------------ -----------------------------------
+      Device                   node101
[mycluster->device*]% commit
[mycluster->device]%
[mycluster->device]% remove node100
[mycluster->device*]% commit
[mycluster->device]%

The modified command is used to check what objects have uncommitted changes, and the new object node101 that is seen to be modified, is saved with a commit. The device node100 is then removed by using the remove command. A commit executes the removal.

The “+” entry in the State column in the output of the modified command in the preceding example indicates the object is a newly added one, but not yet committed. Similarly, a “~” entry indicates an object that is to be removed on committing, while a blank entry indicates that the object has been modified without an addition or removal involved.

Cloning an object is a convenient method of duplicating a fully configured object. When duplicating a device object, cmsh will attempt to automatically assign a new IP address using a number of heuristics. In the preceding example, node101 is assigned IP address 10.141.0.101.

The attempt is a best-effort, and does not guarantee a sensibly-configured object. The cluster administrator should therefore inspect the result.

Sometimes an object may have been misnamed, or physically swapped. For example, node001 exchanged physically with node002 in the rack, or the hardware device eth0 is misnamed by the kernel and should be eth1. In that case it can be convenient to simply swap their names via the cluster manager front end rather than change the physical device or adjust kernel configurations. This is equivalent to exchanging all the attributes from one name to the other.

For example, if the two interfaces on the head node need to have their names exchanged, it can be done as follows:

[mycluster->device]% use mycluster
[mycluster->device[mycluster]]% interfaces
[mycluster->device[mycluster]->interfaces]% list
Type         Network device name  IP               Network
------------ -------------------- ---------------- --------------
physical     eth0 [dhcp]          10.150.4.46      externalnet
physical     eth1 [prov]          10.141.255.254   internalnet
[basecm11->device[mycluster]->interfaces]% swap eth0 eth1; commit
[basecm11->device[mycluster]->interfaces]% list
Type         Network device name  IP               Network
------------ -------------------- ---------------- --------------
physical     eth0 [prov]          10.141.255.254   internalnet
physical     eth1 [dhcp]          10.150.4.46      externalnet
[mycluster->device[mycluster]->interfaces]% exit; exit

2.5.3.4 Working With Objects: get, set, refresh

The get command is used to retrieve a specified property from an object, and set is used to set it:

Example

[mycluster->device]% use node101
[mycluster->device[node101]]% get category
test-category
[mycluster->device[node101]]% set category default
[mycluster->device*[node101*]]% get category
default
[mycluster->device*[node101*]]% modified
State  Type                     Name
------ ------------------------ -------------------------------
       Device                   node101
[mycluster->device*[node101*]]% refresh
[mycluster->device[node101]]% modified
No modified objects of type device
[mycluster->device[node101]]% get category
test-category
[mycluster->device[node101]]%

Here, the category property of the node101 object is retrieved by using the get command. The property is then changed using the set command. Using get confirms that the value of the property has changed, and the modified command reconfirms that node101 has local uncommitted changes.

The refresh command undoes the changes made. The modified command then confirms that no local changes exist. Finally the get command reconfirms that no local change took place.

Among the possible values a property can take on are strings and Booleans:

• A string can be set as a revision label for any object:

  Example

  [mycluster->device[node101]]% set revision "changed on 10th May"
  [mycluster->device*[node101*]]% get revision
  [mycluster->device*[node101*]]% changed on 10th May 2011

  This can be useful when using shell scripts with an input text to label and track revisions when sending commands to cmsh. How to send commands from the shell to cmsh is introduced in section 2.5.1.

• For Booleans, the values “yes”, “1”, “on” and “true” are equivalent to each other, as are their opposites “no”, “0”, “off” and “false”. These values are case-insensitive.

2.5.3.5 Working With Objects: clear

Example

[mycluster->device]% set node101 mac 00:11:22:33:44:55
[mycluster->device*]% get node101 mac
00:11:22:33:44:55
[mycluster->device*]% clear node101 mac
[mycluster->device*]% get node101 mac
00:00:00:00:00:00
[mycluster->device*]%

The get and set commands are used to view and set the MAC address of node101 without running the use command to make node101 the current object. The clear command then unsets the value of the property. The result of clear depends on the type of the property it acts on. In the case of string properties, the empty string is assigned, whereas for MAC addresses the special value 00:00:00:00:00:00 is assigned.

2.5.3.6 Working With Objects: list, format, sort

The list command is used to list objects in a mode. The command has many options. The ones that are valid for the current mode can be viewed by running help list. The -f|--format option is available in all modes, and takes a format string as argument. The string specifies what properties are printed for each object, and how many characters are used to display each property in the output line. In following example a list of objects is requested for device mode, displaying the hostname, switchports and ip properties for each device object.

Example

[basecm11->device]% list -f hostname:14,switchports:15,ip
hostname (key) switchports     ip
-------------- --------------- --------------------
apc01                          10.142.254.1
basecm11       switch01:46     10.142.255.254
node001        switch01:47     10.142.0.1
node002        switch01:45     10.142.0.2
switch01                       10.142.253.1
[basecm11->device]%

Running the list command with no argument uses the current format string for the mode. Running the list command with the -f option, and without specifying a size for the column keeps the column 20 characters wide.

Running the format command without arguments displays the current format string, and also displays all available properties including a description of each property. For example (output truncated):

Example

[basecm11->device]% format
Current list printing format:
-----------------------------
type:22, hostname:[16-32], mac:18, category:[16-32], ip:15, network:[14-32], status:[16-32]
Valid fields:
-------------
activation                : Date on which node was defined
additionalhostnames       : List of additional hostnames that should resolve to the interfaces IP address
allownetworkingrestart    : Allow node to update ifcfg files and restart networking
banks                     : Number of banks
...

The print specification of the format command uses the delimiter “:” to separate the parameter and the value for the width of the parameter column. For example, a width of 10 can be set with:

Example

[basecm11->device]% format hostname:10
[basecm11->device]% list
hostname (
----------
apc01
basecm11
node001
node002
switch01

Setting a width of 0 means that the column takes on the width of the widest entry.

A range of widths can be specified for a column, from a minimum to a maximum, using square brackets. When a range is set, then the column width matches at least the lower bound, and at most the upper bound. The text for the column is fitted into this range. If possible, the column width is shrunk to fit the text that is displayed. If the text is larger than the upper bound, then the text is truncated with a + to that width. For example:

Example

[basecm11->device]% format hostname:[10-14]
[basecm11->device]% list
hostname (key)
--------------
apc01
basecm11
node001
node002
switch01
largenameforn+

Here a node called largenamefornode is truncated to 14 characters, with the + replacing the 14th character.

The parameters to be viewed can be chosen from a list of valid fields by running the format command without any options, as shown earlier.

The format command can take as an argument a string that is made up of multiple parameters in a comma-separated list. Each parameter takes a colon-delimited width specification.

Example

[basecm11->device]% format hostname:[10-14],switchports:14,ip:20
[basecm11->device]% list
hostname (key) switchports    ip
-------------- -------------- --------------------
apc01                         10.142.254.1
basecm11       switch01:46    10.142.255.254
node001        switch01:47    10.142.0.1
node002        switch01:45    10.142.0.2
switch01                      10.142.253.1

The output of the format command without arguments shows the current list printing format string, with spaces. This can be used with enclosing quotes (").

In general, the string used in the format command can be set with enclosing quotes ("), or alternatively, with the spaces removed:

Example

[basecm11->device]% format "hostname:[16-32], network:[14-32], status:[16-32]"
    or
[basecm11->device]% format hostname:[16-32],network:[14-32],status:[16-32]

The default parameter settings can be restored with the -r|--reset option:

Example

[basecm11->device]% format -r
[basecm11->device]% format | head -3
Current list printing format:
-----------------------------
type:22, hostname:[16-32], mac:18, category:[16-32], ip:15, network:[14-32], status:[16-32]
[basecm11->device]%

The sort command sorts output in alphabetical order for specified parameters when the list command is run. The sort is done according to the precedence of the parameters passed to the sort command:

Example

[basecm11->device]% sort type mac
[basecm11->device]% list -f type:15,hostname:15,mac
type            hostname (key)  mac
--------------- --------------- --------------------
HeadNode        basecm11        08:0A:27:BA:B9:43
PhysicalNode    node002         00:00:00:00:00:00
PhysicalNode    log001          52:54:00:DE:E3:6B
[basecm11->device]% sort type hostname
[basecm11->device]% list -f type:15,hostname:15,mac
type            hostname (key)  mac
--------------- --------------- --------------------
HeadNode        basecm11        08:0A:27:BA:B9:43
PhysicalNode    log001          52:54:00:DE:E3:6B
PhysicalNode    node002         00:00:00:00:00:00
[basecm11->device]% sort mac hostname
[basecm11->device]% list -f type:15,hostname:15,mac
type            hostname (key)  mac
--------------- --------------- --------------------
PhysicalNode    node002         00:00:00:00:00:00
HeadNode        basecm11        08:0A:27:BA:B9:43
PhysicalNode    log001          52:54:00:DE:E3:6B