> This page is for version 11 (default). > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Cluster Extension Cloudbursting Cluster Extension cloudbursting (“hybrid” cloudbursting) in NVIDIA Base Command Manager is the case when a cloud service provider is used to provide nodes that are in the cloud as an extension to the number of regular nodes in a cluster. Thus, the head node in a Cluster Extension configuration is always outside the cloud, and there may be some non-cloud-extension regular nodes that are outside the cloud too. Cluster Extension cloudbursting can burst into a cloud that is running within: * AWS (CX-AWS). This is described in Chapters 3 and [4](/cloudbursting/clicloud) of the *Cloudbursting Manual*. * Azure (CX-Azure). This is described in Chapter [5](/cloudbursting/cexcloudazure) of the *Cloudbursting Manual*. #### 3.0.0.1 Requirements Cluster Extension cloudbursting requires: * **An activated cluster license.** One does not simply cloudburst right away in a Cluster Extension configuration. The license must first be made active, or the attempt will fail. A check on the state of the license can be carried out with: **Example** ```bash [root@basecm11 ~]# cmsh -c "main; licenseinfo" License Information --------------------------------- ---------------------------------------- Licensee /C=US/ST=NY/L=WS/O=BCM Mordor/OU=Mt. Doom/CN=BCM 11 Cluster Serial number 1768388 Start time Mon Oct 16 01:00:00 2022 End time Fri Dec 31 23:59:00 2038 Version 10.0 and above Edition Advanced Type Commercial Licensed nodes 6 / 100 Licensed accelerators 0 / 0 Accelerators details 6 * 0 gpu Allow edge sites Yes MAC Address / Cloud ID FA:16:3E:91:14:86 ``` The value of `End time`, and `Type` in the preceding license should be checked. If activation is indeed needed, then simply running the `request-license` command with the product key should in most cases provide activation. Further details on activating the license are given in [Chapter 4 of the *Installation Manual*](/installation/licensing). * **An Amazon account**, if the cloud provider is Amazon. * **An Azure account**, if the cloud provider is Microsoft Azure. * **An open UDP port.** By default, this is port 1194. It is used for the OpenVPN connection from the head node to the cloud and back. To use TCP, and/or ports other than 1194, the Knowledge Articles in the NVIDIA Enterprise Support Portal at [https://enterprise-support.nvidia.com/](https://enterprise-support.nvidia.com/) can be consulted using the keywords “openvpn port”. Outbound SSH access from the head node is also useful, but not strictly required. By default, the Shorewall firewall as provided by BCM on the head node is configured to allow all outbound connections, but other firewalls may need to be considered too. * **A special proxy environment configuration setting**, if an HTTP proxy is used to access the AWS or Azure APIs. The proxy environment configuration is carried out using the `ScriptEnvironment` directive [here](/admin/cmdconf#c82-scriptenvironment-directive), which is a CMDaemon directive that can be set and activated [here](/admin/cmdconf). For example, if the proxy host is `my.proxy` and accepting connections on port `8080` with a username `my` and password `pass`, then the setting can be specified as: ```bash ScriptEnvironment = { "http_proxy=http://my:pass@my.proxy:8080", \ "https_proxy=http://my:pass@my.proxy:8080", "ftp_proxy=http://my:pass@my.proxy:8080" } ``` #### 3.0.0.2 Steps Cluster Extension cloudbursting uses a *cloud director*. A cloud director is a specially connected cloud node used to manage regular cloud nodes, and is described more thoroughly in section [3.2](#32-cluster-extension-with-aws-cloud-director-startup-from-scratch). Assuming the administrator has ownership of a cloud provider account, the following steps can be followed to launch Cluster Extension cloud nodes: 1. The cloud provider is logged into from Base View, and a cloud director is configured (section [3.1](#31-cluster-extension-with-aws-the-base-view-cluster-extension-wizard)). 2. The cloud director is started up (section [3.2](#32-cluster-extension-with-aws-cloud-director-startup-from-scratch)). 3. The cloud nodes are provisioned from the cloud director (section [3.3](#33-cluster-extension-with-aws-cloud-node-startup-from-scratch)). The cloud nodes then become available for general use by the cluster. #### 3.0.0.3 Cluster Extension Cloudbursting With A Hardware VPN BCM recommends, and provides, OpenVPN by default for Cluster Extension cloudbursting VPN connectivity. If there is a wish to use a hardware VPN, for example if there is an existing hardware VPN network already in use at the deployment site, then BCM can optionally be configured to work with the hardware VPN using the cluster extension configuration wizard. The wizard can also guide the cluster administrator on VPN configuration for AWS (section [4.4](/cloudbursting/clicloud#44-connecting-to-aws-with-direct-connect-or-a-hardware-vpn)) or Azure (section [5.5](/cloudbursting/cexcloudazure#55-creating-an-azure-cluster-extension-using-expressroute-or-a-hardware-vpn)). #### 3.0.0.4 Cluster Extension Cloudbursting Logging All AWS logging goes to the CMDaemon logs in `/var/log/cmdaemon`. The `CLOUD` tag in the log is used to indicate cloud-related operations. ## 3.1 Cluster Extension With AWS: The Base View Cluster Extension Wizard Cluster Extension with the Base View cluster extension wizard is described in sections [3.1](#31-cluster-extension-with-aws-the-base-view-cluster-extension-wizard), [3.2](#32-cluster-extension-with-aws-cloud-director-startup-from-scratch), and [3.3](#33-cluster-extension-with-aws-cloud-node-startup-from-scratch). Cluster Extension with Azure is described in Chapter [5](/cloudbursting/cexcloudazure). The Amazon cloud service can be configured for Cluster Extension from Base View. This can be done via Base View (`https://` \<*head node address*>`:8081/base-view/`), and then selecting the navigation path `Cloud` > `AWS` > `AWS Wizard`. A screen introducing the `AWS Wizard` is then displayed. The wizard goes through the following stages: 1. Introduction (section [3.1.1](#311-introduction)) 2. AWS Credentials (section [3.1.2](#312-aws-credentials)) 3. Select Regions (section [3.1.3](#313-select-regions)) 4. Summary & Deployment (section [3.1.7](#317-summary-deployment)) 5. Deploy (section [3.1.8](#318-deploy)) ### 3.1.1 Introduction The first screen displayed by the wizard is the introduction screen (figure [3.1](#fig-bright-view-clouds-add-cloud-provider-wizard-intro)), which reminds the administrator about the prerequisites for cloudbursting. ![Introduction Page For Cluster Extension With Base View](/base-command-manager/product-manuals/_fern-img/da859e726bc090cd210cebf5f42461913398a1f219cbb3a5a7075ab01a783ef2.webp) *Figure 3.1: Introduction Page For Cluster Extension With Base View* During the wizard session: * The `Load config` button allows the wizard to load a previously-saved YAML configuration text file from a file location accessible by the browser. * The `Show config` button allows the wizard to show a YAML configuration text file. The file can be saved to a location by the browser, by using a `Save` button that is available when the configuration is shown. Saving the configuration at the start and end of a wizard run is usually convenient for an administrator who wishes to record the changes that are being made. ### 3.1.2 AWS Credentials The `Next` button brings up the credentials page, if the credentials for the cluster extension cloudburst are not yet known to CMDaemon (figure [3.2](#fig-bright-view-clouds-add-cloud-provider-wizard-accountinfo)). ![AWS Key Configuration For Cluster Extension With Base View](/base-command-manager/product-manuals/_fern-img/b40f1c1780dcfe1b70767daa1ed07368787bfdf568313e375d3f74fc2b73d544.webp) *Figure 3.2: AWS Key Configuration For Cluster Extension With Base View* This asks for: * `Provider Name:` This is can be any user-defined value. If using Amazon, it would be sensible to just put in `amazon` * `AWS Access key ID:` The AWS Access key. Typically a string that is a mixture of upper case letters and numbers. * `AWS Secret key:` The AWS secret key. Typically a longer string made up of alphanumeric characters. In the case of Amazon, the information is obtainable after signing up for Amazon Web Services (AWS) at [https://console.aws.amazon.com/iam/home?#security_credential](https://console.aws.amazon.com/iam/home?#security_credential). The `Validate` button validates the credentials at this point of the configuration when clicked, instead of waiting until the actual deployment. Clicking the `Next` button submits the details of this screen, and inputs for the next screen are then retrieved from Amazon. ### 3.1.3 Select Regions If all goes well, the next screen (figure [3.3](#fig-bright-view-clouds-add-cloud-provider-wizard-defaultoptions)) displays the major region selection options for the Amazon cloud service. ![Selecting Regions With The Cluster Extension Wizard With Base View](/base-command-manager/product-manuals/_fern-img/2189279146e355852cfeb81839fbef932fcb6130772e19de917d910261b63dc2.webp) *Figure 3.3: Selecting Regions With The Cluster Extension Wizard With Base View* The regions are designated by codes, for example `us-east-1`, `eu-west-1`, and so on. The first two letters associated with the region imply a major region as follows: * `af:` Africa * `ap:` Asia Pacific * `ca:` Canada * `cn:` China * `eu:` Europe * `me:` Middle East * `sa:` South America * `us:` United States In addition, the special `us-gov` prefix designates a region that helps customers comply with US government regulations. Similarly, European data regulations may, for example, mean that data should be processed only within an `eu` region. Other than legislative considerations, choosing capacity from a region that is geographically closer is often sensible for avoiding lag with certain applications. On the other hand, using off-peak capacity from a geographically distant location may make more sense if it is cheaper. Further details on regions can be found at [http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html). After the administrator has selected the regions that are to be used in deployment, the `Next` button can be clicked, to bring up the next screen. ### 3.1.4 Select Availability Zones The `Select Availability Zones` screen (figure [3.4](#fig-bright-view-clouds-add-cloud-provider-wizard-availability)) allows subnets in the region to be specified for the private and public subnets of the cluster extension. ![Selecting Availability Zones With The Cluster Extension Wizard With Base View](/base-command-manager/product-manuals/_fern-img/c8798fd7efc3ce4223fd66215c19a96329e94db9f639b7ed666660ff4d9aa76b.webp) *Figure 3.4: Selecting Availability Zones With The Cluster Extension Wizard With Base View* ### 3.1.5 Select Software Images Next, the `Select Software Images` screen (figure [3.5](#fig-bright-view-clouds-add-cloud-provider-wizard-cloud-images)) lets the administrator select possible cloud node software images to be placed on the cloud director node. These can then be provisioned from the director to the cloud nodes. A default cloud node image is selected from one of the possible cloud node software images. ![Selecting Images For The Cloud Director Cloud Nodes With The Cluster Extension Wizard With Base View](/base-command-manager/product-manuals/_fern-img/e29bb8a09392f63658a5d4cec12dc4c394d72e9669f1eadb2c7eaad9fb2ad2c4.webp) *Figure 3.5: Selecting Images For The Cloud Director Cloud Nodes With The Cluster Extension Wizard With Base View* ### 3.1.6 Select Cloud Types Next, the default cloud director type and default cloud node type can be selected (figure [3.6](#fig-bright-view-clouds-add-cloud-provider-wizard-cloud-type)) . The type specifies the virtual hardware type. ![Selecting Cloud Types With The Cluster Extension Wizard With Base View](/base-command-manager/product-manuals/_fern-img/1422220fb87c6d70444c9e9bfadd5edcf322ba301097d95238b289e559ef3fea.webp) *Figure 3.6: Selecting Cloud Types With The Cluster Extension Wizard With Base View* ### 3.1.7 Summary & Deployment The next screen is a `Summary` screen (figure [3.7](#fig-bright-view-clouds-add-cloud-provider-wizard-summary)). ![Summary Screen For The Cluster Extension Wizard With Base View](/base-command-manager/product-manuals/_fern-img/8d0362882bbd576b35627236fce622399678832fbe74f3867f7e0f18f4b8709b.webp) *Figure 3.7: Summary Screen For The Cluster Extension Wizard With Base View* It summarizes the selections that have been made, and lets the administrator review them. The selections can be changed, if needed, by going back to the previous screens, by directly clicking on the values. Otherwise, on clicking the `Deploy` button, the configuration is deployed. ### 3.1.8 Deploy During configuration deployment, a progress meter indicates what is happening as the configuration is processed. Logs showing the more detailed steps can also be viewed. At the end of processing, the display indicates with the `Deployed with success` message that the cluster has been extended successfully (figure [3.8](#fig-bright-view-clouds-add-cloud-provider-wizard-deploying)). ![Cluster Extension Configuration Processing With Base View](/base-command-manager/product-manuals/_fern-img/edc57ed214e6fa0e74d3298ab3fcfa0a308a06648fa78f3e670ed0265dc85649.webp) *Figure 3.8: Cluster Extension Configuration Processing With Base View* No nodes are activated yet within the cloud provider service. To start them up, the components of the cloud provider service must be started up by * powering up the cloud directors (section [3.2](#32-cluster-extension-with-aws-cloud-director-startup-from-scratch)) * powering on the cloud nodes after the cloud directors are up. Often this involves creating new cloud nodes (section [3.3](#33-cluster-extension-with-aws-cloud-node-startup-from-scratch)). ## 3.2 Cluster Extension With AWS: Cloud Director Startup From Scratch The cloud director can take some time to start up the first time when it is *installing from scratch*. The bottleneck is usually due to several provisioning stages, where the bandwidth between the head node and the cloud director means that the provisioning runs typically take tens of minutes to complete. The progress of the cloud director can be followed in the Base View event log viewer ([section 10.2.11 of the *Administrator Manual*](/admin/monitoring#10211-conceptual-overview-the-main-monitoring-interfaces-of-base-view)), or they can be followed in an open `cmsh` session, as the events are sent to the `cmsh` session. The bottleneck is one of the reasons why the cloud director is put in the cloud in the first place: nodes are provisioned from a cloud director in the cloud faster than from a head node outside the cloud. The bottleneck of provisioning from the head node to the cloud director is an issue only the first time around. The next time the cloud director powers up, and assuming persistent storage is used—as is the default—the cloud director runs through the provisioning stages much faster, and completes within a few minutes. The reason why powering up after the first time is faster is because the image that is to power up is already in the cloud. A similar principle—of relying on data already available with the cloud provider—can be used as a technique to make first time startup even faster. The technique is to have a pre-built image—a snapshot—of the cloud director stored already with the cloud provider. The first-time startup of a cloud director based on a snapshot restoration is discussed in section [3.4](#34-cluster-extension-with-aws-cloud-director-and-cloud-node-startup-from-snapshots). The remainder of this section is about starting up a cloud director from scratch—that is, a first time start, and without a pre-built image. To recap: by default, a cloud director object is created during a run of the Cluster Extension wizard (section [3.1](#31-cluster-extension-with-aws-the-base-view-cluster-extension-wizard)). There can be only one cloud director active per region for a cluster. Because a cloud director also has properties specific to the region within which it directs nodes, it means that cloud directors can only be created from scratch, via cluster extension. Once a cloud director object has been made in CMDaemon, then the cloud director is ready to be started up. In Base View the cloud director can be started by powering it up from its node settings, just like a regular node. If the cloud director node is not visible, then a browser refresh should clear up the cache so that it becomes visible. For the cloud director, a navigation path to power it up is: `Devices` > `Cloud Nodes` > `Cloud director` > `Power` > `On` As indicated earlier on, the cloud director acts as a helper instance in the cloud. It provides some of the functions of the head node within the cloud, in order to speed up communications and ensure greater resource efficiency. Amongst the functions the cloud director provides are: * Cloud nodes provisioning * Exporting a copy of the shared directory `/cm/shared` to the cloud nodes so that they can mount it * Providing routing services using an OpenVPN server. While cloud nodes within a region communicate directly with each other, cloud nodes in one region use the OpenVPN server of their cloud director to communicate with the other cloud regions and to communicate with the head node of the cluster. Cloud directors are not regular nodes, so they have their own category, `cloud-director`, into which they are placed by default. The cloud-related properties of the cloud director can be viewed and edited via the navigation path: `Devices` > `Cloud Nodes` > `Cloud director` > `Edit` ### 3.2.1 Setting The Cloud Director Disk Storage Device Type Amazon provides two kinds of storage types as part of EC2: 1. **Instance storage**, using so-called ephemeral devices. Ephemeral means that the device is temporary, and means that whatever is placed on it is lost if the instance is stopped, terminated, or if the underlying disk drive fails. Some instances have ephemeral storage associated with the instance type. For example, at the time of writing (May 2017), the `m3.medium` type of instance has 4GB of SSD storage associated with it. Details on instance storage can be found at [http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html#instance-store-volumes.](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html#instance-store-volumes.) 2. **Elastic Block Storage (EBS) volumes:** EBS is a persistent, reliable, and highly available storage. Normally, EBS is suggested for cloud director and cloud node use. The reasons for this include: * it can be provided to all nodes in the same availability zone * unlike instance storage, EBS remains available for use when an instance using it is stopped or terminated. * instance storage is not available for many instance types such as `t2.micro`, `t2.small`, `c4.large`. #### 3.2.1.1 Using The Ephemeral Device As The Drive For The Cloud Director: Since the cloud director instance type is essential, and contains so much data, it is rare to use an ephemeral device for its storage. However, if for some reason the administrator would like to avoid using EBS, and use the instance storage, then this can be done by removing the default EBS volume suggestion for the cloud director provided by BCM. When doing this, the ephemeral device that is used as the replacement must be renamed. It must take over the name that the EBS volume device had before it was removed. * In Base View, this can be done in the `EC2 Storages` window, which for a cloud director \<*cloud director hostname*> can be viewed and modified via the navigation path: `Devices` > `Cloud Nodes` > \<*cloud director hostname*> > `Edit` > `Settings` > `Cloud settings` >\ `STORAGE` > `Storage` > \<*storage type*> > `Edit` * In `cmsh`, this can be done in `device` mode, by going into the `cloudsettings` submode for the cloud director, and then going a level deeper into the `storage` submode. Within the `storage` submode, the `list` command shows the values of the storage devices associated with the cloud director. The values can be modified as required with the usual object commands. The `set` command can be used to modify the values. **Example** ```bash [basecm11]% device use us-east-1-director [basecm11->device[us-east-1-director]]% cloudsettings [basecm11->device[us-east-1-director]->cloudsettings]% storage [basecm11->...->cloudsettings->storage]% list Type Name (key) Drive Size Volume ID ---------- ------------ --------- ------- ---------- ebs ebs sdb 42GB ephemeral ephemeral0 sdc 0B ephemeral0 [basecm11->...->cloudsettings->storage]% remove ebs [basecm11->...->cloudsettings*->storage*]% set ephemeral0 drive sdb [basecm11->...->cloudsettings*->storage*]% list Type Name (key) Drive Size Volume ID ---------- ------------ --------- ------- --------- ephemeral ephemeral0 sdb 0B ephemeral0 [basecm11->...->cloudsettings*->storage*]% commit ``` ### 3.2.2 Setting The Cloud Director Disk Size The disk size for the cloud director can be set in Base View using the `EC2 Storages` window (section [3.2.1](#321-setting-the-cloud-director-disk-storage-device-type)). By default, an EBS volume size of 42GB is suggested. This is as for a standard node layout ([section D.3 of the *Administrator Manual*](/admin/partitioning#d3-example-default-node-partitioning)), and no use is then made of the ephemeral device. 42GB on its own is unlikely to be enough for most purposes other than running basic `hello world` tests. In actual use, the most important considerations are likely to be that the cloud director should have enough space for: * the user home directories (under `/home/`) * the cluster manager shared directory contents, (under `/cm/shared/`) * the software image directories (under `/cm/images/`) The cluster administrator should therefore properly consider the allocation of space, and decide if the disk layout should be modified. An example of how to access the disk setup XML file to modify the disk layout is given in [section 3.13.3 of the *Administrator Manual*](/admin/configuring#3133-changing-disk-layouts). For the cloud director, an additional sensible option may be to place `/tmp` and the swap space on an ephemeral device, by appropriately modifying the XML layout for the cloud director. ### 3.2.3 Tracking Cloud Director Startup #### 3.2.3.1 Tracking Cloud Director Startup From The EC2 Management Console The boot progress of the cloud director \<*cloud director*> can be followed by watching the status of the instance in the Amazon EC2 management console, as illustrated in figure [2.2](/cloudbursting/codcloud#291-status-checking-using-instance-selection-from-instances-list). The `Instance ID` that is used to identify the instance can be found * with Base View, within the navigation path\ `Devices` > `Cloud Nodes` > \<*cloud director*> > `Edit` > `Settings` > `Cloud settings` > `Instance ID` * with `cmsh`, by running something like: **Example** ```bash [basecm11]% device use us-east-1-director [basecm11->device[us-east-1-director]]% get cloudid i-f98e7441 [basecm11->device[us-east-1-director]]% cloudsettings [basecm11->device[us-east-1-director]->cloudsettings]% get instanceid i-f98e7441 ``` #### 3.2.3.2 Tracking Cloud Director Startup From Base View The boot progress of the cloud director can also be followed by * watching the icon changes for the cloud node states in the navigation path `Devices` > `Cloud Nodes`. The icons indicating the state follow the description given in [section 5.5.1 of the *Administrator Manual*](/admin/provisioning#551-node-states-icons-in-base-view) #### 3.2.3.3 Tracking Cloud Director Startup From The Bash Shell Of The Head Node There are some further possibilities to view the progress of the cloud director after it has reached at least the initrd stage. These possibilities include: * an SSH connection to the cloud director can be made during the pre-init, initrd stage, after the cloud director system has been set up via an rsync. This allows a login to the node-installer shell. * an SSH connection to the cloud director can be also be made after the initrd stage has ended, after the init process runs making an SSH daemon available again. This allows a login on the cloud director when it is fully up. During the initrd stage, the cloud director is provisioned first. The cloud node image(s) and shared directory are then provisioned on the cloud director, still within the initrd stage. To see what rsync is supplying to the cloud director, the command “`ps uww -C rsync`” can be run on the head node. Its output can then be parsed to make obvious the source and target directories currently being transferred: **Example** ```bash [root@basecm11 ~]# ps uww -C rsync | grep -o ' /cm/.*$' /cm/shared/ syncer@172.21.255.251::target//cm/shared/ ``` #### 3.2.3.4 Tracking Cloud Director Startup From `cmsh` The `provisioningstatus` command in `cmsh` can be used to view the provisioning status (some output elided): **Example** ```bash [root@basecm11 ~]# cmsh -c "softwareimage provisioningstatus" ... + us-east-1-director ... Up to date images: none Out of date images: default-image ``` In the preceding output, the absence of an entry for “`Up to date images`” shows that the cloud director does not yet have an image that it can provision to the cloud nodes. After some time, the last few lines of output should change to something like: **Example** ```bash + us-east-1-director ... Up to date images: default-image ``` This indicates the image for the cloud nodes is now ready. With the `-a` option, the `provisioningstatus -a` command gives details that may be helpful. For example, while the cloud director is having the default software image placed on it for provisioning purposes, the source and destination paths are `/cm/images/default-image`: **Example** ```bash [root@basecm11 ~]# cmsh -c "softwareimage provisioningstatus -a" Request ID(s): 4 Source node: basecm11 Source path: /cm/images/default-image Destination node: us-east-1-director Destination path: /cm/images/default-image ... ``` After some time, when the shared filesystem is being provisioned, the source and destination paths should change to the `/cm/shared` directory: ```bash [root@basecm11 ~]# cmsh -c "softwareimage provisioningstatus -a" Request ID(s): 5 Source node: basecm11 Source path: /cm/shared Destination node: us-east-1-director Destination path: /cm/shared ... ``` After the shared directory and the cloud node software images are provisioned, the cloud director is fully up. Cloud node instances can then be powered up and provisioned from the cloud director. ## 3.3 Cluster Extension With AWS: Cloud Node Startup From Scratch This section discusses the configuration of regular cloud node startup from scratch. Configuration of cloud node startup from snapshot is discussed in section [3.4](#34-cluster-extension-with-aws-cloud-director-and-cloud-node-startup-from-snapshots). Regular cloud nodes are the cloud nodes that the cloud director starts up. To configure the regular cloud nodes does not require a working cloud director. However to boot up the regular cloud nodes does require that the cloud director be up, and that the associated networks to the regular cloud nodes and to the head node be configured correctly. If needed, additional cloud provisioning nodes ([section 5.2 of the *Administrator Manual*](/admin/provisioning#52-provisioning-nodes)) can be configured by assigning the provisioning role to cloud nodes, along with appropriate nodegroups ([here](/admin/provisioning#521-provisioning-nodes-configuration-settings)) values, in order to create a provisioning hierarchy. Creation and configuration of regular cloud node objects is conveniently carried out by cloning another regular cloud node from one of the default cloud nodes already created by the cluster extension wizard (section [3.1](#31-cluster-extension-with-aws-the-base-view-cluster-extension-wizard)). A navigation path is: `Device` > `Cloud Nodes` > \<*cloud node hostname*> > `Clone` Cloud node objects can also be created in `cmsh` as described in section [4.2](/cloudbursting/clicloud#42-launching-the-cloud-nodes). The internal network for the regular cloud nodes is by default set to the VPC private network, and is somewhat similar to the internal network for regular nodes. The VPC private network can be contrasted with the VPC public network for cloud directors, which is a network that is by default assigned to cloud directors, and which floating IP addresses can connect to. Both the VPC private and VPC public networks are subnets of a cloud network. If the administrator would like to do so, the regular cloud nodes can be placed in the VPC public network and become directly accessible to the public. If the cloud director is up, then the cloud nodes can be booted up by powering them up ([section 4.2 of the *Administrator Manual*](/admin/introduction)) by category, or individually. ## 3.4 Cluster Extension With AWS: Cloud Director And Cloud Node Startup From Snapshots A technique that speeds up cluster deployment in the cloud is to use snapshots to start up the nodes in the cloud. Snapshots are snapshots of a shutdown state, and are stored by the cloud provider. In Amazon, they can be stored in EBS. It is cheaper to keep a machine in a stored state, rather than have it up but idling. Restoring from a snapshot is also significantly faster than starting up from scratch, due to optimizations by the cloud provider. An administrator should therefore get around to looking at using snapshots once cloudbursting is set up and the usage pattern has become clearer. As a part of regular maintenance, snapshot configuration can be repeated whenever cloud director and cloud node files change significantly, in order to keep usage efficiency up. ### 3.4.1 Cloud Director Startup From Snapshots #### 3.4.1.1 Cloud Director Snapshot Preparation A cloud director, for example `us-east-1-director`, can have a snapshot of its state prepared as follows by the administrator: * The cloud director is started up from scratch (section [3.2](#32-cluster-extension-with-aws-cloud-director-startup-from-scratch)) * After it comes up for the first time, the administrator shuts it down cleanly. For example, with a command similar to `cmsh -c "device use us-east-1-director; shutdown"` * After the cloud-director shutdown is complete, the administrator creates a snapshot of a cloud director using the EC2 Management Console. This can be done by selecting Elastic Block Store in the navigator column, then selecting the `Volumes` item within that menu. The volume associated with the cloud director can be identified by matching the `Attachment Information` column value with the name `us-east-1-director` for this node, and the device to be snapshotted. In a default configuration, the device is `/dev/sdb` at the time of writing, but that may change. The `Actions` button in the main pane then provides a `Create Snapshot` item (figure [3.9](#fig-amazon-ec2-ebs-volumes-actions)). ![Creating A Snapshot From A Selected Volume](/base-command-manager/product-manuals/_fern-img/590e8329dba365e1b16b382ee917c8fdb32155ca394ba15a94651bae6547fbe5.webp) *Figure 3.9: Creating A Snapshot From A Selected Volume* Using it creates a snapshot of a selected volume instance via a dialog. The snapshot ID is displayed at the end of the snapshot creation dialog, and should be noted for CMDaemon use later on, where it is saved as the value of `snapshotid`. Created snapshots can be viewed within the `Snapshots` item of the Elastic Block Store menu. #### 3.4.1.2 Cloud Director Launch From Prepared Snapshot To allow CMDaemon to launch the cloud director from the snapshot, the following procedure can be followed: * The instance must be terminated so that the snapshot can actually be used by the instance on starting it again: **Example** ```bash [basecm11->device[us-east-1-director]]% terminate us-east-1-director terminated ``` * The snapshot ID that was noted earlier during snapshot preparation is set in the EBS storage setting configuration of the CMDaemon database, using a session similar to: **Example** ```bash [root@basecm11 ~]# cmsh [basecm11]% device [basecm11->device]% use us-east-1-director [basecm11->device[us-east-1-director]]% cloudsettings [basecm11->...-director]->cloudsettings]% storage [basecm11->...-director]->cloudsettings->storage]% use ebs [basecm11->...-director]->cloudsettings->storage[ebs]]% set snapshotid snap-2a96d0c6 ``` The cloud director can now be powered on: **Example** ```bash [basecm11->device[us-east-1-director]]% power on ``` The cloud director now starts up much faster than when starting up from scratch. ### 3.4.2 Cloud Node Startup From Snapshots When a regular cloud node is launched from scratch (section [3.3](#33-cluster-extension-with-aws-cloud-node-startup-from-scratch)), it uses the cloud director for provisioning, rather than a node outside the cloud, because this is faster. However, having the cloud director create an EBS volume from its storage in the cloud, and then providing the image to the cloud compute nodes still involves a lot of data I/O. On the other hand, a cloud provider such as Amazon can optimize many of these steps when creating an EBS volume from a snapshot, for example, by using copy-on-write. This means that snapshot-based provisioning is even speedier than the non-snapshot, “from scratch” method. If the administrator wants to make a snapshot that can be used as the base for speedily launching regular cloud nodes, then the same snapshot method that is used for cloud directors (section [3.4.1](#341-cloud-director-startup-from-snapshots)) should be followed to make a snapshot for a regular cloud node. A summary of the steps that can be followed is: * a regular cloud node is started up from scratch (section [3.3](#33-cluster-extension-with-aws-cloud-node-startup-from-scratch)), after the cloud director is up * after the regular cloud node has come up, it is shut down cleanly * a snapshot is created of the cloud node using the EC2 Management Console * the cloud node is terminated * the snapshot ID is set: **Example** ```bash [basecm11->device[cnode001]->cloudsettings->storage[ebs]]% set snapshotid snap-5c9s3991 ``` Powering on the node now launches the regular cloud node much faster than the non-snapshot method. CMDaemon ensures that a snapshot for one cloud node can be used by other cloud nodes too, if the disk partitioning is the same. This is useful when launching cloud nodes that do not differ much from the snapshot. It also means that even the cloud director image can be used as a snapshot to launch a regular cloud node, if the disk partitioning and other settings allow it. However, using a regular node snapshot for launch is usually much wiser, due to the extra filesystems that a cloud director has. ## 3.5 Cluster Extension With AWS: Optimizing AWS For High Performance Computing (HPC) For HPC, performance is a central concern. Optimization for the jobs that are run can be carried out by the administrator considering what are the most cost-effective aspects of the system that can be improved. The considerations in this section for AWS, apply to COD instances as well as to Cluster Extension instances. ### 3.5.1 Optimizing HPC Performance: EBS Volume Type The volume type used for EBS storage can be considered. AWS provides the gp2, io1, sc1, st1, and standard storage volume types. These volume types, and their associated IOPS performances, are described at [https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html). * In `cmsh` these can be set in the storage mode: **Example** ```bash [basecm11->device[cnode001]->cloudsettings->storage[ebs]]% set volumetype io1 ``` * In Base View, the equivalent navigation path is: `Devices` > `Cloud Nodes` > \<*cloud node hostname*> > `Edit` > `Settings` > `Cloud settings` >\ `STORAGE` > `Storage` > `ebs` > `CREATION TIME` > `Volume type` ### 3.5.2 Optimizing HPC Performance: Placement Groups The locality of the cloud nodes with respect to each other and with respect to the cluster that they extend from can be considered. For cloud nodes in the same placement group, lag times are minimized between the nodes. For cloud nodes that are geographically near cluster that they extend from, the lag times are reduced between the cloud nodes and the cluster they extend from. Localizing HPC cloud nodes within the same placement group is usually desirable. This can be achieved using the AWS web console to access the cluster extension AWS account. A placement group can be created for a region via the AWS web console for the instance. The navigation path to carry this out is `Services` > `EC2` > `Services` > `Placement Groups` > `Create Placement Group`. A name should be set. The value of `Strategy` should be set to `Cluster`, to localize the nodes. The same placement group name can then be set via `cmsh` for the not-yet-instantiated cloud nodes. Setting this means that those nodes are now started in that placement group. **Example** ```bash [basecm11->device[cnode001]->cloudsettings]% set placementgroup indahood ``` ### 3.5.3 Optimizing HPC Performance: Disabling Hyper-Threading Hyper-Threading (HT) in many cases hinders HPC performance. When running cluster extension cloud nodes, HT can be left enabled on some instances and disabled on others, depending on the need. To disable HT, the following can be inserted into the `/etc/rc.local` file for the cloud node image, and made executable with a `chmod +x`: ```bash for cpunum in $(cat /sys/devices/system/cpu/cpu*/topology/thread_siblings_list | \ cut -s -d\- -f2- | tr ',' '\n' | sort -un) do echo 0 > /sys/devices/system/cpu/cpu$cpunum/online; done; ``` The delimiter used in the `cut` command is “-” instead of a “,”. If HT is disabled on the cloud node, then it can be confirmed by checking the output of the `lscpu --extended` command (output truncated): **Example** ```bash [root@cnode001 ~]# lscpu --extended CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE 0 0 0 0 0:0:0:0 yes 1 - - - ::: no ... ``` In the line just before the “...”, the “no” indicates that that logical CPU (CPU1 in this case) has been disabled. ### 3.5.4 Optimizing HPC Performance: Using Elastic Network Adapter Instances Enhanced networking that is needed for some instance types is possible if the AMI (Amazon Machine Instance) used has the Elastic Network Adapter (ENA) support attribute set. By default, all AMIs provided by NVIDIA Base Command Manager since release 8.1-9 have ENA, as provided by the `ena` kernel module. If this kernel module is absent, the administrator should update the AMI. ### 3.5.5 Optimizing HPC Performance: Using A Different Clock Source Xen is the default clocksource on AWS. Occasionally, some applications can benefit from using the TSC clocksource. Because of this, the clock source is generally changed as a best practice. The clock source can be changed with the following command: **Example** ```bash [root@cnode001 ~]# echo "tsc" > /sys/devices/system/cl*/cl*/current_clocksource ``` ### 3.5.6 Optimizing HPC Performance: Setting The Socket Buffer Sizes And TCP/IP Parameters In The Sofware Image The socket and TCP/IP window values can be modified by inserting the following values into the `sysctl.conf` file in the image: **Example** ```bash [root@basecm11 ~]# cat >> /cm/images/default-image/etc/sysctl.conf << EOF net.core.netdev_max_backlog = 1000000 net.core.rmem_default = 124928 net.core.rmem_max = 67108864 net.core.wmem_default = 124928 net.core.wmem_max = 67108864 net.ipv4.tcp_keepalive_time = 1800 net.ipv4.tcp_mem = 12184608 16246144 24369216 net.ipv4.tcp_rmem = 4194304 8388608 67108864 net.ipv4.tcp_syn_retries = 5 net.ipv4.tcp_wmem = 4194304 8388608 67108864 EOF ```