Cluster Extension Cloudbursting With AWS Using The Command Line And cmsh

The command line and cmsh can be used to set up Cluster On Demand clusters for AWS and Azure, as discussed in Chapter 2. The command line and cmsh can also be used to set up Cluster Extension clusters, as are discussed in this chapter for AWS, and in Chapter 5 for Azure.

4.1 The cm-cluster-extension Script For Cluster Extension Clusters

4.1.1 Running The cm-cluster-extension Script On The Head Node For Cluster Extension Clusters

The cm-cluster-extension script is run from the head node. It is a part of the BCM cluster-tools package. It allows cloudbursting to be carried out entirely from the command line for Cluster Extension setups. It is a command line way of carrying out the configuration carried out by the GUI steps of section 3.1 for cloud provider login and cloud director configuration. After the script has completed its setup, then cmsh power commands can launch the required cloud nodes (sections 4.1.2 and 4.2).

The cm-cluster-extension script can be run in CLI mode (here) , or as a TUI dialog (here).

4.1.1.1 Running The cm-cluster-extension Command Line Options As A Shell Dialog

The administrator can specify command line options to cm-cluster-extension, as shown in its help text. The help text is displayed with the -h|--help option:

[root@basecm11 ~]# cm-cluster-extension -h
usage: Cluster Extension cm-cluster-extension [-c <config_file>] [--remove] [--remove-provider <provider_name>]
                                              [--remove-region [<provider_name>.]<region_name>]
                                              [--terminate-instances] [--remove-fsx-instances]
                                              [--yes-i-really-mean-it] [--test-networking] [--test-environment]
                                              [--test-configuration] [--test-everything]
                                              [--enable-external-network-connectivity]
                                              [--azure-cloud-name AZURE_CLOUD_NAME] [-v] [--no-distro-checks]
                                              [--json] [--output-remote-execution-runner]
                                              [--on-error-action debug,remotedebug,undo,abort] [--skip-packages]
                                              [--min-reboot-timeout <reboot_timeout_seconds>] [--dev] [-h]
optional arguments:
  -h, --help            Print this screen
common:
  Common arguments
  -c <config_file>      Load runtime configuration for plugins from a YAML config file
removing cluster extension:
  Flags which can be used for removing CX
  --remove              Remove definitions of all objects required for cluster extension, e.g. cloud nodes,
                        directors, cloud networks and cloud interfaces
  --remove-provider <provider_name>
                        Remove specified CX provider
  --remove-region [<provider_name>.]<region_name>
                        Remove specified CX region
  --terminate-instances
                        Terminate all non-terminated VMs.
  --remove-fsx-instances
                        Remove FSX instances.
  --yes-i-really-mean-it
                        Required for additional safety
testing cluster extension:
  Flags which can be used for troubleshooting
  --test-networking     Perform networking checks (e.g. check if API endpoints are reachable)
  --test-environment    Run environment checks (e.g. if proper RPMs are installed)
  --test-configuration  Run configuration checks, which check if cluster extension is properly configured
                        (e.g. if cloud director has correct interfaces, if cloud credentials are valid, if
                        CMDaemon can create/delete objects in the cloud)
  --test-everything     Run all of the above mentioned checks.
cluster extension to Azure:
  Options specific to this cloud type.
  --azure-cloud-name AZURE_CLOUD_NAME
                        Used for deploying to a non-public Azure Cloud. E.g.: AzureUSGovernment
advanced:
  Various *advanced* configuration options flags.
  -v, --verbose         Verbose output
  --no-distro-checks    Disable distribution checks based on ds.json
  --json                Use json formatting for log lines printed to stdout
  --output-remote-execution-runner
                        Format output for CMDaemon
  --on-error-action debug,remotedebug,undo,abort
                        Upon encountering a critical error, instead of asking the user for choice, setup
                        will do selected action.
  --skip-packages       Skip the any stages which install packages. Requires packages to be already
                        installed.
  --min-reboot-timeout <reboot_timeout_seconds>
                        How long to wait for nodes to finish reboot (default and minimum allowed: 300
                        seconds).
  --dev                 Enables additional command line arguments
examples:
  cm-cluster-extension                            (start interactive menu, wizard)
  cm-cluster-extension -c <config>                (configure bursting to AWS)
  cm-cluster-extension --remove                   (remove bursting)
  cm-cluster-extension --remove --yes-i-really-mean-it
    (remove bursting, no confirmation)
     *WARNING* This will remove the cloud extension configuration. Any data located on your cloud nodes
     will be lost, unless you back it up beforehand.
  cm-cluster-extension --test-everything
This tool looks for, and uses if found, the following environment variables:
   AWS_USERNAME, AWS_ACCOUNT_ID, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
   AZURE_SUBSCRIPTION_ID, AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET

It can be run with the options directly (some output skipped):

Example

[root@basecm11 ~]# cm-cluster-extension --test-networking
Please wait...
Found an optional config file, '/root/cm-setup.conf'. Will attempt to load it.
Executing 26 stages
################### Starting execution for 'Running networking checks'
Connecting to CMDaemon
  - cloudstorage
  - clusterextension
#### stage: clusterextension: Testing tcp connection to ec2.us-east-1.amazonaws.com:443
#### stage: clusterextension: Testing tcp connection to ec2.us-west-1.amazonaws.com:443
...
#### stage: clusterextension: Testing udp OpenVPN connectivity
Took:     00:09 min.
################### Finished execution for 'Running networking checks', status: completed
Running networking checks finished!

4.1.1.2 Running The cm-cluster-extension TUI

The more user-friendly way to run cm-cluster-extension is to run it without options. This brings up the main screen for its TUI (figure 4.1).

Figure 4.1: Configuration Processing With cm-cluster-extension: Main Screen

Cluster extension cloudbursting deployment can be carried out by selecting AWS (described in this chapter) or Azure (Chapter 5) from the main screen.

After AWS is selected, a new AWS provider can be set if none is already set up. If an AWS region has been configured previously, then its configuration can be removed. Testing only the network connectivity to the various AWS regions is also possible (figure 4.2).

Figure 4.2: Configuration Processing With cm-cluster-extension: Add, Unconfigure, Or Test

If a new AWS provider is selected, then a screen comes up asking for the credentials for Amazon (figure 4.3):

Figure 4.3: Configuration Processing With cm-cluster-extension: Obtaining Credentials

To paste the credentials from the clipboard, the cluster administrator may find it helpful to know that a paste to the TUI can usually be carried out with a <shift><insert>.

After checking and accepting the credentials, the administrator is asked to choose a setup type for the wizard. This can be a default setup, or it can be an advanced setup (figure 4.4):

Figure 4.4: Configuration Processing With cm-cluster-extension: Default Setup Or Advanced Setup

The default setup is recommended for most use cases. The default setup configures an OpenVPN network connection to the cluster extension. The advanced setup allows the network connection to the cluster extension to use Direct Connect for AWS, or another hardware-based VPN.

After a default or advanced setup has been chosen, the initial number of cloud nodes to be set up in the cloud can be set (figure 4.5). A default of 3 is suggested.

Figure 4.5: Configuration Processing With cm-cluster-extension: Setting The Initial Number Of Cloud Nodes

After setting an initial number of cloud nodes, the major regions into which these can be deployed are displayed (figure 4.6):

Figure 4.6: Configuration Processing With cm-cluster-extension: Major Regions Selection

After selecting a major region, the available regions under it into which the cloud nodes can be deployed are displayed (figure 4.7):

Figure 4.7: Configuration Processing With cm-cluster-extension: Regions Selection

After selecting one or more regions, the default region into which the cloud nodes can be deployed can be set (figure 4.8):

Figure 4.8: Configuration Processing With cm-cluster-extension: Default Region Selection

After setting the default region, the availability zone must be set for the public subnets (figure 4.9):

Figure 4.9: Configuration Processing With cm-cluster-extension: Availability Zone Selection For Subnets

A similar screen is displayed for the private subnets.

After setting the availability zones for the public and private subnets, a default instance type must be set for the regular cloud nodes. The major instance type is first set. For example, the m3 type is a common major instance type that can be set (figure 4.10):

Figure 4.10: Configuration Processing With cm-cluster-extension: Major Instance Type

After selecting the major instance type, the specific default instance type can be set for the regular cloud nodes. The m3.medium type, (3.75GB RAM, 4GB SSD, 3 EC2 compute units) is a common default that can be used: (figure 4.11):

Figure 4.11: Configuration Processing With cm-cluster-extension: Default Instance Type

After setting the default instance type for the regular cloud nodes, screens (similar to figures 4.10 and 4.11) are displayed to guide the administrator into setting a default instance type screen for the cloud director node type.

After having set the default instance types for the regular and cloud director nodes. the VPN connection to the cluster extension is configured.

If the advanced setup (figure 4.4) was selected earlier in the wizard, then the administrator can select the type of VPN connection to the cluster extension (figure 4.12):

Figure 4.12: Configuration Processing With cm-cluster-extension: Custom VPN Configuration

• This can be the OpenVPN option that is automatically configured if the advanced setup is not chosen.

• With the advanced setup it is also possible to configure

  • Direct Connect (for AWS)

  • or another hardware-based VPN

As part of VPN configuration, the wizard guides the administrator through the process of configuring the cluster extension so that the extension:

• creates a new VPC for each cluster region. Alternatively, an existing VPC can be set.

• creates subnets for its private and public networks per region by default. Alternatively, existing subnets can be used instead, but the cloud director must be able to contact the on-premises head node.

• creates a security group for each region by default. Alternatively, existing security groups can be used instead.

After the VPN connection is configured, the summary screen (figure 4.13) is displayed. This lets the administrator look things over before deployment:

Figure 4.13: Configuration Processing With cm-cluster-extension: Summary Screen

The summary screen allows the following:

• An administrator can just go ahead, save the configuration, and deploy the cluster extension. This is usually the expected action.

• The configuration settings YAML file can be viewed. To scroll, the PageUp and PageDown keys are used.

• The advanced configuration settings screen can be accessed in addition to the standard settings. The advanced settings are usually left alone.

• The configuration file, <configuration file>, can be saved and the TUI can be exited. By default, the value of <configuration file> is set to cm-cluster-extension.conf in the home directory of the user. On exiting the TUI, deployment with that configuration can be carried out manually by running:

  cm-cluster-extension -c <configuration file>

After cm-cluster-extension has carried out a successful deployment, the cloud nodes (the cloud director and regular cloud nodes) can be launched.

4.1.2 Launching The Cloud Director For Cluster Extension Clusters

Launching the cluster in the cloud requires that the cloud director (section 3.2) and cloud nodes be powered up. This can be done using Base View as described in sections 3.2 and 3.3. It can also be carried out in cmsh, for example, the cloud director eu-west-1-director can be powered up from device mode with:

Example

cmsh -c "device power on -n eu-west-1-director"

If the administrator is unsure of the exact cloud director name, one way it can easily be found is via tab-completion within the device mode of cmsh. Alternatively, the cloud directors for AWS can be listed with:

Example

cmsh -c "device; list -c aws-cloud-director"

As explained in section 3.2, the cloud director takes some time to power up. Its status can be followed in the notice messages sent to the cmsh session, or in the Base View event viewer. The status can also be queried via the status command in device node. For example, a watch instruction such as:

[root@basecm11 ~]# watch 'cmsh -c "device status -n eu-west-1-director"'

will show a series of outputs similar to:

eu-west-1-director ....... [ PENDING ] (Waiting for instance to start)
eu-west-1-director ....... [ PENDING ] (Waiting for instance to start)
eu-west-1-director ....... [ PENDING ] (IP assigned: 54.220.240.166)
eu-west-1-director ....... [ PENDING ] (setting up tunnel)
eu-west-1-director ....... [ INSTALLER_REBOOTING ]
eu-west-1-director ....... [ INSTALLING  ] (recreating partitions)
eu-west-1-director ....... [ INSTALLING  ] (FULL provisioning to "/")
eu-west-1-director ....... [ INSTALLING  ] (provisioning started)
eu-west-1-director [ INSTALLER_CALLINGINIT ] (switching to local root)
eu-west-1-director ....... [   UP   ]

4.2 Launching The Cloud Nodes

Once the cloud director is up on the cloud provider, the regular cloud nodes can also be powered up. This does however first require that the corresponding cloud node objects exist. That is, that CMDaemon must have a representation of the cloud nodes, even if they do not yet exist on the cloud provider. The objects must each have an IP address assigned to them that is consistent with that of the cloud director that manages them. That is, the network address of the cloud nodes must be what the cloud director expects.

BCM’s cluster extension utilities create 3 cloud node objects by default (figure 4.7, here). Cloning them is an easy way to extend the number of deployable cloud nodes.

With Base View, this can be done with the Clone command to assign properties to the clone that match the original (section 3.3), but advance the relevant IP addresses by 1. In cmsh, the clone command works the same way.

To launch a cloud node that has an object, a command can be run as follows:

[basecm11->device[cnode001]->interfaces]% device power on -n cnode001

4.2.1 Creating And Powering Up Many Nodes

For a large number of cloud nodes, the creation and assignment of IP addresses can be done with the clone option of the foreach command, (section 2.5.5.9 of the Administrator Manual), together with a node range specification. This is the same syntax as used to create non-cloud regular nodes with cmsh.

Earlier on in this section, starting from here, a TUI session was run that ended up creating

• the cloud director eu-west-1-director and

• regular node objects eu-west-1-cnode001 up to eu-west-1-cnode003.

Continuing with the end result of that session, cloning many further regular cloud nodes can now be carried out by cloning eu-west-1-cnode003:

Example

[basecm11->device]% foreach --clone eu-west-1-cnode003 -n eu-west-1-cnode0[04-12] ()
Warning: The Ethernet switch settings were not cloned, and have to be set manually
...
[basecm11->device*]% commit
Successfully committed 9 Devices
[basecm11->device]%

As a reminder, the node range option -n eu-west-1-cnode004..eu-west-1-cnode012 would also be valid for the preceding example, and perhaps easier to comprehend, although longer.

The IP addresses are assigned to the cloud nodes via heuristics based on the value of eu-west1-cnode003 and its cloud director.

Powering up many cloud nodes can carried out using cmsh with the node range option as follows:

Example

[basecm11->device]% power on -n eu-west-1-cnode0[02-10]

4.3 Miscellaneous Cloud Tools

4.3.1 Setting Exclude Lists With excludelistsnippets

The excludelistsnippets submode allows extra exclude list entries to be created and configured for a provisioned file system.

These extra exclude list snippets are added to the regular exclude lists described in section 5.6 of the Administrator Manual. The addition of these exclude list snippets to the regular lists is done by setting a mode parameter to the snippets. The mode parameter sets the exclude list that is associated with an exclude list snippet, as indicated by the following table:

An exclude list snippet with an associated mode parameter behaves as if its entries are run along with the regular exclude list that is associated with it.

For example, to exclude the files /useless/test1 and /useless/test2, a snippet called test may be created as follows:

Example

[basecm11->fspart]% use /cm/shared
[basecm11->fspart[/cm/shared]]% excludelistsnippets
[basecm11->fspart[/cm/shared]->excludelistsnippets]%
[basecm11->fspart[/cm/shared]->excludelistsnippets]% add test
[basecm11->fspart*[/cm/shared*]->excludelistsnippets*[test*]]% set modefull yes
[basecm11->fspart*[/cm/shared*]->excludelistsnippets*[test*]]% set excludelist
[basecm11->fspart*[/cm/shared*]->excludelistsnippets*[test*]]% set excludelist /useless/test1
[basecm11->fspart*[/cm/shared*]->excludelistsnippets*[test*]]% append excludelist /useless/test2
[basecm11->fspart*[/cm/shared*]->excludelistsnippets*[test*]]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Lines                            2
Name                             test
Revision
Exclude list                     /useless/test1,/useless/test2
Disabled                         no
No new files                     no
Mode sync                        yes
Mode full                        yes
Mode update                      yes
Mode grab                        no
Mode grab new                    no

When the test snippet is used, its corresponding .rsync file has the following configuration:

[root@basecm11 cmd]# grep useless /var/spool/cmd/*shared.rsync
- /useless/test1
- /useless/test2

4.3.1.1 Setting Filesystem Synchronization For Edge Director Filesystem Partitions

The edge director role allows the sync/cm/shared property to decide if the /cm/shared filesystem partition should be synchronized.

Example

[basecm11]% device use edge-director
[basecm11->device[edge-director]]% roles
[basecm11->device[edge-director]->roles]% use edgedirector
[basecm11->device[edge-director]->roles[edgedirector]]% show
Parameter                         Value
--------------------------------- ------------------------------------------------
Name                              edgedirector
Revision
Type                              EdgeDirectorRole
Add services                      yes
Node power operations             yes
Director power operations         no
Add named service                 yes
Add slapd service                 yes
Add ntpd service                  yes
Open TCP ports on head node
Open UDP ports on head node
Externally visible IP             0.0.0.0
Externally visible head node IP   0.0.0.0
Sync /cm/shared                   yes
Disable automatic exports         no
Create home directories           never

4.4 Connecting To AWS With Direct Connect Or A Hardware VPN

For simple configurations, BCM recommends, and provides, OpenVPN by default for cluster extension cloudbursting VPN connectivity. If there is a wish to use Direct Connect or a hardware VPN (for example, an IPSec tunnel), then BCM can be configured to work with those.

The AWS cluster extension always runs in an AWS Virtual Private Cloud (VPC). In the default deployment scenario, the head node communicates with the cloud nodes using an OpenVPN connection between the head node and the cloud director. In the case of a Direct Connect connection or a hardware VPN, the head node can be configured to communicate directly with the cloud director and cloud nodes.

Setting up Direct Connect or a VPN for a cluster extension can be carried out according to these three steps:

1. VPC creation (section 4.4.1). This step can be skipped if an existing VPC is to be used.

2. Connecting the local network to the VPC (section 4.4.2). The connection can be with, for example, Direct Connect, or a hardware VPN. This step can be skipped if the local network is already connected to an existing VPC via a Direct Connect or a hardware VPN connection.

3. Configuring and deploying the cluster extension (section 4.4.3).

4.4.1 Creating a VPC

The Amazon Virtual Private Cloud (VPC) is a logically isolated section of the AWS cloud. A VPC allows complete control over the networking environment for cloud resources used by the cluster extension. The VPC enables the cloud director and cloud nodes to securely communicate with each other and can be extended to the local network of the cluster. Documentation for Amazon VPC can be found at:

https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html

A new VPC can be created and configured for the cluster extension as follows:

1. After logging in to the AWS management console, https://console.aws.amazon.com/console/, the following navigation path can be followed:

   Services > VPC > VPC Dashboard > VPCs > Create VPC.

2. An IPv4 CIDR block can be set to a desired range. This range should not conflict with the address space used on-premises. A possible range could be, for example: 10.42.0.0/16

3. Once the VPC range is set, a subnet can be set via the options under the navigation path:

   VIRTUAL PRIVATE CLOUD > Subnets > Create subnet.

4. The VPC created in step 2 is selected as the VPC during the Create subnet dialog. An IPv4 CIDR block that is a subnet of the VPC range must also be defined in the Create subnet dialog. A possible subnet range would be, for example: 10.42.0.0/24

4.4.2 Connecting The Local Network To The VPC

Amazon offers two options to connect the local network to the VPC: Direct Connect and VPN.

4.4.2.1 Connecting Via Direct Connect

To connect the local network to the VPC via Direct Connect requires a private virtual interface. Amazon’s instructions to use AWS Direct Connect to access a VPC are at:

https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html

4.4.2.2 Connecting Via A Site-To-Site Connection Using A VPN

Connecting an on-premises network to the virtual network using a VPN requires an Amazon Site-to-Site connection. Amazon’s instructions for configuring a site-to-site VPN connection are at:

https://docs.aws.amazon.com/vpn/latest/s2svpn/SetUpVPNConnections.html

The site-to-site connection consists of 4 components:

1. a customer gateway (routes traffic from VPC to local network)

2. a target gateway (routes traffic from local network to VPC)

3. a security group

4. a connection

The target gateway can either be a virtual private gateway or a transit gateway, depending on the configuration of the other AWS resources.

If the customer gateway device does not support BGP, then the site-to-site connection needs to be configured for static routing. The default Static IP Prefixes are: 10.141.0.0/16, 192.168.200.0/24.

4.4.3 Configuring And Deploying The Cluster Extension

Once IP connectivity from on-prem to the AWS virtual network is running, the final step is creating the cluster extension using the newly created site-to-site connection.

4.4.3.1 Getting Through Shorewall

First, the firewall on the head node must be adjusted to accept traffic from the cloud subnet. Using the openports mode of the firewall role in cmsh, one rule is added per port that must be reachable from the peer subnet, restricting the source address to that subnet where possible:

[root@basecm11 ~]# cmsh
[basecm11]% device use basecm11; roles; use firewall; openports
[basecm11->device[basecm11]->roles[firewall]->openports]% add ACCEPT net <port> tcp fw <subnet CIDR>
[basecm11->device[basecm11]->roles[firewall]->openports]% commit

The add command is repeated for each required port and protocol. The shorewall service is restarted automatically after commit.

4.4.3.2 Cluster Extension Advanced Settings Configuration

A cluster extension can now be created using the cm-cluster-extension utility in TUI mode as explained in section 4.1.1, here.

However, on reaching the summary screen (figure 4.13, here), the cluster administrator should not immediately select the Save config & deploy option, but should first go to the Advanced settings option, which opens up an Advanced plugin configuration screen (figure 4.14):

Figure 4.14: Cluster Extension Configuration Processing: Advanced Plugins Screen

The clusterextension option should be selected, which opens up an advanced options settings screen (figure 4.15):

Figure 4.15: Cluster Extension Configuration Processing: Advanced Plugins For ’clusterextension’ Options Screen

In this screen:

• the Create tunnel networks option should be set to False

• the VPC for region <region> should be set to the VPC created for the cluster extension. The cluster extension is what is being connected to via Direct Connect or via a site-to-site connection

• The Security groups to use should be set for the cloud nodes and the cloud director.

4.4.3.3 Custom Security Groups Configuration

For the security group it is important to realize that the default security groups, as created by the cluster extension utility, do not accept traffic from the IPSec tunnel. It is recommended to create a custom security group, and to use that for both the director and the nodes. To create this security group:

1. From the AWS management console, the following navigation path is used:

   VPC Dashboard > Security Groups > Create security group

2. The cluster extension VPC is selected

3. A new inbound rule is added, to accept all traffic from the cluster and the VPC. For example: 192.168.200.0/24 and 10.42.0.0/16.

The TUI advanced settings screens can be backed out of by selecting Back twice. The option Save config & deploy can then be selected to create the cluster extension.

4.4.3.4 Cloud Node Certificate Autosigning

By default BCM does not issue certificates for nodes on the external network. This means that for cloud nodes the certificates need to be issued manually, once for every new cloud node.

Alternatively, to automatically sign certificate requests by cloud nodes, autosign can be enabled by the administrator for external networks. Autosigning may be a security concern, as this allows anyone on the external network to request a node certificate. Autosign can be enabled on externalnet in cmsh as follows:

[root@basecm11~]# cmsh
[basecm11]% network
[basecm11->network]% set externalnet allowautosign always
[basecm11->network]% commit

The cluster extension can now be deployed as explained in section 4.1.2.