Cluster Extension Cloudbursting With AWS Using The Command Line And cmsh

The command line and cmsh can be used to set up Cluster On Demand clusters for AWS and Azure, as discussed in Chapter 2. The command line and cmsh can also be used to set up Cluster Extension clusters, as are discussed in this chapter for AWS, and in Chapter 5 for Azure.

4.1 The cm-cluster-extension Script For Cluster Extension Clusters

4.1.1 Running The cm-cluster-extension Script On The Head Node For Cluster Extension Clusters

The cm-cluster-extension script is run from the head node. It is a part of the BCM cluster-tools package. It allows cloudbursting to be carried out entirely from the command line for Cluster Extension setups. It is a command line way of carrying out the configuration carried out by the GUI steps of section 3.1 for cloud provider login and cloud director configuration. After the script has completed its setup, then cmsh power commands can launch the required cloud nodes (sections 4.1.2 and 4.2).

The cm-cluster-extension script can be run in CLI mode (here) , or as a TUI dialog (here).

4.1.1.1 Running The cm-cluster-extension Command Line Options As A Shell Dialog

The administrator can specify command line options to cm-cluster-extension, as shown in its help text. The help text is displayed with the -h|--help option:

[root@basecm10 ~]# cm-cluster-extension -h
usage: Cluster Extension cm-cluster-extension [-c <config_file>] [--remove] [--remove-provider <provider_name>]
                                              [--remove-region [<provider_name>.]<region_name>]
                                              [--terminate-instances] [--remove-fsx-instances]
                                              [--yes-i-really-mean-it] [--test-networking] [--test-environment]
                                              [--test-configuration] [--test-everything]
                                              [--enable-external-network-connectivity]
                                              [--azure-cloud-name AZURE_CLOUD_NAME] [-v] [--no-distro-checks]
                                              [--json] [--output-remote-execution-runner]
                                              [--on-error-action debug,remotedebug,undo,abort] [--skip-packages]
                                              [--min-reboot-timeout <reboot_timeout_seconds>] [--dev] [-h]
optional arguments:
  -h, --help            Print this screen
common:
  Common arguments
  -c <config_file>      Load runtime configuration for plugins from a YAML config file
removing cluster extension:
  Flags which can be used for removing CX
  --remove              Remove definitions of all objects required for cluster extension, e.g. cloud nodes,
                        directors, cloud networks and cloud interfaces
  --remove-provider <provider_name>
                        Remove specified CX provider
  --remove-region [<provider_name>.]<region_name>
                        Remove specified CX region
  --terminate-instances
                        Terminate all non-terminated VMs.
  --remove-fsx-instances
                        Remove FSX instances.
  --yes-i-really-mean-it
                        Required for additional safety
testing cluster extension:
  Flags which can be used for troubleshooting
  --test-networking     Perform networking checks (e.g. check if API endpoints are reachable)
  --test-environment    Run environment checks (e.g. if proper RPMs are installed)
  --test-configuration  Run configuration checks, which check if cluster extension is properly configured
                        (e.g. if cloud director has correct interfaces, if cloud credentials are valid, if
                        CMDaemon can create/delete objects in the cloud)
  --test-everything     Run all of the above mentioned checks.
cluster extension to Azure:
  Options specific to this cloud type.
  --azure-cloud-name AZURE_CLOUD_NAME
                        Used for deploying to a non-public Azure Cloud. E.g.: AzureUSGovernment
advanced:
  Various *advanced* configuration options flags.
  -v, --verbose         Verbose output
  --no-distro-checks    Disable distribution checks based on ds.json
  --json                Use json formatting for log lines printed to stdout
  --output-remote-execution-runner
                        Format output for CMDaemon
  --on-error-action debug,remotedebug,undo,abort
                        Upon encountering a critical error, instead of asking the user for choice, setup
                        will do selected action.
  --skip-packages       Skip the any stages which install packages. Requires packages to be already
                        installed.
  --min-reboot-timeout <reboot_timeout_seconds>
                        How long to wait for nodes to finish reboot (default and minimum allowed: 300
                        seconds).
  --dev                 Enables additional command line arguments
examples:
  cm-cluster-extension                            (start interactive menu, wizard)
  cm-cluster-extension -c <config>                (configure bursting to AWS)
  cm-cluster-extension --remove                   (remove bursting)
  cm-cluster-extension --remove --yes-i-really-mean-it
    (remove bursting, no confirmation)
     *WARNING* This will remove the cloud extension configuration. Any data located on your cloud nodes
     will be lost, unless you back it up beforehand.
  cm-cluster-extension --test-everything
This tool looks for, and uses if found, the following environment variables:
   AWS_USERNAME, AWS_ACCOUNT_ID, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
   AZURE_SUBSCRIPTION_ID, AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET

It can be run with the options directly (some output skipped):

Example

[root@basecm10 ~]# cm-cluster-extension --test-networking
Please wait...
Found an optional config file, '/root/cm-setup.conf'. Will attempt to load it.
Executing 26 stages
################### Starting execution for 'Running networking checks'
Connecting to CMDaemon
  - cloudstorage
  - clusterextension
#### stage: clusterextension: Testing tcp connection to ec2.us-east-1.amazonaws.com:443
#### stage: clusterextension: Testing tcp connection to ec2.us-west-1.amazonaws.com:443
...
#### stage: clusterextension: Testing udp OpenVPN connectivity
Took:     00:09 min.
################### Finished execution for 'Running networking checks', status: completed
Running networking checks finished!

4.1.1.2 Running The cm-cluster-extension TUI

The more user-friendly way to run cm-cluster-extension is to run it without options. This brings up the main screen for its TUI (figure 4.1).

Figure 4.1: Configuration Processing With cm-cluster-extension: Main Screen

Cluster extension cloudbursting deployment can be carried out by selecting AWS (described in this chapter) or Azure (Chapter 5) from the main screen.

After AWS is selected, a new AWS provider can be set if none is already set up. If an AWS region has been configured previously, then its configuration can be removed. Testing only the network connectivity to the various AWS regions is also possible (figure 4.2).

Figure 4.2: Configuration Processing With cm-cluster-extension: Add, Unconfigure, Or Test

If a new AWS provider is selected, then a screen comes up asking for the credentials for Amazon (figure 4.3):

Figure 4.3: Configuration Processing With cm-cluster-extension: Obtaining Credentials

To paste the credentials from the clipboard, the cluster administrator may find it helpful to know that a paste to the TUI can usually be carried out with a <shift><insert>.

After checking and accepting the credentials, the administrator is asked to choose a setup type for the wizard. This can be a default setup, or it can be an advanced setup (figure 4.4):

Figure 4.4: Configuration Processing With cm-cluster-extension: Default Setup Or Advanced Setup

The default setup is recommended for most use cases. The default setup configures an OpenVPN network connection to the cluster extension. The advanced setup allows the network connection to the cluster extension to use Direct Connect for AWS, or another hardware-based VPN.

After a default or advanced setup has been chosen, the initial number of cloud nodes to be set up in the cloud can be set (figure 4.5). A default of 3 is suggested.

Figure 4.5: Configuration Processing With cm-cluster-extension: Setting The Initial Number Of Cloud Nodes

After setting an initial number of cloud nodes, the major regions into which these can be deployed are displayed (figure 4.6):

Figure 4.6: Configuration Processing With cm-cluster-extension: Major Regions Selection

After selecting a major region, the available regions under it into which the cloud nodes can be deployed are displayed (figure 4.7):

Figure 4.7: Configuration Processing With cm-cluster-extension: Regions Selection

After selecting one or more regions, the default region into which the cloud nodes can be deployed can be set (figure 4.8):

Figure 4.8: Configuration Processing With cm-cluster-extension: Default Region Selection

After setting the default region, the availability zone must be set for the public subnets (figure 4.9):

Figure 4.9: Configuration Processing With cm-cluster-extension: Availability Zone Selection For Subnets

A similar screen is displayed for the private subnets.

After setting the availability zones for the public and private subnets, a default instance type must be set for the regular cloud nodes. The major instance type is first set. For example, the m3 type is a common major instance type that can be set (figure 4.10):

Figure 4.10: Configuration Processing With cm-cluster-extension: Major Instance Type

After selecting the major instance type, the specific default instance type can be set for the regular cloud nodes. The m3.medium type, (3.75GB RAM, 4GB SSD, 3 EC2 compute units) is a common default that can be used: (figure 4.11):

Figure 4.11: Configuration Processing With cm-cluster-extension: Default Instance Type

After setting the default instance type for the regular cloud nodes, screens (similar to figures 4.10 and 4.11) are displayed to guide the administrator into setting a default instance type screen for the cloud director node type.

After having set the default instance types for the regular and cloud director nodes. the VPN connection to the cluster extension is configured.

If the advanced setup (figure 4.4) was selected earlier in the wizard, then the administrator can select the type of VPN connection to the cluster extension (figure 4.12):

Figure 4.12: Configuration Processing With cm-cluster-extension: Custom VPN Configuration

• This can be the OpenVPN option that is automatically configured if the advanced setup is not chosen.

• With the advanced setup it is also possible to configure

  • Direct Connect (for AWS)

  • or another hardware-based VPN

As part of VPN configuration, the wizard guides the administrator through the process of configuring the cluster extension so that the extension:

• creates a new VPC for each cluster region. Alternatively, an existing VPC can be set.

• creates subnets for its private and public networks per region by default. Alternatively, existing subnets can be used instead, but the cloud director must be able to contact the on-premises head node.

• creates a security group for each region by default. Alternatively, existing security groups can be used instead.

After the VPN connection is configured, the summary screen (figure 4.13) is displayed. This lets the administrator look things over before deployment:

Figure 4.13: Configuration Processing With cm-cluster-extension: Summary Screen

The summary screen allows the following:

• An administrator can just go ahead, save the configuration, and deploy the cluster extension. This is usually the expected action.

• The configuration settings YAML file can be viewed. To scroll, the PageUp and PageDown keys are used.

• The advanced configuration settings screen can be accessed in addition to the standard settings. The advanced settings are usually left alone.

• The configuration file, <configuration file>, can be saved and the TUI can be exited. By default, the value of <configuration file> is set to cm-cluster-extension.conf in the home directory of the user. On exiting the TUI, deployment with that configuration can be carried out manually by running:

  cm-cluster-extension -c <configuration file>

After cm-cluster-extension has carried out a successful deployment, the cloud nodes (the cloud director and regular cloud nodes) can be launched.

4.1.2 Launching The Cloud Director For Cluster Extension Clusters

Launching the cluster in the cloud requires that the cloud director (section 3.2) and cloud nodes be powered up. This can be done using Base View as described in sections 3.2 and 3.3. It can also be carried out in cmsh, for example, the cloud director eu-west-1-director can be powered up from device mode with:

Example

cmsh -c "device power on -n eu-west-1-director"

If the administrator is unsure of the exact cloud director name, one way it can easily be found is via tab-completion within the device mode of cmsh. Alternatively, the cloud directors for AWS can be listed with:

Example

cmsh -c "device; list -c aws-cloud-director"

As explained in section 3.2, the cloud director takes some time to power up. Its status can be followed in the notice messages sent to the cmsh session, or in the Base View event viewer. The status can also be queried via the status command in device node. For example, a watch instruction such as:

[root@basecm10 ~]# watch 'cmsh -c "device status -n eu-west-1-director"'

will show a series of outputs similar to:

eu-west-1-director ....... [ PENDING ] (Waiting for instance to start)
eu-west-1-director ....... [ PENDING ] (Waiting for instance to start)
eu-west-1-director ....... [ PENDING ] (IP assigned: 54.220.240.166)
eu-west-1-director ....... [ PENDING ] (setting up tunnel)
eu-west-1-director ....... [ INSTALLER_REBOOTING ]
eu-west-1-director ....... [ INSTALLING  ] (recreating partitions)
eu-west-1-director ....... [ INSTALLING  ] (FULL provisioning to "/")
eu-west-1-director ....... [ INSTALLING  ] (provisioning started)
eu-west-1-director [ INSTALLER_CALLINGINIT ] (switching to local root)
eu-west-1-director ....... [   UP   ]

4.2 Launching The Cloud Nodes

Once the cloud director is up on the cloud provider, the regular cloud nodes can also be powered up. This does however first require that the corresponding cloud node objects exist. That is, that CMDaemon must have a representation of the cloud nodes, even if they do not yet exist on the cloud provider. The objects must each have an IP address assigned to them that is consistent with that of the cloud director that manages them. That is, the network address of the cloud nodes must be what the cloud director expects.

BCM’s cluster extension utilities create 3 cloud node objects by default (figure 4.7, here). Cloning them is an easy way to extend the number of deployable cloud nodes.

With Base View, this can be done with the Clone command to assign properties to the clone that match the original (section 3.3), but advance the relevant IP addresses by 1. In cmsh, the clone command works the same way.

To launch a cloud node that has an object, a command can be run as follows:

[basecm10->device[cnode001]->interfaces]% device power on -n cnode001

4.2.1 Creating And Powering Up Many Nodes

For a large number of cloud nodes, the creation and assignment of IP addresses can be done with the clone option of the foreach command, (section 2.5.5.9 of the Administrator Manual), together with a node range specification. This is the same syntax as used to create non-cloud regular nodes with cmsh.

Earlier on in this section, starting from here, a TUI session was run that ended up creating

• the cloud director eu-west-1-director and

• regular node objects eu-west-1-cnode001 up to eu-west-1-cnode003.

Continuing with the end result of that session, cloning many further regular cloud nodes can now be carried out by cloning eu-west-1-cnode003:

Example

[basecm10->device]% foreach --clone eu-west-1-cnode003 -n eu-west-1-cnode0[04-12] ()
Warning: The Ethernet switch settings were not cloned, and have to be set manually
...
[basecm10->device*]% commit
Successfully committed 9 Devices
[basecm10->device]%

As a reminder, the node range option -n eu-west-1-cnode004..eu-west-1-cnode012 would also be valid for the preceding example, and perhaps easier to comprehend, although longer.

The IP addresses are assigned to the cloud nodes via heuristics based on the value of eu-west1-cnode003 and its cloud director.

Powering up many cloud nodes can carried out using cmsh with the node range option as follows:

Example

[basecm10->device]% power on -n eu-west-1-cnode0[02-10]

4.3 Submitting Jobs With cmjob And Cloud Storage Nodes, For Cluster Extension Clusters

The cmjob command is a user command wrapper that submits job scripts to a workload manager in a Cluster Extension cluster, so that jobs are considered for running in the cloud. Its usage for an end user is covered in section 4.7 of the User Manual.

The cmjob command is available from the BCM repository as part of the cmdaemon-cmjob package. The cmjob command needs the cmjob environment module (section 2.2 of the Administrator Manual) to be loaded by the end user before use.

In addition, the cluster administrator must have created a cmjob certificate for the user that is using the cmjob utility. By default, the certificate key pair is /.cm/cmjob.pem and /.cm/cmjob.key when the certificate is created.

Example

[root@basecm10 ~]# cmsh
[basecm10]% user use henry
[basecm10->user[henry]]% set createcmjobcertificate yes; commit

The value of createcmjobcertificate can take on these 2 values:

• no: the default, with no access to cmjob

• yes: gives access to cmjob, as well as the AWS FSx and Azure ANF integrations, if those integrations are enabled

If the cmjob command is run by the user to submit a job, then the job is submitted to the workload manager, and the data-aware scheduling mechanism is initiated.

A cluster with data-aware scheduling is a cluster that ensures that it has the data needed for the cloud computing job already accessible on cloud storage nodes.

Cloud storage nodes are nodes that are set up by the cluster manager, before the job is executed in the cloud. Because data stored can be written and read from many cloud storage nodes for each job that is placed in the cloud, the data throughput in the cloud becomes more efficient than if only one storage space were used.

Cloud storage nodes are powered up automatically if cmjob has been installed and configured. They must however be powered down explicitly, and this must be done before the cloud director that it depends on is powered down.

4.3.1 Installation And Configuration of cmjob For Data-aware Scheduling To The Cloud

The configuration of data-aware scheduling means configuring the cluster so that the tools that allow data-aware scheduling to work correctly are configured. The configuration that is carried out depends on the workload manager that is to be used.

If cmjob has not yet been set up, or if it needs reconfiguration, then the following steps should be carried out:

1. The cmdaemon-cmjob package is installed. It must be installed on the head node and in the software image that is to be used for compute cloud nodes and storage cloud nodes.

   Example

   [root@basecm10 ~]# yum install cmdaemon-cmjob
   [...]
   [root@basecm10 ~]# yum --installroot /cm/images/default-image install cmdaemon-cmjob
   [...]

2. The cm-cloud-storage-setup utility is run. Example runs are provided later, starting here, but an explanatory background is given here first.

   The utility is part of the cluster-tools package, which is installed by default. The utility

   • configures cmjob properties

   • creates

     • templates for cloud storage nodes

     • storage policies for cmjob

4.3.1.1 Templates For Cloud Storage Nodes And Storage Policy

Templates for cloud storage nodes: are a cloud node definition associated with a cloud provider. Templates for cloud storage nodes, more conveniently called template nodes, provide a template that is used by the cloud storage nodes. Template nodes, being templates, are never powered on, and are therefore always in a Down state in cmsh and Base View. Actual cloud storage nodes, on the other hand, can be powered on by the cluster manager, so that they can be used to store cloud job data.

In addition, any network interfaces associated with a template node can generally be regarded as non-functioning as far as the administrator is concerned. One feature of template nodes however is that the tunnel IP address set in the template is an offset to the network address that is used to assign IP addresses to actual storage nodes. A storage policy: defines other parameters for how storage for cloud jobs is handled. Its parameters include:

• Name: the name set for the policy

• Bucket Name: the S3 bucket used for cloud jobs to transfer input and output job data

• Default job output size: specifies the default free storage space that is provisioned for the result that a job produces

• Storage node name prefix: specifies a prefix for how storage nodes are to be named. The prefix is cstorage by default. The number suffix scheme is as for regular nodes. Thus, by default, the storage nodes are cstorage001, cstorage002 and so on.

• Template for cloud nodes: the template to use as the prototype for storage nodes

Example

4.3.1.2 Configuration Of cmjob Properties With cm-cloud-storage-setup

The cm-cloud-storage-setup is a TUI utility that configures cmjob properties for a cloud deployment. When run with the -h|--help option its usage is displayed:

[root@basecm10 ~]# cm-cloud-storage-setup -h
Please wait...
usage: Cloud storage setup cm-cloud-storage-setup [-v] [-h] [-c <config_file>]
                                                  [--dev]
                                                  [--on-error-action
                                                            {debug,remotedebug,undo,abort}]
                                                  [--output-remote-execution-runner]
                                                  [--json]
                                                  [--no-distro-checks]
                                                  [--min-reboot-timeout <reboot_timeout_seconds>]
                                                  [--skip-reboot] [--remove]
optional arguments:
  --skip-reboot         Don't reboot the nodes
common:
  Common arguments
  -v                    Verbose output
  -h, --help            Print this screen
  -c <config_file>      Load runtime configuration for modules from a YAML config file
advanced:
  Various *advanced* configuration options flags.
  --dev                 Enables additional command line arguments
  --on-error-action {debug,remotedebug,undo,abort}
                        Upon encountering a critical error, instead of asking the user for
                        choice,setup will undo (revert) the deployment stages.
  --output-remote-execution-runner
                        Format output for CMDaemon
  --json                Use json formatting for log lines printed to stdout
  --no-distro-checks
  --min-reboot-timeout  <reboot_timeout_seconds>
                        How long to wait for nodes to finish reboot (default and minimum allowed:
                        300 seconds).
Remove storage:
  --remove              Cleanup storage setup

The administrator is usually expected to run cm-cloud-storage-setup without arguments. This brings up a TUI, which starts with an introductory page (figure 4.14):

Figure 4.14: Cloud Storage Configuration Processing With cm-cloud-storage: Main Screen

Continuing on brings up the network selection screen, which sets the network in which the cloud storage is to be placed (figure 4.15):

Figure 4.15: Cloud Storage Configuration Processing With cm-cloud-storage: Network Selection

After having selected the network, a category for the storage nodes is set (figure 4.16):

Figure 4.16: Cloud Storage Configuration Processing With cm-cloud-storage: Category Selection

It is recommended that a new category be created (figure 4.17):

Figure 4.17: Cloud Storage Configuration Processing With cm-cloud-storage: Category Creation

A bucket name can be set (figure 4.18):

Figure 4.18: Cloud Storage Configuration Processing With cm-cloud-storage: S3 Bucket Entry

A workload manager is set for the cloud storage (figure 4.19):

Figure 4.19: Cloud Storage Configuration Processing With cm-cloud-storage: Workload Manager Selection

It is recommended that a new queue be created for jobs that use the storage (figure 4.20):

Figure 4.20: Cloud Storage Configuration Processing With cm-cloud-storage: Queue Creation

The queues can be assigned to the cloud category (figure 4.21):

Figure 4.21: Cloud Storage Configuration Processing With cm-cloud-storage: Setting The Queue Category

The summary screen allows the configuration to be saved and to be deployed (figure 4.22):

Figure 4.22: Cloud Storage Configuration Processing With cm-cloud-storage: Summary Screen

If the configuration is to be saved, then the file path should be specified (figure 4.23):

Figure 4.23: Cloud Storage Configuration Processing With cm-cloud-storage: Save & Deploy Screen

/root/cm-cloud-storage-setup.conf is the suggested default. On exit, the saved configuration can be run with:

Example

[root@basecm10 ~]# cm-cloud-storage-setup -c cm-cloud-storage-setup.conf

4.3.1.3 Configuring cmjob Properties With cmsh

The storage node policy settings can be modified via CMDaemon using cmsh:

[root@basecm10 ~]# cmsh
[basecm10]% cmjob
[basecm10->cmjob[cmjob]]% storagenodepolicies
[basecm10->cmjob[cmjob]->storagenodepolicies]% list
Name (key)        Template for cloud nodes
----------------- ---------------------------
us-east-1-policy  us-east-1-storage-template
[basecm10->cmjob[cmjob]->storagenodepolicies]% use us-east-1-policy
[basecm10->cmjob[cmjob]->storagenodepolicies[us-east-1-policy]]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Default job output size          30.0GiB
Exported directories             home
Intermediate storage             <submode>
Max download time                2h
Max jobs per node                10
Max storage nodes                5
Max upload time                  2h
Minimum Storage volume size      30.0GiB
Name                             us-east-1-policy
Revision
Storage node idle time limit     15m
Storage node name prefix         us-east-1-cstorage
Storage volume filesystem        ext3
Template for cloud nodes         us-east-1-storage-template
Tunnel IP start address          0.0.100.0
[basecm10->cmjob[cmjob]->storagenodepolicies[us-east-1-policy]]% intermediatestorage
[basecm10->cmjob[cmjob]->storagenodepolicies[us-east-1-policy]->intermediatestorage]% show
Parameter                             Value
------------------------------------- ------------------------------------------------
Bucket name                           basecm101b86bdc32219494f9a21
Max object lifetime                   4w 2d
Region                                us-east-1
Revision
Type                                  AWSIntermediateStorage
SecGroupFSx                           sg-0f1fe4fc6d3d49e66
Default FSx Instance Capacity (GiB)   1200
Max FSx Instance Capacity (GiB)       7200
Max FSx Instance Count Per User       5
[basecm10->cmjob[cmjob]->storagenodepolicies[us-east-1-policy]->intermediatestorage]%

By default the S3 bucket storage expires after 30 days.

Cloud jobs running under cmjob are listed under the cloudjobs submode:

[root@basecm10 ~]# cmsh
[basecm10]% cmjob
[basecm10->cmjob[cmjob]]% cloudjobs
[basecm10->cmjob[cmjob]->cloudjobs]% list
Name (key)  jobStatus                                        storageNode            user
----------- ------------------------------------------------ ---------------------- ----
slurm-35    Error: Job failed. Exit code: 0. Status: FAILED  us-east-1-cstorage001  fred
slurm-36    Error: Failed to acquire storage node: Failed t+                        fred
slurm-37    Running...                                       us-east-1-cstorage001  fred

4.3.2 Integration Of cmjob With AWS FSx For Lustre

FSx for Lustre is a high-performance highly scalable distributed file system that is available as a cloud service within some AWS regions. A BCM cluster that has burst into such a region can use FSx for Lustre to create a shared storage volume that can be read from, and can be written to, by many compute nodes simultaneously.

On a cluster that has burst into such a region, cmjob can provide workload managers with an FSx for Lustre instance to store input and output data.

Sequential workload manager jobs can then continue to use the same FSx for Lustre instance. This means that the output of one job can immediately become the input of the next one without having to transmit the data to and from an S3 bucket, or to and from the head node.

Additionally, workload manager jobs that run on multiple compute nodes in parallel can obtain their shared input from the same FSx for Lustre instance.

Providing shared storage to cloud compute nodes via FSx for Lustre is an alternative to providing it via cloud storage nodes (section 4.3). The storage node approach is not sufficiently scalable when there are more than several dozen compute nodes working simultaneously with the same storage node. FSx for Lustre is significantly more scalable and far more performant than storage nodes—although at a higher cost.

BCM currently offers three different methods of integrating cmjob with FSx for Lustre:

1. On-demand FSx: An FSx file system is created on a per-job basis and is deleted automatically when the job completes.

2. User-managed FSx: A user creates an FSx instance and then submits one or more jobs that use that same FSx instance. These jobs can execute in sequence or simultaneously. When the last job has finished, the user is responsible for deleting the FSx instance.

3. Admin-managed FSx: An administrator creates an FSx instance and shares it with one or more users. These users are able to use the instance in the same way as with User-managed FSx, but they cannot delete the instance or see each other’s files.

4.3.2.1 Prerequisites

To be able to use the integration of cmjob with FSx for Lustre, the following requirements need to be met:

• The cluster must be bursting into an AWS region that supports FSx for Lustre. AWS does not offer this feature in all regions. The AWS documentation at https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/ describes the regions that support FSx for Lustre.

• AWS needs to offer Lustre installation packages for the operating system and kernel that is used to extend into AWS. While the necessary Lustre package could perhaps be obtained directly from the Lustre project, it cannot be guaranteed that these packages are compatible with the FSx for Lustre file systems that are offered by AWS. The AWS packages are built and tested to work only with specific kernel versions.

• At the time of writing (November 2023), the BCM-supported operating systems are RHEL8, Rocky8, RHEL9, Rocky9, Ubuntu 20.04, and Ubuntu 22.04. A list of AWS-supported operating systems as well as the packages for installing the Lustre client for such systems can be found at https://docs.aws.amazon.com/fsx/latest/LustreGuide/install-lustre-client.html.

4.3.2.2 Enabling cmjob integration with FSx for Lustre:

The integration of cmjob with FSx for Lustre can be accomplished in two ways, either:

• through the cm-cloud-storage-setup wizard (section 4.3.1)

  or

• manually

Enabling cmjob integration through the wizard: When running cm-cloud-storage-setup, the administrator is asked to choose if FSx support should be enabled. If it is set to be enabled, then:

• If FSx support is not available, then the administrator is informed that one of the requirements was not met (here under the heading “Prerequisites”).

• If FSx support is available, then no further action is required from the administrator.

After the deployment has finished, all users with the createcmjobcertificate property set to yes can create user-managed and on-demand FSx volumes through cmjob. These volumes can then be used for running jobs.

Enabling cmjob integration manually: The manual integration approach should be followed when cmjob has already been configured on a cluster. That is, if the cm-cloud-storage-setup tool has already been run but the cluster has been running without FSx support so far.

Two separate actions need to be executed to manually add FSx for Lustre integration:

1. Installation of the required packages to the images

   The AWS documentation should mention the packages that are required to install the Lustre Client. Often these two packages are needed: kmod-lustre-client and lustre-client. Care must be taken to match the package version to the Linux kernel version of the images. The packages must be installed to the images under /cm/images that are used by the director and by all compute nodes within that cluster extension.

   Example

   At the time of writing (November 2023), the instructions at https://docs.aws.amazon.com/fsx/latest/LustreGuide/install-lustre-client.html suggested the following steps:

   The Amazon FSx RPM public key is picked up and imported to the images with:

   [root@basecm10 ~]# cm-chroot-sw-img /cm/images/<image used by director and compute nodes>
   ...
   [root@basecm10 /]# curl https://fsx-lustre-client-repo-public-keys.s3.amazonaws.com/fsx-rpm-public
   -key.asc -o /tmp/fsx-rpm-public-key.asc
   ...
   [root@basecm10 /]# rpm --import /tmp/fsx-rpm-public-key.asc

   The repository is then added in the image with:

   [root@basecm10 /]# curl https://fsx-lustre-client-repo.s3.amazonaws.com/el/8/fsx-lustre-client.repo -o \
   /etc/yum.repos.d/aws-fsx.repo

   The Amazon FSx Lustre client YUM repository can then be updated. By default, it is configured to install the Lustre client that is compatible with the kernel version that initially shipped with the latest supported releases. The kernel version on the compute image can be found with:

   [root@basecm10 /]# uname -r

   The version found is used by the cluster administrator to determine how the repository configuration file should be edited. The change that should be made should follow the instructions at https://docs.aws.amazon.com/fsx/latest/LustreGuide/install-lustre-client.html.

   [root@basecm10 /]# vi /etc/yum.repos.d/aws-fsx.repo
               \hspace{10pt}(some numbers are changed)

   The YUM cache should then be cleaned and the client installed:

       [root@basecm10 /]# yum clean all
   ...
   [root@basecm10 /]# yum install -y kmod-lustre-client lustre-client
   ...
   [root@basecm10 /]# exit
       \hspace{10pt}(gets out of the chroot)

2. Addition of the FSx tokens to the cloudjob profiles

   Every user that should be able to create FSx for Lustre instances should have their createcmjobcertificate property set to yes. Then, one or both of the following FSx tokens should be appended to the cloujob profile:

   • ON_DEMAND_FSX_TOKEN

   • USER_MANAGED_FSX_TOKEN.

   Example

   A user, fred is already using cluster extension with AWS, but without FSx. Now the user would like to launch and manage FSx storage on demand.

   The administrator should generate the cmjob certificate for the user:

   [basecm10->user[fred]]% set createcmjobcertificate yes; commit

   The administrator should also add the desired tokens to the cloudjob profile:

   [basecm10]% profile use cloudjob
   [basecm10->profile*[cloudjob*]]% append tokens ON_DEMAND_FSX_TOKEN USER_MANAGED_FSX_TOKEN
   [basecm10->profile*[cloudjob*]]% get tokens
   SUBMIT_CLOUD_JOB_DESCRIPTION_TOKEN
   GET_CLOUD_JOB_DESCRIPTION_TOKEN
   ON_DEMAND_FSX_TOKEN
   USER_MANAGED_FSX_TOKEN
   [basecm10->profile*[cloudjob*]]% commit

   The token assignment is global, and not per-user.

When the two preceding actions have been carried out, the director and all compute nodes should be rebooted. A user should then be able to create and submit jobs with a user-managed FSx instance, or submit jobs with an OnDemand FSx instance, depending on which tokens were assigned to the cloudjob profile.

4.3.2.3 Configuration

Through cmsh, an administrator can set profile tokens (section 6.4 of the Administrator Manual) to control how users can work with FSx for Lustre. An administrator can also set quotas that control how many FSx instances a user can have simultaneously, and how large these FSx instances can be. The aim of this is to help manage costs.

Profile tokens:

The following profile tokens are available for user management in FSx for Lustre:

• ON_DEMAND_FSX_TOKEN: allows a user to use the --on-demand-fsx flag when submitting jobs with cmjob.

• USER_MANAGED_FSX_TOKEN: allows a user to create, manage and delete FSx instances.

• DELETE_ANY_FSX_INSTANCE_TOKEN: allows a user to delete FSx instances that are owned by other users. Normally reserved for root only.

• LIST_ALL_FSX_INSTANCES_TOKEN: allows a user to see FSx instances owned by other users that are not shared with that user. Normally reserved for root only.

• SHARE_FSX_INSTANCE_TOKEN: allows a user to share FSx instances with other users. Only the owner is allowed to share it. Normally reserved for root only.

Settings:

The following cmsh path:

cmsh > cmjob > storagenodepolicies > use <policy> > intermediatestorage

leads to these storage parameters:

• Max FSx Instance Capacity (GiB)

• Max FSx Instance Count Per User

which control quota limits for FSx instances.

Each quota affects both on-demand and user-managed FSx instances. So if a user has already reached the maximum number of user-managed FSx instances, then the user cannot submit a job that uses an OnDemand FSx instance.

The same navigation path also leads to the storage parameter:

• Default FSx Instance Capacity (GiB)

which controls the default capacity when a user or admin creates a user-managed or admin-managed FSx instance.

4.3.2.4 Usage

As listed earlier, the three methods that BCM provides for cmjob integration with FSx for Lustre are:

1. on-demand FSx,

2. user-managed FSx, and

3. admin-managed FSx.

How to run these methods using cmjob is covered in the next sections.

Using on-demand FSx:

When using cmjob to submit a job, a user can now choose whether to use EBS-backed storage nodes or an FSx for Lustre file system to host the input data and output data. For a user there is little difference between how two shared storage options are used, besides their performance.

The steps are:

1. an EBS volume or FSx for Lustre file system is created as shared storage,

2. the input data is copied from the head node to this shared storage

3. all compute nodes chosen by the workload manager mount this shared storage, and then run their jobs,

4. storing their output data on the same shared storage.

5. the output data is uploaded from the shared storage to the head node

6. the EBS volume or FSx for Lustre file system is deleted.

Submitting a job to run with storage nodes is the default.

If a user wants a job to use FSx for Lustre, then the user must set the --on-demand-fsx flag when submitting the job:

Example

cmjob submit --on-demand-fsx job.sh

The capacity of the on-demand FSx instance is determined by doubling the size of the input data. This calculation takes into account both the data uploaded from the head node, and labeled data downloaded from an S3 bucket. If the user thinks that the calculated capacity is not large enough to hold the output data (the output includes temporary data), then the capacity can be increased with the --expected-output-size option. The specified size is combined with the size of the input data and rounded up to the nearest valid FSx volume size. The option also works for EBS storage.

Using user-managed FSx:

cmjob with a user-managed FSx file system operates differently from an on-demand system. With user-managed FSx, the user is responsible for creating and deleting the FSx file system.

In some cases this can be more expensive than using the on-demand approach, especially if the user forgets to delete an FSx instance after no longer needing it.

However with user-managed FSx a user can run multiple concurrent jobs with the same FSx file system. It is also possible to run multiple jobs in sequence, each using the output of the previous job as input, without having the overhead of copying files over from the FSx file system to an S3 bucket, and vice versa. Depending on the size of the workload, this option may be cheaper, or more desirable, than on-demand FSx.

To create an FSx instance, the user executes cmjob with the create-fs-instance option as follows:

Example

cmjob create-fsx-instance <name> --capacity <capacity>

<name> can be a sequence of alphanumeric characters and dashes, and <capacity> is the total size of the FSx file system in gigabytes and is ideally set to a multiple of 1200, to prevent rejection by AWS. If not specified, the default value is used. The name does not have to be unique.

To delete an FSx instance, the user can run cmjob with the delete-fsx-instance option:

Example

cmjob delete-fsx-instance <name or FSx ID>

If the name is not unique, then the user can always specify the FSx ID. The ID can be found using the list-fsx-instances option. This lists all existing FSx instances, their names, unique IDs and statuses:

Example

cmjob list-fsx-instances

Once the FSx instance has reached a status of Available (which can take a few minutes), it is possible to submit a job with this instance.

Example

cmjob submit job.sh --fsx-instance <name or FSx ID>

Using admin-managed FSx: Using cmjob with an admin-managed file system is very similar to using it with a user-managed file system. The main difference is that the admin is responsible for creating and deleting the file system, and that the file system of the admin is shared with specific users. To share the file system, the share-fsx-instance option can be run by the admin, together with the --with option, as follows:

Example

cmjob share-fsx-instance <name or FSx ID> --with <user names>

Users do not need FSx-specific tokens to submit a job with an FSx instance. Using this approach for user-managed FSx instances gives the cluster administrator more control over the creation and deletion of FSx instances.

To stop sharing an FSx instance, the admin can run the --stop-sharing option:

Example

cmjob share-fsx-instance <name or FSx ID> --stop-sharing

Tracking FSx costs:

All FSx instances created via cmjob have a BCM Owner tag assigned to them. This tag allows the cluster administrator to track the FSx costs on a per-user basis using AWS’s cost management capabilities.

4.4 Miscellaneous Cloud Tools

4.4.1 Setting Exclude Lists With excludelistsnippets

The excludelistsnippets submode allows extra exclude list entries to be created and configured for a provisioned file system.

These extra exclude list snippets are added to the regular exclude lists described in section 5.6 of the Administrator Manual. The addition of these exclude list snippets to the regular lists is done by setting a mode parameter to the snippets. The mode parameter sets the exclude list that is associated with an exclude list snippet, as indicated by the following table:

An exclude list snippet with an associated mode parameter behaves as if its entries are run along with the regular exclude list that is associated with it.

For example, to exclude the files /useless/test1 and /useless/test2, a snippet called test may be created as follows:

Example

[basecm10->fspart]% use /cm/shared
[basecm10->fspart[/cm/shared]]% excludelistsnippets
[basecm10->fspart[/cm/shared]->excludelistsnippets]%
[basecm10->fspart[/cm/shared]->excludelistsnippets]% add test
[basecm10->fspart*[/cm/shared*]->excludelistsnippets*[test*]]% set modefull yes
[basecm10->fspart*[/cm/shared*]->excludelistsnippets*[test*]]% set excludelist
[basecm10->fspart*[/cm/shared*]->excludelistsnippets*[test*]]% set excludelist /useless/test1
[basecm10->fspart*[/cm/shared*]->excludelistsnippets*[test*]]% append excludelist /useless/test2
[basecm10->fspart*[/cm/shared*]->excludelistsnippets*[test*]]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Lines                            2
Name                             test
Revision
Exclude list                     /useless/test1,/useless/test2
Disabled                         no
No new files                     no
Mode sync                        yes
Mode full                        yes
Mode update                      yes
Mode grab                        no
Mode grab new                    no

When the test snippet is used, its corresponding .rsync file has the following configuration:

[root@basecm10 cmd]# grep useless /var/spool/cmd/*shared.rsync
- /useless/test1
- /useless/test2

4.4.1.1 Setting Filesystem Synchronization For Edge Director Filesystem Partitions

The edge director role allows the sync/cm/shared property to decide if the /cm/shared filesystem partition should be synchronized.

Example

[basecm10]% device use edge-director
[basecm10->device[edge-director]]% roles
[basecm10->device[edge-director]->roles]% use edgedirector
[basecm10->device[edge-director]->roles[edgedirector]]% show
Parameter                         Value
--------------------------------- ------------------------------------------------
Name                              edgedirector
Revision
Type                              EdgeDirectorRole
Add services                      yes
Node power operations             yes
Director power operations         no
Add named service                 yes
Add slapd service                 yes
Add ntpd service                  yes
Open TCP ports on head node
Open UDP ports on head node
Externally visible IP             0.0.0.0
Externally visible head node IP   0.0.0.0
Sync /cm/shared                   yes
Disable automatic exports         no
Create home directories           never

4.5 Connecting To AWS With Direct Connect Or A Hardware VPN

For simple configurations, BCM recommends, and provides, OpenVPN by default for cluster extension cloudbursting VPN connectivity. If there is a wish to use Direct Connect or a hardware VPN (for example, an IPSec tunnel), then BCM can be configured to work with those.

The AWS cluster extension always runs in an AWS Virtual Private Cloud (VPC). In the default deployment scenario, the head node communicates with the cloud nodes using an OpenVPN connection between the head node and the cloud director. In the case of a Direct Connect connection or a hardware VPN, the head node can be configured to communicate directly with the cloud director and cloud nodes.

Setting up Direct Connect or a VPN for a cluster extension can be carried out according to these three steps:

1. VPC creation (section 4.5.1). This step can be skipped if an existing VPC is to be used.

2. Connecting the local network to the VPC (section 4.5.2). The connection can be with, for example, Direct Connect, or a hardware VPN. This step can be skipped if the local network is already connected to an existing VPC via a Direct Connect or a hardware VPN connection.

3. Configuring and deploying the cluster extension (section 4.5.3).

4.5.1 Creating a VPC

The Amazon Virtual Private Cloud (VPC) is a logically isolated section of the AWS cloud. A VPC allows complete control over the networking environment for cloud resources used by the cluster extension. The VPC enables the cloud director and cloud nodes to securely communicate with each other and can be extended to the local network of the cluster. Documentation for Amazon VPC can be found at:

https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html

A new VPC can be created and configured for the cluster extension as follows:

1. After logging in to the AWS management console, https://console.aws.amazon.com/console/, the following navigation path can be followed:

   Services > VPC > VPC Dashboard > VPCs > Create VPC.

2. An IPv4 CIDR block can be set to a desired range. This range should not conflict with the address space used on-premises. A possible range could be, for example: 10.42.0.0/16

3. Once the VPC range is set, a subnet can be set via the options under the navigation path:

   VIRTUAL PRIVATE CLOUD > Subnets > Create subnet.

4. The VPC created in step 2 is selected as the VPC during the Create subnet dialog. An IPv4 CIDR block that is a subnet of the VPC range must also be defined in the Create subnet dialog. A possible subnet range would be, for example: 10.42.0.0/24

4.5.2 Connecting The Local Network To The VPC

Amazon offers two options to connect the local network to the VPC: Direct Connect and VPN.

4.5.2.1 Connecting Via Direct Connect

To connect the local network to the VPC via Direct Connect requires a private virtual interface. Amazon’s instructions to use AWS Direct Connect to access a VPC are at:

https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html

4.5.2.2 Connecting Via A Site-To-Site Connection Using A VPN

Connecting an on-premises network to the virtual network using a VPN requires an Amazon Site-to-Site connection. Amazon’s instructions for configuring a site-to-site VPN connection are at:

https://docs.aws.amazon.com/vpn/latest/s2svpn/SetUpVPNConnections.html

The site-to-site connection consists of 4 components:

1. a customer gateway (routes traffic from VPC to local network)

2. a target gateway (routes traffic from local network to VPC)

3. a security group

4. a connection

The target gateway can either be a virtual private gateway or a transit gateway, depending on the configuration of the other AWS resources.

If the customer gateway device does not support BGP, then the site-to-site connection needs to be configured for static routing. The default Static IP Prefixes are: 10.141.0.0/16, 192.168.200.0/24.

4.5.3 Configuring And Deploying The Cluster Extension

Once IP connectivity from on-prem to the AWS virtual network is running, the final step is creating the cluster extension using the newly created site-to-site connection.

4.5.3.1 Getting Through Shorewall

First, the firewall rules on the head node must be adjusted to accept traffic from the subnet. The file /etc/shorewall/rules can be edited so that the net section allows packets from the CIDR subnet. A quick and dirty way to do it is to append to the file with:

[root@basecm10 ~]# echo "ACCEPT net:<subnet CIDR> fw - -" >> /etc/shorewall/rules
[root@basecm10 ~]# shorewall reload

4.5.3.2 Cluster Extension Advanced Settings Configuration

A cluster extension can now be created using the cm-cluster-extension utility in TUI mode as explained in section 4.1.1, here.

However, on reaching the summary screen (figure 4.13, here), the cluster administrator should not immediately select the Save config & deploy option, but should first go to the Advanced settings option, which opens up an Advanced plugin configuration screen (figure 4.24):

Figure 4.24: Cluster Extension Configuration Processing: Advanced Plugins Screen

The clusterextension option should be selected, which opens up an advanced options settings screen (figure 4.25):

Figure 4.25: Cluster Extension Configuration Processing: Advanced Plugins For ’clusterextension’ Options Screen

In this screen:

• the Create tunnel networks option should be set to False

• the VPC for region <region> should be set to the VPC created for the cluster extension. The cluster extension is what is being connected to via Direct Connect or via a site-to-site connection

• The Security groups to use should be set for the cloud nodes and the cloud director.

4.5.3.3 Custom Security Groups Configuration

For the security group it is important to realize that the default security groups, as created by the cluster extension utility, do not accept traffic from the IPSec tunnel. It is recommended to create a custom security group, and to use that for both the director and the nodes. To create this security group:

1. From the AWS management console, the following navigation path is used:

   VPC Dashboard > Security Groups > Create security group

2. The cluster extension VPC is selected

3. A new inbound rule is added, to accept all traffic from the cluster and the VPC. For example: 192.168.200.0/24 and 10.42.0.0/16.

The TUI advanced settings screens can be backed out of by selecting Back twice. The option Save config & deploy can then be selected to create the cluster extension.

4.5.3.4 Cloud Node Certificate Autosigning

By default BCM does not issue certificates for nodes on the external network. This means that for cloud nodes the certificates need to be issued manually, once for every new cloud node.

Alternatively, to automatically sign certificate requests by cloud nodes, autosign can be enabled by the administrator for external networks. Autosigning may be a security concern, as this allows anyone on the external network to request a node certificate. Autosign can be enabled on externalnet in cmsh as follows:

[root@basecm10~]# cmsh
[basecm10]% network
[basecm10->network]% set externalnet allowautosign always
[basecm10->network]% commit

The cluster extension can now be deployed as explained in section 4.1.2.