> This page is for version 10. > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Cluster Extension Cloudbursting With AWS Using The Command Line And `cmsh` The command line and `cmsh` can be used to set up Cluster On Demand clusters for AWS and Azure, as discussed in Chapter [2](/cloudbursting/codcloud). The command line and `cmsh` can also be used to set up Cluster Extension clusters, as are discussed in this chapter for AWS, and in Chapter [5](/cloudbursting/cexcloudazure) for Azure. ## 4.1 The `cm-cluster-extension` Script For Cluster Extension Clusters ### 4.1.1 Running The `cm-cluster-extension` Script On The Head Node For Cluster Extension Clusters The `cm-cluster-extension` script is run from the head node. It is a part of the BCM `cluster-tools` package. It allows cloudbursting to be carried out entirely from the command line for Cluster Extension setups. It is a command line way of carrying out the configuration carried out by the GUI steps of section [3.1](/cloudbursting/cexcloud#31-cluster-extension-with-aws-the-base-view-cluster-extension-wizard) for cloud provider login and cloud director configuration. After the script has completed its setup, then `cmsh` power commands can launch the required cloud nodes (sections [4.1.2](#412-launching-the-cloud-director-for-cluster-extension-clusters) and [4.2](#42-launching-the-cloud-nodes)). The `cm-cluster-extension` script can be run in CLI mode ([here](#4111-running-the-cm-cluster-extension-command-line-options-as-a-shell-dialog)) , or as a TUI dialog ([here](#4112-running-the-cm-cluster-extension-tui)). #### 4.1.1.1 Running The `cm-cluster-extension` Command Line Options As A Shell Dialog The administrator can specify command line options to `cm-cluster-extension`, as shown in its help text. The help text is displayed with the `-h|--help` option: ```bash [root@basecm10 ~]# cm-cluster-extension -h usage: Cluster Extension cm-cluster-extension [-c ] [--remove] [--remove-provider ] [--remove-region [.]] [--terminate-instances] [--remove-fsx-instances] [--yes-i-really-mean-it] [--test-networking] [--test-environment] [--test-configuration] [--test-everything] [--enable-external-network-connectivity] [--azure-cloud-name AZURE_CLOUD_NAME] [-v] [--no-distro-checks] [--json] [--output-remote-execution-runner] [--on-error-action debug,remotedebug,undo,abort] [--skip-packages] [--min-reboot-timeout ] [--dev] [-h] optional arguments: -h, --help Print this screen common: Common arguments -c Load runtime configuration for plugins from a YAML config file removing cluster extension: Flags which can be used for removing CX --remove Remove definitions of all objects required for cluster extension, e.g. cloud nodes, directors, cloud networks and cloud interfaces --remove-provider Remove specified CX provider --remove-region [.] Remove specified CX region --terminate-instances Terminate all non-terminated VMs. --remove-fsx-instances Remove FSX instances. --yes-i-really-mean-it Required for additional safety testing cluster extension: Flags which can be used for troubleshooting --test-networking Perform networking checks (e.g. check if API endpoints are reachable) --test-environment Run environment checks (e.g. if proper RPMs are installed) --test-configuration Run configuration checks, which check if cluster extension is properly configured (e.g. if cloud director has correct interfaces, if cloud credentials are valid, if CMDaemon can create/delete objects in the cloud) --test-everything Run all of the above mentioned checks. cluster extension to Azure: Options specific to this cloud type. --azure-cloud-name AZURE_CLOUD_NAME Used for deploying to a non-public Azure Cloud. E.g.: AzureUSGovernment advanced: Various *advanced* configuration options flags. -v, --verbose Verbose output --no-distro-checks Disable distribution checks based on ds.json --json Use json formatting for log lines printed to stdout --output-remote-execution-runner Format output for CMDaemon --on-error-action debug,remotedebug,undo,abort Upon encountering a critical error, instead of asking the user for choice, setup will do selected action. --skip-packages Skip the any stages which install packages. Requires packages to be already installed. --min-reboot-timeout How long to wait for nodes to finish reboot (default and minimum allowed: 300 seconds). --dev Enables additional command line arguments examples: cm-cluster-extension (start interactive menu, wizard) cm-cluster-extension -c (configure bursting to AWS) cm-cluster-extension --remove (remove bursting) cm-cluster-extension --remove --yes-i-really-mean-it (remove bursting, no confirmation) *WARNING* This will remove the cloud extension configuration. Any data located on your cloud nodes will be lost, unless you back it up beforehand. cm-cluster-extension --test-everything This tool looks for, and uses if found, the following environment variables: AWS_USERNAME, AWS_ACCOUNT_ID, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY AZURE_SUBSCRIPTION_ID, AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET ``` It can be run with the options directly (some output skipped): **Example** ```bash [root@basecm10 ~]# cm-cluster-extension --test-networking Please wait... Found an optional config file, '/root/cm-setup.conf'. Will attempt to load it. Executing 26 stages ################### Starting execution for 'Running networking checks' Connecting to CMDaemon - cloudstorage - clusterextension #### stage: clusterextension: Testing tcp connection to ec2.us-east-1.amazonaws.com:443 #### stage: clusterextension: Testing tcp connection to ec2.us-west-1.amazonaws.com:443 ... #### stage: clusterextension: Testing udp OpenVPN connectivity Took: 00:09 min. ################### Finished execution for 'Running networking checks', status: completed Running networking checks finished! ``` #### 4.1.1.2 Running The `cm-cluster-extension` TUI The more user-friendly way to run `cm-cluster-extension` is to run it without options. This brings up the main screen for its TUI (figure [4.1](#fig-cm-cluster-extension-configuration-processing)). ![Configuration Processing With cm-cluster-extension: Main Screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/4f41cc82a47177a7793bad7f4494bc7e6afcddb05e15b0435a87f454b3db6da4/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing.png) *Figure 4.1: Configuration Processing With cm-cluster-extension: Main Screen* Cluster extension cloudbursting deployment can be carried out by selecting `AWS` (described in this chapter) or `Azure` (Chapter [5](/cloudbursting/cexcloudazure)) from the main screen. After AWS is selected, a new AWS provider can be set if none is already set up. If an AWS region has been configured previously, then its configuration can be removed. Testing only the network connectivity to the various AWS regions is also possible (figure [4.2](#fig-cm-cluster-extension-configuration-processing-add-remove-test)). ![Configuration Processing With cm-cluster-extension: Add, Unconfigure, Or Test](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/49d62f2c7c3c11aa41bdaf7490f0d7ac502ff83af7502ca5a8565ad04ff357c1/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-add-remove-test.png) *Figure 4.2: Configuration Processing With cm-cluster-extension: Add, Unconfigure, Or Test* If a new AWS provider is selected, then a screen comes up asking for the credentials for Amazon (figure [4.3](#fig-cm-cluster-extension-configuration-processing-credentials)): ![Configuration Processing With cm-cluster-extension: Obtaining Credentials](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/b756fd232f3607b3a22d415c010af5295b989ec9118182bf727596e64c260032/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-credentials.png) *Figure 4.3: Configuration Processing With cm-cluster-extension: Obtaining Credentials* To paste the credentials from the clipboard, the cluster administrator may find it helpful to know that a paste to the TUI can usually be carried out with a ``. After checking and accepting the credentials, the administrator is asked to choose a setup type for the wizard. This can be a `default` setup, or it can be an `advanced` setup (figure [4.4](#fig-cm-cluster-extension-configuration-processing-advanced-setup)): ![Configuration Processing With cm-cluster-extension: Default Setup Or Advanced Setup](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/02ab1033275ce48a99a2cf21f47fd526b1a4581d5eb35f7ea14eed95c6bb82b2/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-advanced-setup.png) *Figure 4.4: Configuration Processing With cm-cluster-extension: Default Setup Or Advanced Setup* The `default` setup is recommended for most use cases. The `default` setup configures an OpenVPN network connection to the cluster extension. The `advanced` setup allows the network connection to the cluster extension to use Direct Connect for AWS, or another hardware-based VPN. After a `default` or `advanced` setup has been chosen, the initial number of cloud nodes to be set up in the cloud can be set (figure [4.5](#fig-cm-cluster-extension-configuration-processing-number-of-cloudnodes.png)). A default of 3 is suggested. ![Configuration Processing With cm-cluster-extension: Setting The Initial Number Of Cloud Nodes](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/3b949d352af65849bc281113250ea0dafc2f623a36d2d2706fc39a5325b9e1dd/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-number-of-cloudnodes.png) *Figure 4.5: Configuration Processing With cm-cluster-extension: Setting The Initial Number Of Cloud Nodes* After setting an initial number of cloud nodes, the major regions into which these can be deployed are displayed (figure [4.6](#fig-cm-cluster-extension-configuration-processing-major-regions)): ![Configuration Processing With cm-cluster-extension: Major Regions Selection](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/fcecef99855fd2c17065485fe1335dd28f4711ef493a156455211e2d3b6763ab/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-major-regions.png) *Figure 4.6: Configuration Processing With cm-cluster-extension: Major Regions Selection* After selecting a major region, the available regions under it into which the cloud nodes can be deployed are displayed (figure [4.7](#fig-cm-cluster-extension-configuration-processing-regions)): ![Configuration Processing With cm-cluster-extension: Regions Selection](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/e5e15ad266332d93016c2f9a17752066c69be6125e34916323d3fc0decb24a58/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-regions.png) *Figure 4.7: Configuration Processing With cm-cluster-extension: Regions Selection* After selecting one or more regions, the default region into which the cloud nodes can be deployed can be set (figure [4.8](#fig-cm-cluster-extension-configuration-processing-default-region)): ![Configuration Processing With cm-cluster-extension: Default Region Selection](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/4c5207dc33e7509428723cd546fa9fc0c12560562d19f0f686f70200eb2ac6eb/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-default-region.png) *Figure 4.8: Configuration Processing With cm-cluster-extension: Default Region Selection* After setting the default region, the availability zone must be set for the public subnets (figure [4.9](#fig-cm-cluster-extension-configuration-processing-availability-zones)): ![Configuration Processing With cm-cluster-extension: Availability Zone Selection For Subnets](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/62c689840d4abddf5dd0d19ee881f881c8e1ac0434392dc03193e8c65ab2cc3c/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-availability-zones.png) *Figure 4.9: Configuration Processing With cm-cluster-extension: Availability Zone Selection For Subnets* A similar screen is displayed for the private subnets. After setting the availability zones for the public and private subnets, a default instance type must be set for the regular cloud nodes. The major instance type is first set. For example, the `m3` type is a common major instance type that can be set (figure [4.10](#fig-cm-cluster-extension-configuration-processing-major-instance-type)): ![Configuration Processing With cm-cluster-extension: Major Instance Type](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/613c4f07425fc9b8552b64a214f975010c8986c11c0a17e48dc641b0156ca0e6/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-major-instance-type.png) *Figure 4.10: Configuration Processing With cm-cluster-extension: Major Instance Type* After selecting the major instance type, the specific default instance type can be set for the regular cloud nodes. The `m3.medium` type, (3.75GB RAM, 4GB SSD, 3 EC2 compute units) is a common default that can be used: (figure [4.11](#fig-cm-cluster-extension-configuration-processing-default-instance-type)): ![Configuration Processing With cm-cluster-extension: Default Instance Type](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/1721f2465c6a974135d18f22c0c1e69a7dd81a72f08ef3ca1a5563dd2312dc6d/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-default-instance-type.png) *Figure 4.11: Configuration Processing With cm-cluster-extension: Default Instance Type* After setting the default instance type for the regular cloud nodes, screens (similar to figures [4.10](#fig-cm-cluster-extension-configuration-processing-major-instance-type) and [4.11](#fig-cm-cluster-extension-configuration-processing-default-instance-type)) are displayed to guide the administrator into setting a default instance type screen for the cloud director node type. After having set the default instance types for the regular and cloud director nodes. the VPN connection to the cluster extension is configured. If the `advanced` setup (figure [4.4](#fig-cm-cluster-extension-configuration-processing-advanced-setup)) was selected earlier in the wizard, then the administrator can select the type of VPN connection to the cluster extension (figure [4.12](#fig-cm-cluster-extension-configuration-processing-custom-vpn)): ![Configuration Processing With cm-cluster-extension: Custom VPN Configuration](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/f1d918cb8efe779ee2e5bb3040649975157d151283370745b76c84a4dffc60b3/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-custom-vpn.png) *Figure 4.12: Configuration Processing With cm-cluster-extension: Custom VPN Configuration* * This can be the OpenVPN option that is automatically configured if the `advanced` setup is not chosen. * With the `advanced` setup it is also possible to configure * Direct Connect (for AWS) * or another hardware-based VPN As part of VPN configuration, the wizard guides the administrator through the process of configuring the cluster extension so that the extension: * creates a new VPC for each cluster region. Alternatively, an existing VPC can be set. * creates subnets for its private and public networks per region by default. Alternatively, existing subnets can be used instead, but the cloud director must be able to contact the on-premises head node. * creates a security group for each region by default. Alternatively, existing security groups can be used instead. After the VPN connection is configured, the summary screen (figure [4.13](#fig-cm-cluster-extension-configuration-processing-summary-screen)) is displayed. This lets the administrator look things over before deployment: ![Configuration Processing With cm-cluster-extension: Summary Screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/558c7616b72a5b998e2c8c992e0bebc6489f2d8344190e9b16403034061f9213/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-summary-screen.png) *Figure 4.13: Configuration Processing With cm-cluster-extension: Summary Screen* The summary screen allows the following: * An administrator can just go ahead, save the configuration, and deploy the cluster extension. This is usually the expected action. * The configuration settings YAML file can be viewed. To scroll, the PageUp and PageDown keys are used. * The advanced configuration settings screen can be accessed in addition to the standard settings. The advanced settings are usually left alone. * The configuration file, \<*configuration file*>, can be saved and the TUI can be exited. By default, the value of \<*configuration file*> is set to `cm-cluster-extension.conf` in the home directory of the user. On exiting the TUI, deployment with that configuration can be carried out manually by running: ```bash cm-cluster-extension -c ``` After `cm-cluster-extension` has carried out a successful deployment, the cloud nodes (the cloud director and regular cloud nodes) can be launched. ### 4.1.2 Launching The Cloud Director For Cluster Extension Clusters Launching the cluster in the cloud requires that the cloud director (section [3.2](/cloudbursting/cexcloud#32-cluster-extension-with-aws-cloud-director-startup-from-scratch)) and cloud nodes be powered up. This can be done using Base View as described in sections [3.2](/cloudbursting/cexcloud#32-cluster-extension-with-aws-cloud-director-startup-from-scratch) and [3.3](/cloudbursting/cexcloud#33-cluster-extension-with-aws-cloud-node-startup-from-scratch). It can also be carried out in `cmsh`, for example, the cloud director `eu-west-1-director` can be powered up from device mode with: **Example** ```bash cmsh -c "device power on -n eu-west-1-director" ``` If the administrator is unsure of the exact cloud director name, one way it can easily be found is via tab-completion within the `device` mode of `cmsh`. Alternatively, the cloud directors for AWS can be listed with: **Example** ```bash cmsh -c "device; list -c aws-cloud-director" ``` As explained in section [3.2](/cloudbursting/cexcloud#32-cluster-extension-with-aws-cloud-director-startup-from-scratch), the cloud director takes some time to power up. Its status can be followed in the notice messages sent to the `cmsh` session, or in the Base View event viewer. The status can also be queried via the `status` command in device node. For example, a `watch` instruction such as: ```bash [root@basecm10 ~]# watch 'cmsh -c "device status -n eu-west-1-director"' ``` will show a series of outputs similar to: ```bash eu-west-1-director ....... [ PENDING ] (Waiting for instance to start) eu-west-1-director ....... [ PENDING ] (Waiting for instance to start) eu-west-1-director ....... [ PENDING ] (IP assigned: 54.220.240.166) eu-west-1-director ....... [ PENDING ] (setting up tunnel) eu-west-1-director ....... [ INSTALLER_REBOOTING ] eu-west-1-director ....... [ INSTALLING ] (recreating partitions) eu-west-1-director ....... [ INSTALLING ] (FULL provisioning to "/") eu-west-1-director ....... [ INSTALLING ] (provisioning started) eu-west-1-director [ INSTALLER_CALLINGINIT ] (switching to local root) eu-west-1-director ....... [ UP ] ``` ## 4.2 Launching The Cloud Nodes Once the cloud director is up on the cloud provider, the regular cloud nodes can also be powered up. This does however first require that the corresponding cloud node objects exist. That is, that CMDaemon must have a representation of the cloud nodes, even if they do not yet exist on the cloud provider. The objects must each have an IP address assigned to them that is consistent with that of the cloud director that manages them. That is, the network address of the cloud nodes must be what the cloud director expects. BCM’s cluster extension utilities create 3 cloud node objects by default (figure [4.7](#fig-cm-cluster-extension-configuration-processing-regions), [here](#fig-cm-cluster-extension-configuration-processing-regions)). Cloning them is an easy way to extend the number of deployable cloud nodes. With Base View, this can be done with the `Clone` command to assign properties to the clone that match the original (section [3.3](/cloudbursting/cexcloud#33-cluster-extension-with-aws-cloud-node-startup-from-scratch)), but advance the relevant IP addresses by 1. In `cmsh`, the `clone` command works the same way. To launch a cloud node that has an object, a command can be run as follows: ```bash [basecm10->device[cnode001]->interfaces]% device power on -n cnode001 ``` ### 4.2.1 Creating And Powering Up Many Nodes For a large number of cloud nodes, the creation and assignment of IP addresses can be done with the clone option of the `foreach` command, ([section 2.5.5.9 of the *Administrator Manual*](/admin/clustermanagement#2559-looping-over-objects-with-foreach)), together with a node range specification. This is the same syntax as used to create non-cloud regular nodes with `cmsh`. Earlier on in this section, starting from [here](#4112-running-the-cm-cluster-extension-tui), a TUI session was run that ended up creating * the cloud director `eu-west-1-director` and * regular node objects `eu-west-1-cnode001` up to `eu-west-1-cnode003`. Continuing with the end result of that session, cloning many further regular cloud nodes can now be carried out by cloning `eu-west-1-cnode003`: **Example** ```bash [basecm10->device]% foreach --clone eu-west-1-cnode003 -n eu-west-1-cnode0[04-12] () Warning: The Ethernet switch settings were not cloned, and have to be set manually ... [basecm10->device*]% commit Successfully committed 9 Devices [basecm10->device]% ``` As a reminder, the node range option `-n eu-west-1-cnode004..eu-west-1-cnode012` would also be valid for the preceding example, and perhaps easier to comprehend, although longer. The IP addresses are assigned to the cloud nodes via heuristics based on the value of `eu-west1-cnode003` and its cloud director. Powering up many cloud nodes can carried out using `cmsh` with the node range option as follows: **Example** ```bash [basecm10->device]% power on -n eu-west-1-cnode0[02-10] ``` ## 4.3 Submitting Jobs With `cmjob` And Cloud Storage Nodes, For Cluster Extension Clusters The `cmjob` command is a user command wrapper that submits job scripts to a workload manager in a Cluster Extension cluster, so that jobs are considered for running in the cloud. Its usage for an end user is covered in [section 4.7 of the *User Manual*](/user/workloadmanagement#47-running-jobs-in-cluster-extension-cloud-nodes-using-cmjob). The `cmjob` command is available from the BCM repository as part of the `cmdaemon-cmjob` package. The `cmjob` command needs the `cmjob` environment module ([section 2.2 of the *Administrator Manual*](/admin/clustermanagement#22-modules-environment)) to be loaded by the end user before use. In addition, the cluster administrator must have created a `cmjob` certificate for the user that is using the `cmjob` utility. By default, the certificate key pair is ` /.cm/cmjob.pem` and ` /.cm/cmjob.key` when the certificate is created. **Example** ```bash [root@basecm10 ~]# cmsh [basecm10]% user use henry [basecm10->user[henry]]% set createcmjobcertificate yes; commit ``` The value of `createcmjobcertificate` can take on these 2 values: * `no`: the default, with no access to `cmjob` * `yes`: gives access to `cmjob`, as well as the AWS FSx and Azure ANF integrations, if those integrations are enabled If the `cmjob` command is run by the user to submit a job, then the job is submitted to the workload manager, and the *data-aware scheduling* mechanism is initiated. A cluster with data-aware scheduling is a cluster that ensures that it has the data needed for the cloud computing job already accessible on *cloud storage nodes*. Cloud storage nodes are nodes that are set up by the cluster manager, before the job is executed in the cloud. Because data stored can be written and read from many cloud storage nodes for each job that is placed in the cloud, the data throughput in the cloud becomes more efficient than if only one storage space were used. Cloud storage nodes are powered up automatically if `cmjob` has been installed and configured. They must however be powered down explicitly, and this must be done before the cloud director that it depends on is powered down. ### 4.3.1 Installation And Configuration of `cmjob` For Data-aware Scheduling To The Cloud The configuration of data-aware scheduling means configuring the cluster so that the tools that allow data-aware scheduling to work correctly are configured. The configuration that is carried out depends on the workload manager that is to be used. If `cmjob` has not yet been set up, or if it needs reconfiguration, then the following steps should be carried out: 1. The `cmdaemon-cmjob` package is installed. It must be installed on the head node and in the software image that is to be used for compute cloud nodes and storage cloud nodes. **Example** ```bash [root@basecm10 ~]# yum install cmdaemon-cmjob [...] [root@basecm10 ~]# yum --installroot /cm/images/default-image install cmdaemon-cmjob [...] ``` 2. The `cm-cloud-storage-setup` utility is run. Example runs are provided later, starting [here](#4312-configuration-of-cmjob-properties-with-cm-cloud-storage-setup), but an explanatory background is given here first. The utility is part of the `cluster-tools` package, which is installed by default. The utility * configures `cmjob` properties * creates * *templates for cloud storage nodes* * *storage policies* for `cmjob` #### 4.3.1.1 Templates For Cloud Storage Nodes And Storage Policy **Templates for cloud storage nodes:** are a cloud node definition associated with a cloud provider. Templates for cloud storage nodes, more conveniently called template nodes, provide a template that is used by the cloud storage nodes. Template nodes, being templates, are never powered on, and are therefore always in a `Down` state in `cmsh` and Base View. Actual cloud storage nodes, on the other hand, can be powered on by the cluster manager, so that they can be used to store cloud job data. In addition, any network interfaces associated with a template node can generally be regarded as non-functioning as far as the administrator is concerned. One feature of template nodes however is that the tunnel IP address set in the template is an offset to the network address that is used to assign IP addresses to actual storage nodes. **A storage policy:** defines other parameters for how storage for cloud jobs is handled. Its parameters include: * `Name`: the name set for the policy * `Bucket Name`: the S3 bucket used for cloud jobs to transfer input and output job data * `Default job output size`: specifies the default free storage space that is provisioned for the result that a job produces * `Storage node name prefix`: specifies a prefix for how storage nodes are to be named. The prefix is `cstorage` by default. The number suffix scheme is as for regular nodes. Thus, by default, the storage nodes are `cstorage001`, `cstorage002` and so on. * `Template for cloud nodes`: the template to use as the prototype for storage nodes **Example** #### 4.3.1.2 Configuration Of `cmjob` Properties With `cm-cloud-storage-setup` The `cm-cloud-storage-setup` is a TUI utility that configures `cmjob` properties for a cloud deployment. When run with the `-h|--help` option its usage is displayed: ```bash [root@basecm10 ~]# cm-cloud-storage-setup -h Please wait... usage: Cloud storage setup cm-cloud-storage-setup [-v] [-h] [-c ] [--dev] [--on-error-action {debug,remotedebug,undo,abort}] [--output-remote-execution-runner] [--json] [--no-distro-checks] [--min-reboot-timeout ] [--skip-reboot] [--remove] optional arguments: --skip-reboot Don't reboot the nodes common: Common arguments -v Verbose output -h, --help Print this screen -c Load runtime configuration for modules from a YAML config file advanced: Various *advanced* configuration options flags. --dev Enables additional command line arguments --on-error-action {debug,remotedebug,undo,abort} Upon encountering a critical error, instead of asking the user for choice,setup will undo (revert) the deployment stages. --output-remote-execution-runner Format output for CMDaemon --json Use json formatting for log lines printed to stdout --no-distro-checks --min-reboot-timeout How long to wait for nodes to finish reboot (default and minimum allowed: 300 seconds). Remove storage: --remove Cleanup storage setup ``` The administrator is usually expected to run `cm-cloud-storage-setup` without arguments. This brings up a TUI, which starts with an introductory page (figure [4.14](#fig-cm-cloud-storage-setup)): ![Cloud Storage Configuration Processing With cm-cloud-storage: Main Screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/c208b957c739bffeeee12de5b028aef7ea76c69165d5119cf407e13542bd868a/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cloud-storage-setup.png) *Figure 4.14: Cloud Storage Configuration Processing With cm-cloud-storage: Main Screen* Continuing on brings up the network selection screen, which sets the network in which the cloud storage is to be placed (figure [4.15](#fig-cm-cloud-storage-setup-network-selection)): ![Cloud Storage Configuration Processing With cm-cloud-storage: Network Selection](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/c79821b0bba41ee20469d939540987a27876736a6a1afd22b8bd88ddf096bc33/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cloud-storage-setup-network-selection.png) *Figure 4.15: Cloud Storage Configuration Processing With cm-cloud-storage: Network Selection* After having selected the network, a category for the storage nodes is set (figure [4.16](#fig-cm-cloud-storage-setup-category-selection)): ![Cloud Storage Configuration Processing With cm-cloud-storage: Category Selection](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/ea78f66cb5b5e454b23a012380b8844f6cdbd8f0665926461ab03f2931b544bd/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cloud-storage-setup-category-selection.png) *Figure 4.16: Cloud Storage Configuration Processing With cm-cloud-storage: Category Selection* It is recommended that a new category be created (figure [4.17](#fig-cm-cloud-storage-setup-category-creation)): ![Cloud Storage Configuration Processing With cm-cloud-storage: Category Creation](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/32e372bf4a6142a1fa8006ebea152c207a54e93fd97c918c7443b22e32cb7393/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cloud-storage-setup-category-creation.png) *Figure 4.17: Cloud Storage Configuration Processing With cm-cloud-storage: Category Creation* A bucket name can be set (figure [4.18](#fig-cm-cloud-storage-setup-s3bucket)): ![Cloud Storage Configuration Processing With cm-cloud-storage: S3 Bucket Entry](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/175752d067c5d810074539f0d6dc03c7d9d5aa2873203c4a34e4dfb7272878ea/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cloud-storage-setup-s3bucket.png) *Figure 4.18: Cloud Storage Configuration Processing With cm-cloud-storage: S3 Bucket Entry* A workload manager is set for the cloud storage (figure [4.19](#fig-cm-cloud-storage-setup-wlm-select)): ![Cloud Storage Configuration Processing With cm-cloud-storage: Workload Manager Selection](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/3c5230f59bcaff000caf5d2e4ea06173e55526fc4754785b1ad9b7144831cdbb/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cloud-storage-setup-wlm-select.png) *Figure 4.19: Cloud Storage Configuration Processing With cm-cloud-storage: Workload Manager Selection* It is recommended that a new queue be created for jobs that use the storage (figure [4.20](#fig-cm-cloud-storage-setup-q-creation)): ![Cloud Storage Configuration Processing With cm-cloud-storage: Queue Creation](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/dc0eebe842e1b3c56a43b7d17c864bac91fdc90fa0a0740b4652ec38a8e7e58e/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cloud-storage-setup-q-creation.png) *Figure 4.20: Cloud Storage Configuration Processing With cm-cloud-storage: Queue Creation* The queues can be assigned to the cloud category (figure [4.21](#fig-cm-cloud-storage-setup-q-category-set)): ![Cloud Storage Configuration Processing With cm-cloud-storage: Setting The Queue Category](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/8ae7ba431defb52d068cf8c7c79ac7ca9068132f3fcd01786861a1c9b2d67177/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cloud-storage-setup-q-category-set.png) *Figure 4.21: Cloud Storage Configuration Processing With cm-cloud-storage: Setting The Queue Category* The summary screen allows the configuration to be saved and to be deployed (figure [4.22](#fig-cm-cloud-storage-setup-summary)): ![Cloud Storage Configuration Processing With cm-cloud-storage: Summary Screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/474b42b96ff66056c4cba9a2778161c32157fb73673983dd459d6fbd6942b59c/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cloud-storage-setup-summary.png) *Figure 4.22: Cloud Storage Configuration Processing With cm-cloud-storage: Summary Screen* If the configuration is to be saved, then the file path should be specified (figure [4.23](#fig-cm-cloud-storage-setup-save-deploy)): ![Cloud Storage Configuration Processing With cm-cloud-storage: Save & Deploy Screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/32984ef488085473779278b1e040f99d28c66e68a24fb9a7a1f438b10880c0d8/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cloud-storage-setup-save-deploy.png) *Figure 4.23: Cloud Storage Configuration Processing With cm-cloud-storage: Save & Deploy Screen* `/root/cm-cloud-storage-setup.conf` is the suggested default. On exit, the saved configuration can be run with: **Example** ```bash [root@basecm10 ~]# cm-cloud-storage-setup -c cm-cloud-storage-setup.conf ``` #### 4.3.1.3 Configuring `cmjob` Properties With `cmsh` The storage node policy settings can be modified via CMDaemon using `cmsh`: ```bash [root@basecm10 ~]# cmsh [basecm10]% cmjob [basecm10->cmjob[cmjob]]% storagenodepolicies [basecm10->cmjob[cmjob]->storagenodepolicies]% list Name (key) Template for cloud nodes ----------------- --------------------------- us-east-1-policy us-east-1-storage-template [basecm10->cmjob[cmjob]->storagenodepolicies]% use us-east-1-policy [basecm10->cmjob[cmjob]->storagenodepolicies[us-east-1-policy]]% show Parameter Value -------------------------------- ------------------------------------------------ Default job output size 30.0GiB Exported directories home Intermediate storage Max download time 2h Max jobs per node 10 Max storage nodes 5 Max upload time 2h Minimum Storage volume size 30.0GiB Name us-east-1-policy Revision Storage node idle time limit 15m Storage node name prefix us-east-1-cstorage Storage volume filesystem ext3 Template for cloud nodes us-east-1-storage-template Tunnel IP start address 0.0.100.0 [basecm10->cmjob[cmjob]->storagenodepolicies[us-east-1-policy]]% intermediatestorage [basecm10->cmjob[cmjob]->storagenodepolicies[us-east-1-policy]->intermediatestorage]% show Parameter Value ------------------------------------- ------------------------------------------------ Bucket name basecm101b86bdc32219494f9a21 Max object lifetime 4w 2d Region us-east-1 Revision Type AWSIntermediateStorage SecGroupFSx sg-0f1fe4fc6d3d49e66 Default FSx Instance Capacity (GiB) 1200 Max FSx Instance Capacity (GiB) 7200 Max FSx Instance Count Per User 5 [basecm10->cmjob[cmjob]->storagenodepolicies[us-east-1-policy]->intermediatestorage]% ``` By default the S3 bucket storage expires after 30 days. Cloud jobs running under `cmjob` are listed under the `cloudjobs` submode: ```bash [root@basecm10 ~]# cmsh [basecm10]% cmjob [basecm10->cmjob[cmjob]]% cloudjobs [basecm10->cmjob[cmjob]->cloudjobs]% list Name (key) jobStatus storageNode user ----------- ------------------------------------------------ ---------------------- ---- slurm-35 Error: Job failed. Exit code: 0. Status: FAILED us-east-1-cstorage001 fred slurm-36 Error: Failed to acquire storage node: Failed t+ fred slurm-37 Running... us-east-1-cstorage001 fred ``` ### 4.3.2 Integration Of `cmjob` With AWS FSx For Lustre *FSx for Lustre* is a high-performance highly scalable distributed file system that is available as a cloud service within some AWS regions. A BCM cluster that has burst into such a region can use FSx for Lustre to create a shared storage volume that can be read from, and can be written to, by many compute nodes simultaneously. On a cluster that has burst into such a region, `cmjob` can provide workload managers with an FSx for Lustre instance to store input and output data. Sequential workload manager jobs can then continue to use the same FSx for Lustre instance. This means that the output of one job can immediately become the input of the next one without having to transmit the data to and from an S3 bucket, or to and from the head node. Additionally, workload manager jobs that run on multiple compute nodes in parallel can obtain their shared input from the same FSx for Lustre instance. Providing shared storage to cloud compute nodes via FSx for Lustre is an alternative to providing it via cloud storage nodes (section [4.3](#43-submitting-jobs-with-cmjob-and-cloud-storage-nodes-for-cluster-extension-clusters)). The storage node approach is not sufficiently scalable when there are more than several dozen compute nodes working simultaneously with the same storage node. FSx for Lustre is significantly more scalable and far more performant than storage nodes—although at a higher cost. BCM currently offers three different methods of integrating `cmjob` with FSx for Lustre: 1. *On-demand FSx*: An FSx file system is created on a per-job basis and is deleted automatically when the job completes. 2. *User-managed FSx*: A user creates an FSx instance and then submits one or more jobs that use that same FSx instance. These jobs can execute in sequence or simultaneously. When the last job has finished, the user is responsible for deleting the FSx instance. 3. *Admin-managed FSx*: An administrator creates an FSx instance and shares it with one or more users. These users are able to use the instance in the same way as with User-managed FSx, but they cannot delete the instance or see each other’s files. #### 4.3.2.1 Prerequisites To be able to use the integration of `cmjob` with FSx for Lustre, the following requirements need to be met: * The cluster must be bursting into an AWS region that supports FSx for Lustre. AWS does not offer this feature in all regions. The AWS documentation at [https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services/) describes the regions that support FSx for Lustre. * AWS needs to offer Lustre installation packages for the operating system and kernel that is used to extend into AWS. While the necessary Lustre package could perhaps be obtained directly from the Lustre project, it cannot be guaranteed that these packages are compatible with the FSx for Lustre file systems that are offered by AWS. The AWS packages are built and tested to work only with specific kernel versions. * At the time of writing (November 2023), the BCM-supported operating systems are RHEL8, Rocky8, RHEL9, Rocky9, Ubuntu 20.04, and Ubuntu 22.04. [A list of AWS-supported operating systems as well as the packages for installing the Lustre client for such systems can be found at](https://docs.aws.amazon.com/fsx/latest/LustreGuide/install-lustre-client.html) [https://docs.aws.amazon.com/fsx/latest/LustreGuide/install-lustre-client.html](https://docs.aws.amazon.com/fsx/latest/LustreGuide/install-lustre-client.html). #### 4.3.2.2 Enabling `cmjob` integration with FSx for Lustre: The integration of `cmjob` with FSx for Lustre can be accomplished in two ways, either: * through the `cm-cloud-storage-setup` wizard (section [4.3.1](#431-installation-and-configuration-of-cmjob-for-data-aware-scheduling-to-the-cloud)) or * manually **Enabling `cmjob` integration through the wizard:** When running `cm-cloud-storage-setup`, the administrator is asked to choose if FSx support should be enabled. If it is set to be enabled, then: * If FSx support is not available, then the administrator is informed that one of the requirements was not met ([here](#4321-prerequisites) under the heading “**Prerequisites**”). * If FSx support is available, then no further action is required from the administrator. After the deployment has finished, all users with the `createcmjobcertificate` property set to `yes` can create user-managed and on-demand FSx volumes through `cmjob`. These volumes can then be used for running jobs. **Enabling `cmjob` integration manually:** The manual integration approach should be followed when `cmjob` has already been configured on a cluster. That is, if the `cm-cloud-storage-setup` tool has already been run but the cluster has been running without FSx support so far. Two separate actions need to be executed to manually add FSx for Lustre integration: 1. **Installation of the required packages to the images** The AWS documentation should mention the packages that are required to install the Lustre Client. Often these two packages are needed: `kmod-lustre-client` and `lustre-client`. Care must be taken to match the package version to the Linux kernel version of the images. The packages must be installed to the images under `/cm/images` that are used by the director and by all compute nodes within that cluster extension. **Example** At the time of writing (November 2023), the instructions at [https://docs.aws.amazon.com/fsx/latest/LustreGuide/install-lustre-client.html](https://docs.aws.amazon.com/fsx/latest/LustreGuide/install-lustre-client.html) suggested the following steps: The Amazon FSx RPM public key is picked up and imported to the images with: ```bash [root@basecm10 ~]# cm-chroot-sw-img /cm/images/ ... [root@basecm10 /]# curl https://fsx-lustre-client-repo-public-keys.s3.amazonaws.com/fsx-rpm-public -key.asc -o /tmp/fsx-rpm-public-key.asc ... [root@basecm10 /]# rpm --import /tmp/fsx-rpm-public-key.asc ``` The repository is then added in the image with: ```bash [root@basecm10 /]# curl https://fsx-lustre-client-repo.s3.amazonaws.com/el/8/fsx-lustre-client.repo -o \ /etc/yum.repos.d/aws-fsx.repo ``` The Amazon FSx Lustre client YUM repository can then be updated. By default, it is configured to install the Lustre client that is compatible with the kernel version that initially shipped with the latest supported releases. The kernel version on the compute image can be found with: ```bash [root@basecm10 /]# uname -r ``` The version found is used by the cluster administrator to determine how the repository configuration file should be edited. The change that should be made should follow the instructions at [https://docs.aws.amazon.com/fsx/latest/LustreGuide/install-lustre-client.html](https://docs.aws.amazon.com/fsx/latest/LustreGuide/install-lustre-client.html). ```bash [root@basecm10 /]# vi /etc/yum.repos.d/aws-fsx.repo \hspace{10pt}(some numbers are changed) ``` The YUM cache should then be cleaned and the client installed: ```bash [root@basecm10 /]# yum clean all ... [root@basecm10 /]# yum install -y kmod-lustre-client lustre-client ... [root@basecm10 /]# exit \hspace{10pt}(gets out of the chroot) ``` 2. **Addition of the FSx tokens to the `cloudjob` profiles** Every user that should be able to create FSx for Lustre instances should have their `createcmjobcertificate` property set to `yes`. Then, one or both of the following FSx tokens should be appended to the `cloujob` profile: * `ON_DEMAND_FSX_TOKEN` * `USER_MANAGED_FSX_TOKEN`. **Example** A user, `fred` is already using cluster extension with AWS, but without FSx. Now the user would like to launch and manage FSx storage on demand. The administrator should generate the `cmjob` certificate for the user: ```bash [basecm10->user[fred]]% set createcmjobcertificate yes; commit ``` The administrator should also add the desired tokens to the `cloudjob` profile: ```bash [basecm10]% profile use cloudjob [basecm10->profile*[cloudjob*]]% append tokens ON_DEMAND_FSX_TOKEN USER_MANAGED_FSX_TOKEN [basecm10->profile*[cloudjob*]]% get tokens SUBMIT_CLOUD_JOB_DESCRIPTION_TOKEN GET_CLOUD_JOB_DESCRIPTION_TOKEN ON_DEMAND_FSX_TOKEN USER_MANAGED_FSX_TOKEN [basecm10->profile*[cloudjob*]]% commit ``` The token assignment is global, and not per-user. When the two preceding actions have been carried out, the director and all compute nodes should be rebooted. A user should then be able to create and submit jobs with a user-managed FSx instance, or submit jobs with an OnDemand FSx instance, depending on which tokens were assigned to the `cloudjob` profile. #### 4.3.2.3 Configuration Through `cmsh`, an administrator can set profile tokens ([section 6.4 of the *Administrator Manual*](/admin/usermanagement#64-tokens-and-profiles)) to control how users can work with FSx for Lustre. An administrator can also set quotas that control how many FSx instances a user can have simultaneously, and how large these FSx instances can be. The aim of this is to help manage costs. **Profile tokens:** The following profile tokens are available for user management in FSx for Lustre: * `ON_DEMAND_FSX_TOKEN`: allows a user to use the `--on-demand-fsx` flag when submitting jobs with `cmjob`. * `USER_MANAGED_FSX_TOKEN`: allows a user to create, manage and delete FSx instances. * `DELETE_ANY_FSX_INSTANCE_TOKEN`: allows a user to delete FSx instances that are owned by other users. Normally reserved for root only. * `LIST_ALL_FSX_INSTANCES_TOKEN`: allows a user to see FSx instances owned by other users that are not shared with that user. Normally reserved for root only. * `SHARE_FSX_INSTANCE_TOKEN`: allows a user to share FSx instances with other users. Only the owner is allowed to share it. Normally reserved for root only. **Settings:** The following `cmsh` path: `cmsh` > `cmjob` > `storagenodepolicies` > `use` \<*policy*> > `intermediatestorage` leads to these storage parameters: * `Max FSx Instance Capacity (GiB)` * `Max FSx Instance Count Per User` which control quota limits for FSx instances. Each quota affects both on-demand and user-managed FSx instances. So if a user has already reached the maximum number of user-managed FSx instances, then the user cannot submit a job that uses an OnDemand FSx instance. The same navigation path also leads to the storage parameter: * `Default FSx Instance Capacity (GiB)` which controls the default capacity when a user or admin creates a user-managed or admin-managed FSx instance. #### 4.3.2.4 Usage As listed earlier, the three methods that BCM provides for `cmjob` integration with FSx for Lustre are: 1. on-demand FSx, 2. user-managed FSx, and 3. admin-managed FSx. How to run these methods using `cmjob` is covered in the next sections. **Using on-demand FSx:** When using `cmjob` to submit a job, a user can now choose whether to use EBS-backed storage nodes or an FSx for Lustre file system to host the input data and output data. For a user there is little difference between how two shared storage options are used, besides their performance. The steps are: 1. an EBS volume or FSx for Lustre file system is created as shared storage, 2. the input data is copied from the head node to this shared storage 3. all compute nodes chosen by the workload manager mount this shared storage, and then run their jobs, 4. storing their output data on the same shared storage. 5. the output data is uploaded from the shared storage to the head node 6. the EBS volume or FSx for Lustre file system is deleted. Submitting a job to run with storage nodes is the default. If a user wants a job to use FSx for Lustre, then the user must set the `--on-demand-fsx` flag when submitting the job: **Example** ```bash cmjob submit --on-demand-fsx job.sh ``` The capacity of the on-demand FSx instance is determined by doubling the size of the input data. This calculation takes into account both the data uploaded from the head node, and labeled data downloaded from an S3 bucket. If the user thinks that the calculated capacity is not large enough to hold the output data (the output includes temporary data), then the capacity can be increased with the `--expected-output-size` option. The specified size is combined with the size of the input data and rounded up to the nearest valid FSx volume size. The option also works for EBS storage. **Using user-managed FSx:** `cmjob` with a user-managed FSx file system operates differently from an on-demand system. With user-managed FSx, the user is responsible for creating and deleting the FSx file system. In some cases this can be more expensive than using the on-demand approach, especially if the user forgets to delete an FSx instance after no longer needing it. However with user-managed FSx a user can run multiple concurrent jobs with the same FSx file system. It is also possible to run multiple jobs in sequence, each using the output of the previous job as input, without having the overhead of copying files over from the FSx file system to an S3 bucket, and vice versa. Depending on the size of the workload, this option may be cheaper, or more desirable, than on-demand FSx. To create an FSx instance, the user executes `cmjob` with the `create-fs-instance` option as follows: **Example** ```bash cmjob create-fsx-instance --capacity ``` \<*name*> can be a sequence of alphanumeric characters and dashes, and \<*capacity*> is the total size of the FSx file system in gigabytes and is ideally set to a multiple of 1200, to prevent rejection by AWS. If not specified, the default value is used. The name does not have to be unique. To delete an FSx instance, the user can run `cmjob` with the `delete-fsx-instance` option: **Example** ```bash cmjob delete-fsx-instance ``` If the name is not unique, then the user can always specify the FSx ID. The ID can be found using the `list-fsx-instances` option. This lists all existing FSx instances, their names, unique IDs and statuses: **Example** ```bash cmjob list-fsx-instances ``` Once the FSx instance has reached a status of `Available` (which can take a few minutes), it is possible to submit a job with this instance. **Example** ```bash cmjob submit job.sh --fsx-instance ``` **Using admin-managed FSx:** Using `cmjob` with an admin-managed file system is very similar to using it with a user-managed file system. The main difference is that the admin is responsible for creating and deleting the file system, and that the file system of the admin is shared with specific users. To share the file system, the `share-fsx-instance` option can be run by the admin, together with the `--with` option, as follows: **Example** ```bash cmjob share-fsx-instance --with ``` Users do not need FSx-specific tokens to submit a job with an FSx instance. Using this approach for user-managed FSx instances gives the cluster administrator more control over the creation and deletion of FSx instances. To stop sharing an FSx instance, the admin can run the `--stop-sharing` option: **Example** ```bash cmjob share-fsx-instance --stop-sharing ``` **Tracking FSx costs:** All FSx instances created via `cmjob` have a `BCM Owner` tag assigned to them. This tag allows the cluster administrator to track the FSx costs on a per-user basis using AWS’s cost management capabilities. ## 4.4 Miscellaneous Cloud Tools ### 4.4.1 Setting Exclude Lists With `excludelistsnippets` The `excludelistsnippets` submode allows extra exclude list entries to be created and configured for a provisioned file system. These extra exclude list snippets are added to the regular exclude lists described in [section 5.6 of the *Administrator Manual*](/admin/provisioning#56-updating-running-nodes). The addition of these exclude list snippets to the regular lists is done by setting a mode parameter to the snippets. The mode parameter sets the exclude list that is associated with an exclude list snippet, as indicated by the following table:
**Mode** **Exclude list**
`sync` `excludelistsyncinstall` (active by default)
`full` `excludelistfullinstall`
`update` `excludelistupdate` (active by default)
`grab` `excludelistgrab`
`grab new` `excludelistgrabnew`
An exclude list snippet with an associated mode parameter behaves as if its entries are run along with the regular exclude list that is associated with it. For example, to exclude the files `/useless/test1` and `/useless/test2`, a snippet called `test` may be created as follows: **Example** ```bash [basecm10->fspart]% use /cm/shared [basecm10->fspart[/cm/shared]]% excludelistsnippets [basecm10->fspart[/cm/shared]->excludelistsnippets]% [basecm10->fspart[/cm/shared]->excludelistsnippets]% add test [basecm10->fspart*[/cm/shared*]->excludelistsnippets*[test*]]% set modefull yes [basecm10->fspart*[/cm/shared*]->excludelistsnippets*[test*]]% set excludelist [basecm10->fspart*[/cm/shared*]->excludelistsnippets*[test*]]% set excludelist /useless/test1 [basecm10->fspart*[/cm/shared*]->excludelistsnippets*[test*]]% append excludelist /useless/test2 [basecm10->fspart*[/cm/shared*]->excludelistsnippets*[test*]]% show Parameter Value -------------------------------- ------------------------------------------------ Lines 2 Name test Revision Exclude list /useless/test1,/useless/test2 Disabled no No new files no Mode sync yes Mode full yes Mode update yes Mode grab no Mode grab new no ``` When the `test` snippet is used, its corresponding `.rsync` file has the following configuration: ```bash [root@basecm10 cmd]# grep useless /var/spool/cmd/*shared.rsync - /useless/test1 - /useless/test2 ``` #### 4.4.1.1 Setting Filesystem Synchronization For Edge Director Filesystem Partitions The edge director role allows the `sync/cm/shared` property to decide if the `/cm/shared` filesystem partition should be synchronized. **Example** ```bash [basecm10]% device use edge-director [basecm10->device[edge-director]]% roles [basecm10->device[edge-director]->roles]% use edgedirector [basecm10->device[edge-director]->roles[edgedirector]]% show Parameter Value --------------------------------- ------------------------------------------------ Name edgedirector Revision Type EdgeDirectorRole Add services yes Node power operations yes Director power operations no Add named service yes Add slapd service yes Add ntpd service yes Open TCP ports on head node Open UDP ports on head node Externally visible IP 0.0.0.0 Externally visible head node IP 0.0.0.0 Sync /cm/shared yes Disable automatic exports no Create home directories never ``` ## 4.5 Connecting To AWS With Direct Connect Or A Hardware VPN For simple configurations, BCM recommends, and provides, OpenVPN by default for cluster extension cloudbursting VPN connectivity. If there is a wish to use Direct Connect or a hardware VPN (for example, an IPSec tunnel), then BCM can be configured to work with those. The AWS cluster extension always runs in an AWS Virtual Private Cloud (VPC). In the default deployment scenario, the head node communicates with the cloud nodes using an OpenVPN connection between the head node and the cloud director. In the case of a Direct Connect connection or a hardware VPN, the head node can be configured to communicate directly with the cloud director and cloud nodes. Setting up Direct Connect or a VPN for a cluster extension can be carried out according to these three steps: 1. VPC creation (section [4.5.1](#451-creating-a-vpc)). This step can be skipped if an existing VPC is to be used. 2. Connecting the local network to the VPC (section [4.5.2](#452-connecting-the-local-network-to-the-vpc)). The connection can be with, for example, Direct Connect, or a hardware VPN. This step can be skipped if the local network is already connected to an existing VPC via a Direct Connect or a hardware VPN connection. 3. Configuring and deploying the cluster extension (section [4.5.3](#453-configuring-and-deploying-the-cluster-extension)). ### 4.5.1 Creating a VPC The Amazon Virtual Private Cloud (VPC) is a logically isolated section of the AWS cloud. A VPC allows complete control over the networking environment for cloud resources used by the cluster extension. The VPC enables the cloud director and cloud nodes to securely communicate with each other and can be extended to the local network of the cluster. Documentation for Amazon VPC can be found at: [https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html) A new VPC can be created and configured for the cluster extension as follows: 1. After logging in to the AWS management console, [https://console.aws.amazon.com/console/](https://console.aws.amazon.com/console/), the following navigation path can be followed: `Services` > `VPC` > `VPC Dashboard` > VPCs > `Create VPC`. 2. An IPv4 CIDR block can be set to a desired range. This range should not conflict with the address space used on-premises. A possible range could be, for example: `10.42.0.0/16` 3. Once the VPC range is set, a subnet can be set via the options under the navigation path: `VIRTUAL PRIVATE CLOUD` > `Subnets` > `Create subnet`. 4. The VPC created in step 2 is selected as the VPC during the `Create subnet` dialog. An IPv4 CIDR block that is a subnet of the VPC range must also be defined in the `Create subnet` dialog. A possible subnet range would be, for example: `10.42.0.0/24` ### 4.5.2 Connecting The Local Network To The VPC Amazon offers two options to connect the local network to the VPC: Direct Connect and VPN. #### 4.5.2.1 Connecting Via Direct Connect To connect the local network to the VPC via Direct Connect requires a private virtual interface. Amazon’s instructions to use AWS Direct Connect to access a VPC are at: [https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html](https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html) #### 4.5.2.2 Connecting Via A Site-To-Site Connection Using A VPN Connecting an on-premises network to the virtual network using a VPN requires an *Amazon Site-to-Site connection*. Amazon’s instructions for configuring a site-to-site VPN connection are at: [https://docs.aws.amazon.com/vpn/latest/s2svpn/SetUpVPNConnections.html](https://docs.aws.amazon.com/vpn/latest/s2svpn/SetUpVPNConnections.html) The site-to-site connection consists of 4 components: 1. a customer gateway (routes traffic from VPC to local network) 2. a target gateway (routes traffic from local network to VPC) 3. a security group 4. a connection The target gateway can either be a virtual private gateway or a transit gateway, depending on the configuration of the other AWS resources. If the customer gateway device does not support BGP, then the site-to-site connection needs to be configured for static routing. The default Static IP Prefixes are: `10.141.0.0/16`, `192.168.200.0/24`. ### 4.5.3 Configuring And Deploying The Cluster Extension Once IP connectivity from on-prem to the AWS virtual network is running, the final step is creating the cluster extension using the newly created site-to-site connection. #### 4.5.3.1 Getting Through Shorewall First, the firewall rules on the head node must be adjusted to accept traffic from the subnet. The file `/etc/shorewall/rules` can be edited so that the `net` section allows packets from the CIDR subnet. A quick and dirty way to do it is to append to the file with: ```bash [root@basecm10 ~]# echo "ACCEPT net: fw - -" >> /etc/shorewall/rules [root@basecm10 ~]# shorewall reload ``` #### 4.5.3.2 Cluster Extension Advanced Settings Configuration A cluster extension can now be created using the `cm-cluster-extension` utility in TUI mode as explained in section [4.1.1](#411-running-the-cm-cluster-extension-script-on-the-head-node-for-cluster-extension-clusters), [here](#4112-running-the-cm-cluster-extension-tui). However, on reaching the summary screen (figure [4.13](#fig-cm-cluster-extension-configuration-processing-summary-screen), [here](#fig-cm-cluster-extension-configuration-processing-summary-screen)), the cluster administrator should not immediately select the `Save config & deploy` option, but should first go to the `Advanced settings` option, which opens up an `Advanced plugin configuration` screen (figure [4.24](#fig-cm-cluster-extension-configuration-processing-advanced-plugins)): ![Cluster Extension Configuration Processing: Advanced Plugins Screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/53dd2cb40411d94845d35ded33c3aca39a58662cf74f378f372ecdd654b571f7/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-advanced-plugins.png) *Figure 4.24: Cluster Extension Configuration Processing: Advanced Plugins Screen* The `clusterextension` option should be selected, which opens up an advanced options settings screen (figure [4.25](#fig-cm-cluster-extension-configuration-processing-advanced-plugins-clusterextension-options)): ![Cluster Extension Configuration Processing: Advanced Plugins For ’clusterextension’ Options Screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/704cb66fe8455b0ef983f1c9d2818b093f4fbb8726edd354e470ae07a6faa033/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-advanced-plugins-clusterextension-options.png) *Figure 4.25: Cluster Extension Configuration Processing: Advanced Plugins For ’clusterextension’ Options Screen* In this screen: * the `Create tunnel networks` option should be set to `False` * the `VPC for region` \<*`region`*> should be set to the VPC created for the cluster extension. The cluster extension is what is being connected to via Direct Connect or via a site-to-site connection * The `Security groups` to use should be set for the cloud nodes and the cloud director. #### 4.5.3.3 Custom Security Groups Configuration For the security group it is important to realize that the default security groups, as created by the cluster extension utility, do not accept traffic from the IPSec tunnel. It is recommended to create a custom security group, and to use that for both the director and the nodes. To create this security group: 1. From the AWS management console, the following navigation path is used: `VPC Dashboard` > `Security Groups` > `Create security group` 2. The cluster extension VPC is selected 3. A new inbound rule is added, to accept all traffic from the cluster and the VPC. For example: `192.168.200.0/24` and `10.42.0.0/16`. The TUI advanced settings screens can be backed out of by selecting `Back` twice. The option `Save config & deploy` can then be selected to create the cluster extension. #### 4.5.3.4 Cloud Node Certificate Autosigning By default BCM does not issue certificates for nodes on the external network. This means that for cloud nodes the certificates need to be issued manually, once for every new cloud node. Alternatively, to automatically sign certificate requests by cloud nodes, autosign can be enabled by the administrator for external networks. Autosigning may be a security concern, as this allows anyone on the external network to request a node certificate. Autosign can be enabled on externalnet in `cmsh` as follows: ```bash [root@basecm10~]# cmsh [basecm10]% network [basecm10->network]% set externalnet allowautosign always [basecm10->network]% commit ``` The cluster extension can now be deployed as explained in section [4.1.2](#412-launching-the-cloud-director-for-cluster-extension-clusters).