> This page is for version 11 (default). > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Cluster Extension Cloudbursting With AWS Using The Command Line And `cmsh` The command line and `cmsh` can be used to set up Cluster On Demand clusters for AWS and Azure, as discussed in Chapter [2](/cloudbursting/codcloud). The command line and `cmsh` can also be used to set up Cluster Extension clusters, as are discussed in this chapter for AWS, and in Chapter [5](/cloudbursting/cexcloudazure) for Azure. ## 4.1 The `cm-cluster-extension` Script For Cluster Extension Clusters ### 4.1.1 Running The `cm-cluster-extension` Script On The Head Node For Cluster Extension Clusters The `cm-cluster-extension` script is run from the head node. It is a part of the BCM `cluster-tools` package. It allows cloudbursting to be carried out entirely from the command line for Cluster Extension setups. It is a command line way of carrying out the configuration carried out by the GUI steps of section [3.1](/cloudbursting/cexcloud#31-cluster-extension-with-aws-the-base-view-cluster-extension-wizard) for cloud provider login and cloud director configuration. After the script has completed its setup, then `cmsh` power commands can launch the required cloud nodes (sections [4.1.2](#412-launching-the-cloud-director-for-cluster-extension-clusters) and [4.2](#42-launching-the-cloud-nodes)). The `cm-cluster-extension` script can be run in CLI mode ([here](#4111-running-the-cm-cluster-extension-command-line-options-as-a-shell-dialog)) , or as a TUI dialog ([here](#4112-running-the-cm-cluster-extension-tui)). #### 4.1.1.1 Running The `cm-cluster-extension` Command Line Options As A Shell Dialog The administrator can specify command line options to `cm-cluster-extension`, as shown in its help text. The help text is displayed with the `-h|--help` option: ```bash [root@basecm11 ~]# cm-cluster-extension -h usage: Cluster Extension cm-cluster-extension [-c ] [--remove] [--remove-provider ] [--remove-region [.]] [--terminate-instances] [--remove-fsx-instances] [--yes-i-really-mean-it] [--test-networking] [--test-environment] [--test-configuration] [--test-everything] [--enable-external-network-connectivity] [--azure-cloud-name AZURE_CLOUD_NAME] [-v] [--no-distro-checks] [--json] [--output-remote-execution-runner] [--on-error-action debug,remotedebug,undo,abort] [--skip-packages] [--min-reboot-timeout ] [--dev] [-h] optional arguments: -h, --help Print this screen common: Common arguments -c Load runtime configuration for plugins from a YAML config file removing cluster extension: Flags which can be used for removing CX --remove Remove definitions of all objects required for cluster extension, e.g. cloud nodes, directors, cloud networks and cloud interfaces --remove-provider Remove specified CX provider --remove-region [.] Remove specified CX region --terminate-instances Terminate all non-terminated VMs. --remove-fsx-instances Remove FSX instances. --yes-i-really-mean-it Required for additional safety testing cluster extension: Flags which can be used for troubleshooting --test-networking Perform networking checks (e.g. check if API endpoints are reachable) --test-environment Run environment checks (e.g. if proper RPMs are installed) --test-configuration Run configuration checks, which check if cluster extension is properly configured (e.g. if cloud director has correct interfaces, if cloud credentials are valid, if CMDaemon can create/delete objects in the cloud) --test-everything Run all of the above mentioned checks. cluster extension to Azure: Options specific to this cloud type. --azure-cloud-name AZURE_CLOUD_NAME Used for deploying to a non-public Azure Cloud. E.g.: AzureUSGovernment advanced: Various *advanced* configuration options flags. -v, --verbose Verbose output --no-distro-checks Disable distribution checks based on ds.json --json Use json formatting for log lines printed to stdout --output-remote-execution-runner Format output for CMDaemon --on-error-action debug,remotedebug,undo,abort Upon encountering a critical error, instead of asking the user for choice, setup will do selected action. --skip-packages Skip the any stages which install packages. Requires packages to be already installed. --min-reboot-timeout How long to wait for nodes to finish reboot (default and minimum allowed: 300 seconds). --dev Enables additional command line arguments examples: cm-cluster-extension (start interactive menu, wizard) cm-cluster-extension -c (configure bursting to AWS) cm-cluster-extension --remove (remove bursting) cm-cluster-extension --remove --yes-i-really-mean-it (remove bursting, no confirmation) *WARNING* This will remove the cloud extension configuration. Any data located on your cloud nodes will be lost, unless you back it up beforehand. cm-cluster-extension --test-everything This tool looks for, and uses if found, the following environment variables: AWS_USERNAME, AWS_ACCOUNT_ID, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY AZURE_SUBSCRIPTION_ID, AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET ``` It can be run with the options directly (some output skipped): **Example** ```bash [root@basecm11 ~]# cm-cluster-extension --test-networking Please wait... Found an optional config file, '/root/cm-setup.conf'. Will attempt to load it. Executing 26 stages ################### Starting execution for 'Running networking checks' Connecting to CMDaemon - cloudstorage - clusterextension #### stage: clusterextension: Testing tcp connection to ec2.us-east-1.amazonaws.com:443 #### stage: clusterextension: Testing tcp connection to ec2.us-west-1.amazonaws.com:443 ... #### stage: clusterextension: Testing udp OpenVPN connectivity Took: 00:09 min. ################### Finished execution for 'Running networking checks', status: completed Running networking checks finished! ``` #### 4.1.1.2 Running The `cm-cluster-extension` TUI The more user-friendly way to run `cm-cluster-extension` is to run it without options. This brings up the main screen for its TUI (figure [4.1](#fig-cm-cluster-extension-configuration-processing)). ![Configuration Processing With cm-cluster-extension: Main Screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/4f41cc82a47177a7793bad7f4494bc7e6afcddb05e15b0435a87f454b3db6da4/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/cm-cluster-extension-configuration-processing.png) *Figure 4.1: Configuration Processing With cm-cluster-extension: Main Screen* Cluster extension cloudbursting deployment can be carried out by selecting `AWS` (described in this chapter) or `Azure` (Chapter [5](/cloudbursting/cexcloudazure)) from the main screen. After AWS is selected, a new AWS provider can be set if none is already set up. If an AWS region has been configured previously, then its configuration can be removed. Testing only the network connectivity to the various AWS regions is also possible (figure [4.2](#fig-cm-cluster-extension-configuration-processing-add-remove-test)). ![Configuration Processing With cm-cluster-extension: Add, Unconfigure, Or Test](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/49d62f2c7c3c11aa41bdaf7490f0d7ac502ff83af7502ca5a8565ad04ff357c1/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/cm-cluster-extension-configuration-processing-add-remove-test.png) *Figure 4.2: Configuration Processing With cm-cluster-extension: Add, Unconfigure, Or Test* If a new AWS provider is selected, then a screen comes up asking for the credentials for Amazon (figure [4.3](#fig-cm-cluster-extension-configuration-processing-credentials)): ![Configuration Processing With cm-cluster-extension: Obtaining Credentials](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/b756fd232f3607b3a22d415c010af5295b989ec9118182bf727596e64c260032/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/cm-cluster-extension-configuration-processing-credentials.png) *Figure 4.3: Configuration Processing With cm-cluster-extension: Obtaining Credentials* To paste the credentials from the clipboard, the cluster administrator may find it helpful to know that a paste to the TUI can usually be carried out with a ``. After checking and accepting the credentials, the administrator is asked to choose a setup type for the wizard. This can be a `default` setup, or it can be an `advanced` setup (figure [4.4](#fig-cm-cluster-extension-configuration-processing-advanced-setup)): ![Configuration Processing With cm-cluster-extension: Default Setup Or Advanced Setup](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/02ab1033275ce48a99a2cf21f47fd526b1a4581d5eb35f7ea14eed95c6bb82b2/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/cm-cluster-extension-configuration-processing-advanced-setup.png) *Figure 4.4: Configuration Processing With cm-cluster-extension: Default Setup Or Advanced Setup* The `default` setup is recommended for most use cases. The `default` setup configures an OpenVPN network connection to the cluster extension. The `advanced` setup allows the network connection to the cluster extension to use Direct Connect for AWS, or another hardware-based VPN. After a `default` or `advanced` setup has been chosen, the initial number of cloud nodes to be set up in the cloud can be set (figure [4.5](#fig-cm-cluster-extension-configuration-processing-number-of-cloudnodes.png)). A default of 3 is suggested. ![Configuration Processing With cm-cluster-extension: Setting The Initial Number Of Cloud Nodes](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/3b949d352af65849bc281113250ea0dafc2f623a36d2d2706fc39a5325b9e1dd/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/cm-cluster-extension-configuration-processing-number-of-cloudnodes.png) *Figure 4.5: Configuration Processing With cm-cluster-extension: Setting The Initial Number Of Cloud Nodes* After setting an initial number of cloud nodes, the major regions into which these can be deployed are displayed (figure [4.6](#fig-cm-cluster-extension-configuration-processing-major-regions)): ![Configuration Processing With cm-cluster-extension: Major Regions Selection](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/fcecef99855fd2c17065485fe1335dd28f4711ef493a156455211e2d3b6763ab/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/cm-cluster-extension-configuration-processing-major-regions.png) *Figure 4.6: Configuration Processing With cm-cluster-extension: Major Regions Selection* After selecting a major region, the available regions under it into which the cloud nodes can be deployed are displayed (figure [4.7](#fig-cm-cluster-extension-configuration-processing-regions)): ![Configuration Processing With cm-cluster-extension: Regions Selection](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/e5e15ad266332d93016c2f9a17752066c69be6125e34916323d3fc0decb24a58/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/cm-cluster-extension-configuration-processing-regions.png) *Figure 4.7: Configuration Processing With cm-cluster-extension: Regions Selection* After selecting one or more regions, the default region into which the cloud nodes can be deployed can be set (figure [4.8](#fig-cm-cluster-extension-configuration-processing-default-region)): ![Configuration Processing With cm-cluster-extension: Default Region Selection](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/4c5207dc33e7509428723cd546fa9fc0c12560562d19f0f686f70200eb2ac6eb/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/cm-cluster-extension-configuration-processing-default-region.png) *Figure 4.8: Configuration Processing With cm-cluster-extension: Default Region Selection* After setting the default region, the availability zone must be set for the public subnets (figure [4.9](#fig-cm-cluster-extension-configuration-processing-availability-zones)): ![Configuration Processing With cm-cluster-extension: Availability Zone Selection For Subnets](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/62c689840d4abddf5dd0d19ee881f881c8e1ac0434392dc03193e8c65ab2cc3c/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/cm-cluster-extension-configuration-processing-availability-zones.png) *Figure 4.9: Configuration Processing With cm-cluster-extension: Availability Zone Selection For Subnets* A similar screen is displayed for the private subnets. After setting the availability zones for the public and private subnets, a default instance type must be set for the regular cloud nodes. The major instance type is first set. For example, the `m3` type is a common major instance type that can be set (figure [4.10](#fig-cm-cluster-extension-configuration-processing-major-instance-type)): ![Configuration Processing With cm-cluster-extension: Major Instance Type](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/613c4f07425fc9b8552b64a214f975010c8986c11c0a17e48dc641b0156ca0e6/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/cm-cluster-extension-configuration-processing-major-instance-type.png) *Figure 4.10: Configuration Processing With cm-cluster-extension: Major Instance Type* After selecting the major instance type, the specific default instance type can be set for the regular cloud nodes. The `m3.medium` type, (3.75GB RAM, 4GB SSD, 3 EC2 compute units) is a common default that can be used: (figure [4.11](#fig-cm-cluster-extension-configuration-processing-default-instance-type)): ![Configuration Processing With cm-cluster-extension: Default Instance Type](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/1721f2465c6a974135d18f22c0c1e69a7dd81a72f08ef3ca1a5563dd2312dc6d/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/cm-cluster-extension-configuration-processing-default-instance-type.png) *Figure 4.11: Configuration Processing With cm-cluster-extension: Default Instance Type* After setting the default instance type for the regular cloud nodes, screens (similar to figures [4.10](#fig-cm-cluster-extension-configuration-processing-major-instance-type) and [4.11](#fig-cm-cluster-extension-configuration-processing-default-instance-type)) are displayed to guide the administrator into setting a default instance type screen for the cloud director node type. After having set the default instance types for the regular and cloud director nodes. the VPN connection to the cluster extension is configured. If the `advanced` setup (figure [4.4](#fig-cm-cluster-extension-configuration-processing-advanced-setup)) was selected earlier in the wizard, then the administrator can select the type of VPN connection to the cluster extension (figure [4.12](#fig-cm-cluster-extension-configuration-processing-custom-vpn)): ![Configuration Processing With cm-cluster-extension: Custom VPN Configuration](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/f1d918cb8efe779ee2e5bb3040649975157d151283370745b76c84a4dffc60b3/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/cm-cluster-extension-configuration-processing-custom-vpn.png) *Figure 4.12: Configuration Processing With cm-cluster-extension: Custom VPN Configuration* * This can be the OpenVPN option that is automatically configured if the `advanced` setup is not chosen. * With the `advanced` setup it is also possible to configure * Direct Connect (for AWS) * or another hardware-based VPN As part of VPN configuration, the wizard guides the administrator through the process of configuring the cluster extension so that the extension: * creates a new VPC for each cluster region. Alternatively, an existing VPC can be set. * creates subnets for its private and public networks per region by default. Alternatively, existing subnets can be used instead, but the cloud director must be able to contact the on-premises head node. * creates a security group for each region by default. Alternatively, existing security groups can be used instead. After the VPN connection is configured, the summary screen (figure [4.13](#fig-cm-cluster-extension-configuration-processing-summary-screen)) is displayed. This lets the administrator look things over before deployment: ![Configuration Processing With cm-cluster-extension: Summary Screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/558c7616b72a5b998e2c8c992e0bebc6489f2d8344190e9b16403034061f9213/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/cm-cluster-extension-configuration-processing-summary-screen.png) *Figure 4.13: Configuration Processing With cm-cluster-extension: Summary Screen* The summary screen allows the following: * An administrator can just go ahead, save the configuration, and deploy the cluster extension. This is usually the expected action. * The configuration settings YAML file can be viewed. To scroll, the PageUp and PageDown keys are used. * The advanced configuration settings screen can be accessed in addition to the standard settings. The advanced settings are usually left alone. * The configuration file, \<*configuration file*>, can be saved and the TUI can be exited. By default, the value of \<*configuration file*> is set to `cm-cluster-extension.conf` in the home directory of the user. On exiting the TUI, deployment with that configuration can be carried out manually by running: ```bash cm-cluster-extension -c ``` After `cm-cluster-extension` has carried out a successful deployment, the cloud nodes (the cloud director and regular cloud nodes) can be launched. ### 4.1.2 Launching The Cloud Director For Cluster Extension Clusters Launching the cluster in the cloud requires that the cloud director (section [3.2](/cloudbursting/cexcloud#32-cluster-extension-with-aws-cloud-director-startup-from-scratch)) and cloud nodes be powered up. This can be done using Base View as described in sections [3.2](/cloudbursting/cexcloud#32-cluster-extension-with-aws-cloud-director-startup-from-scratch) and [3.3](/cloudbursting/cexcloud#33-cluster-extension-with-aws-cloud-node-startup-from-scratch). It can also be carried out in `cmsh`, for example, the cloud director `eu-west-1-director` can be powered up from device mode with: **Example** ```bash cmsh -c "device power on -n eu-west-1-director" ``` If the administrator is unsure of the exact cloud director name, one way it can easily be found is via tab-completion within the `device` mode of `cmsh`. Alternatively, the cloud directors for AWS can be listed with: **Example** ```bash cmsh -c "device; list -c aws-cloud-director" ``` As explained in section [3.2](/cloudbursting/cexcloud#32-cluster-extension-with-aws-cloud-director-startup-from-scratch), the cloud director takes some time to power up. Its status can be followed in the notice messages sent to the `cmsh` session, or in the Base View event viewer. The status can also be queried via the `status` command in device node. For example, a `watch` instruction such as: ```bash [root@basecm11 ~]# watch 'cmsh -c "device status -n eu-west-1-director"' ``` will show a series of outputs similar to: ```bash eu-west-1-director ....... [ PENDING ] (Waiting for instance to start) eu-west-1-director ....... [ PENDING ] (Waiting for instance to start) eu-west-1-director ....... [ PENDING ] (IP assigned: 54.220.240.166) eu-west-1-director ....... [ PENDING ] (setting up tunnel) eu-west-1-director ....... [ INSTALLER_REBOOTING ] eu-west-1-director ....... [ INSTALLING ] (recreating partitions) eu-west-1-director ....... [ INSTALLING ] (FULL provisioning to "/") eu-west-1-director ....... [ INSTALLING ] (provisioning started) eu-west-1-director [ INSTALLER_CALLINGINIT ] (switching to local root) eu-west-1-director ....... [ UP ] ``` ## 4.2 Launching The Cloud Nodes Once the cloud director is up on the cloud provider, the regular cloud nodes can also be powered up. This does however first require that the corresponding cloud node objects exist. That is, that CMDaemon must have a representation of the cloud nodes, even if they do not yet exist on the cloud provider. The objects must each have an IP address assigned to them that is consistent with that of the cloud director that manages them. That is, the network address of the cloud nodes must be what the cloud director expects. BCM’s cluster extension utilities create 3 cloud node objects by default (figure [4.7](#fig-cm-cluster-extension-configuration-processing-regions), [here](#fig-cm-cluster-extension-configuration-processing-regions)). Cloning them is an easy way to extend the number of deployable cloud nodes. With Base View, this can be done with the `Clone` command to assign properties to the clone that match the original (section [3.3](/cloudbursting/cexcloud#33-cluster-extension-with-aws-cloud-node-startup-from-scratch)), but advance the relevant IP addresses by 1. In `cmsh`, the `clone` command works the same way. To launch a cloud node that has an object, a command can be run as follows: ```bash [basecm11->device[cnode001]->interfaces]% device power on -n cnode001 ``` ### 4.2.1 Creating And Powering Up Many Nodes For a large number of cloud nodes, the creation and assignment of IP addresses can be done with the clone option of the `foreach` command, ([section 2.5.5.9 of the *Administrator Manual*](/admin/clustermanagement#2559-looping-over-objects-with-foreach)), together with a node range specification. This is the same syntax as used to create non-cloud regular nodes with `cmsh`. Earlier on in this section, starting from [here](#4112-running-the-cm-cluster-extension-tui), a TUI session was run that ended up creating * the cloud director `eu-west-1-director` and * regular node objects `eu-west-1-cnode001` up to `eu-west-1-cnode003`. Continuing with the end result of that session, cloning many further regular cloud nodes can now be carried out by cloning `eu-west-1-cnode003`: **Example** ```bash [basecm11->device]% foreach --clone eu-west-1-cnode003 -n eu-west-1-cnode0[04-12] () Warning: The Ethernet switch settings were not cloned, and have to be set manually ... [basecm11->device*]% commit Successfully committed 9 Devices [basecm11->device]% ``` As a reminder, the node range option `-n eu-west-1-cnode004..eu-west-1-cnode012` would also be valid for the preceding example, and perhaps easier to comprehend, although longer. The IP addresses are assigned to the cloud nodes via heuristics based on the value of `eu-west1-cnode003` and its cloud director. Powering up many cloud nodes can carried out using `cmsh` with the node range option as follows: **Example** ```bash [basecm11->device]% power on -n eu-west-1-cnode0[02-10] ``` ## 4.3 Miscellaneous Cloud Tools ### 4.3.1 Setting Exclude Lists With `excludelistsnippets` The `excludelistsnippets` submode allows extra exclude list entries to be created and configured for a provisioned file system. These extra exclude list snippets are added to the regular exclude lists described in [section 5.6 of the *Administrator Manual*](/admin/provisioning#56-updating-running-nodes). The addition of these exclude list snippets to the regular lists is done by setting a mode parameter to the snippets. The mode parameter sets the exclude list that is associated with an exclude list snippet, as indicated by the following table:
**Mode** **Exclude list**
`sync` `excludelistsyncinstall` (active by default)
`full` `excludelistfullinstall`
`update` `excludelistupdate` (active by default)
`grab` `excludelistgrab`
`grab new` `excludelistgrabnew`
An exclude list snippet with an associated mode parameter behaves as if its entries are run along with the regular exclude list that is associated with it. For example, to exclude the files `/useless/test1` and `/useless/test2`, a snippet called `test` may be created as follows: **Example** ```bash [basecm11->fspart]% use /cm/shared [basecm11->fspart[/cm/shared]]% excludelistsnippets [basecm11->fspart[/cm/shared]->excludelistsnippets]% [basecm11->fspart[/cm/shared]->excludelistsnippets]% add test [basecm11->fspart*[/cm/shared*]->excludelistsnippets*[test*]]% set modefull yes [basecm11->fspart*[/cm/shared*]->excludelistsnippets*[test*]]% set excludelist [basecm11->fspart*[/cm/shared*]->excludelistsnippets*[test*]]% set excludelist /useless/test1 [basecm11->fspart*[/cm/shared*]->excludelistsnippets*[test*]]% append excludelist /useless/test2 [basecm11->fspart*[/cm/shared*]->excludelistsnippets*[test*]]% show Parameter Value -------------------------------- ------------------------------------------------ Lines 2 Name test Revision Exclude list /useless/test1,/useless/test2 Disabled no No new files no Mode sync yes Mode full yes Mode update yes Mode grab no Mode grab new no ``` When the `test` snippet is used, its corresponding `.rsync` file has the following configuration: ```bash [root@basecm11 cmd]# grep useless /var/spool/cmd/*shared.rsync - /useless/test1 - /useless/test2 ``` #### 4.3.1.1 Setting Filesystem Synchronization For Edge Director Filesystem Partitions The edge director role allows the `sync/cm/shared` property to decide if the `/cm/shared` filesystem partition should be synchronized. **Example** ```bash [basecm11]% device use edge-director [basecm11->device[edge-director]]% roles [basecm11->device[edge-director]->roles]% use edgedirector [basecm11->device[edge-director]->roles[edgedirector]]% show Parameter Value --------------------------------- ------------------------------------------------ Name edgedirector Revision Type EdgeDirectorRole Add services yes Node power operations yes Director power operations no Add named service yes Add slapd service yes Add ntpd service yes Open TCP ports on head node Open UDP ports on head node Externally visible IP 0.0.0.0 Externally visible head node IP 0.0.0.0 Sync /cm/shared yes Disable automatic exports no Create home directories never ``` ## 4.4 Connecting To AWS With Direct Connect Or A Hardware VPN For simple configurations, BCM recommends, and provides, OpenVPN by default for cluster extension cloudbursting VPN connectivity. If there is a wish to use Direct Connect or a hardware VPN (for example, an IPSec tunnel), then BCM can be configured to work with those. The AWS cluster extension always runs in an AWS Virtual Private Cloud (VPC). In the default deployment scenario, the head node communicates with the cloud nodes using an OpenVPN connection between the head node and the cloud director. In the case of a Direct Connect connection or a hardware VPN, the head node can be configured to communicate directly with the cloud director and cloud nodes. Setting up Direct Connect or a VPN for a cluster extension can be carried out according to these three steps: 1. VPC creation (section [4.4.1](#441-creating-a-vpc)). This step can be skipped if an existing VPC is to be used. 2. Connecting the local network to the VPC (section [4.4.2](#442-connecting-the-local-network-to-the-vpc)). The connection can be with, for example, Direct Connect, or a hardware VPN. This step can be skipped if the local network is already connected to an existing VPC via a Direct Connect or a hardware VPN connection. 3. Configuring and deploying the cluster extension (section [4.4.3](#443-configuring-and-deploying-the-cluster-extension)). ### 4.4.1 Creating a VPC The Amazon Virtual Private Cloud (VPC) is a logically isolated section of the AWS cloud. A VPC allows complete control over the networking environment for cloud resources used by the cluster extension. The VPC enables the cloud director and cloud nodes to securely communicate with each other and can be extended to the local network of the cluster. Documentation for Amazon VPC can be found at: [https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html) A new VPC can be created and configured for the cluster extension as follows: 1. After logging in to the AWS management console, [https://console.aws.amazon.com/console/](https://console.aws.amazon.com/console/), the following navigation path can be followed: `Services` > `VPC` > `VPC Dashboard` > VPCs > `Create VPC`. 2. An IPv4 CIDR block can be set to a desired range. This range should not conflict with the address space used on-premises. A possible range could be, for example: `10.42.0.0/16` 3. Once the VPC range is set, a subnet can be set via the options under the navigation path: `VIRTUAL PRIVATE CLOUD` > `Subnets` > `Create subnet`. 4. The VPC created in step 2 is selected as the VPC during the `Create subnet` dialog. An IPv4 CIDR block that is a subnet of the VPC range must also be defined in the `Create subnet` dialog. A possible subnet range would be, for example: `10.42.0.0/24` ### 4.4.2 Connecting The Local Network To The VPC Amazon offers two options to connect the local network to the VPC: Direct Connect and VPN. #### 4.4.2.1 Connecting Via Direct Connect To connect the local network to the VPC via Direct Connect requires a private virtual interface. Amazon’s instructions to use AWS Direct Connect to access a VPC are at: [https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html](https://docs.aws.amazon.com/directconnect/latest/UserGuide/Welcome.html) #### 4.4.2.2 Connecting Via A Site-To-Site Connection Using A VPN Connecting an on-premises network to the virtual network using a VPN requires an *Amazon Site-to-Site connection*. Amazon’s instructions for configuring a site-to-site VPN connection are at: [https://docs.aws.amazon.com/vpn/latest/s2svpn/SetUpVPNConnections.html](https://docs.aws.amazon.com/vpn/latest/s2svpn/SetUpVPNConnections.html) The site-to-site connection consists of 4 components: 1. a customer gateway (routes traffic from VPC to local network) 2. a target gateway (routes traffic from local network to VPC) 3. a security group 4. a connection The target gateway can either be a virtual private gateway or a transit gateway, depending on the configuration of the other AWS resources. If the customer gateway device does not support BGP, then the site-to-site connection needs to be configured for static routing. The default Static IP Prefixes are: `10.141.0.0/16`, `192.168.200.0/24`. ### 4.4.3 Configuring And Deploying The Cluster Extension Once IP connectivity from on-prem to the AWS virtual network is running, the final step is creating the cluster extension using the newly created site-to-site connection. #### 4.4.3.1 Getting Through Shorewall First, the firewall on the head node must be adjusted to accept traffic from the cloud subnet. Using the `openports` mode of the `firewall` role in `cmsh`, one rule is added per port that must be reachable from the peer subnet, restricting the source address to that subnet where possible: ```bash [root@basecm11 ~]# cmsh [basecm11]% device use basecm11; roles; use firewall; openports [basecm11->device[basecm11]->roles[firewall]->openports]% add ACCEPT net tcp fw [basecm11->device[basecm11]->roles[firewall]->openports]% commit ``` The `add` command is repeated for each required port and protocol. The `shorewall` service is restarted automatically after `commit`. #### 4.4.3.2 Cluster Extension Advanced Settings Configuration A cluster extension can now be created using the `cm-cluster-extension` utility in TUI mode as explained in section [4.1.1](#411-running-the-cm-cluster-extension-script-on-the-head-node-for-cluster-extension-clusters), [here](#4112-running-the-cm-cluster-extension-tui). However, on reaching the summary screen (figure [4.13](#fig-cm-cluster-extension-configuration-processing-summary-screen), [here](#fig-cm-cluster-extension-configuration-processing-summary-screen)), the cluster administrator should not immediately select the `Save config & deploy` option, but should first go to the `Advanced settings` option, which opens up an `Advanced plugin configuration` screen (figure [4.14](#fig-cm-cluster-extension-configuration-processing-advanced-plugins)): ![Cluster Extension Configuration Processing: Advanced Plugins Screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/53dd2cb40411d94845d35ded33c3aca39a58662cf74f378f372ecdd654b571f7/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/cm-cluster-extension-configuration-processing-advanced-plugins.png) *Figure 4.14: Cluster Extension Configuration Processing: Advanced Plugins Screen* The `clusterextension` option should be selected, which opens up an advanced options settings screen (figure [4.15](#fig-cm-cluster-extension-configuration-processing-advanced-plugins-clusterextension-options)): ![Cluster Extension Configuration Processing: Advanced Plugins For ’clusterextension’ Options Screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/704cb66fe8455b0ef983f1c9d2818b093f4fbb8726edd354e470ae07a6faa033/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/cm-cluster-extension-configuration-processing-advanced-plugins-clusterextension-options.png) *Figure 4.15: Cluster Extension Configuration Processing: Advanced Plugins For ’clusterextension’ Options Screen* In this screen: * the `Create tunnel networks` option should be set to `False` * the `VPC for region` \<*`region`*> should be set to the VPC created for the cluster extension. The cluster extension is what is being connected to via Direct Connect or via a site-to-site connection * The `Security groups` to use should be set for the cloud nodes and the cloud director. #### 4.4.3.3 Custom Security Groups Configuration For the security group it is important to realize that the default security groups, as created by the cluster extension utility, do not accept traffic from the IPSec tunnel. It is recommended to create a custom security group, and to use that for both the director and the nodes. To create this security group: 1. From the AWS management console, the following navigation path is used: `VPC Dashboard` > `Security Groups` > `Create security group` 2. The cluster extension VPC is selected 3. A new inbound rule is added, to accept all traffic from the cluster and the VPC. For example: `192.168.200.0/24` and `10.42.0.0/16`. The TUI advanced settings screens can be backed out of by selecting `Back` twice. The option `Save config & deploy` can then be selected to create the cluster extension. #### 4.4.3.4 Cloud Node Certificate Autosigning By default BCM does not issue certificates for nodes on the external network. This means that for cloud nodes the certificates need to be issued manually, once for every new cloud node. Alternatively, to automatically sign certificate requests by cloud nodes, autosign can be enabled by the administrator for external networks. Autosigning may be a security concern, as this allows anyone on the external network to request a node certificate. Autosign can be enabled on externalnet in `cmsh` as follows: ```bash [root@basecm11~]# cmsh [basecm11]% network [basecm11->network]% set externalnet allowautosign always [basecm11->network]% commit ``` The cluster extension can now be deployed as explained in section [4.1.2](#412-launching-the-cloud-director-for-cluster-extension-clusters).