> This page is for version 11 (default). > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Cluster On Demand Cloudbursting With Azure, AWS, OCI, Or GCP ## 2.1 Introduction Cluster On Demand (COD) cloudbursting is when a separate cluster is started up in a cloud, with the cluster head node that manages the cluster also in that cloud. A COD cluster is regarded as an independent virtual cluster (sometimes described as a ’pure’ cloud cluster), and not an extension of an existing physical cluster. This chapter describes how COD can run within the following cloud service providers: * Azure (COD-Azure) * AWS (COD-AWS) * Oracle Cloud Infrastructure (COD-OCI) * Google Cloud Platform (COD-GCP) If the credential requirements for the cloud service providers are met (section [2.2](#22-requirements-for-cod-cloudbursting)), then COD clusters can be managed (created, deleted, and so on) on all cloud platforms by using the COD command line interface tool for that cloud provider. The COD CLI tool is provided by a Docker image or a Python package (section [2.3](#23-cod-using-the-cod-command-line-tool)). After a COD head node has been launched in the cloud, then compute nodes are managed by using the various management interfaces provided by NVIDIA Base Command Manager running on the head node (section [2.12](#212-cod-cloud-node-start-up)). ## 2.2 Requirements For COD Cloudbursting The high level requirements for COD are: * a BCM product key. This key is later activated when the license is installed ([Chapter 4 of the *Installation Manual*](/installation/licensing)) on the head node. * Credentials for the COD cluster, as described in: * Azure (section [2.2.1](#221-credentials-for-azure)) * AWS (section [2.2.2](#222-credentials-for-aws)) * OCI (section [2.2.3](#223-credentials-for-oci)) * GCP (section [2.2.4](#224-credentials-for-gcp)) ### 2.2.1 Credentials For Azure To use COD on Azure, an Azure account subscription is needed from Microsoft. COD-Azure requires the following associated Azure credentials: * tenant ID * subscription ID * Client ID * Client Secret These credentials allow the COD command line tool (section [2.3](#23-cod-using-the-cod-command-line-tool)) to launch a COD with Azure. A CLI-centric way to obtain these credentials requires logging into the Azure web portal using an account that has sufficient privileges. The Azure web bash console is then opened, and the subscription ID can then be listed for the account with: **Example** ```bash azure@Azure:~$ az account list -o table Name CloudName SubscriptionId State IsDefault ------------ ----------- ------------------------------------ ------- ----------- anne AzureCloud 23748c3e-507b-11e9-a994-fa163e9854eb Enabled False nerds AzureCloud b9e22a88-507a-11e9-9352-fa163e9854eb Enabled False ``` A service principal (`sp`) is now created for role-based access control `create-for-rbac` in Active Directory (`ad`), and the remaining 3 credentials can then be seen: **Example** ```bash azure@Azure:~$ az ad sp create-for-rbac --name my-temp-service-principal-for-fred Changing "my-temp-service-principal-for-fred" to a valid URI of "http://my-temp-service-principal-for-fred", which is the required format used for service principal names Retrying role assignment creation: 1/36 Retrying role assignment creation: 2/36 { "appId": "dcf8151e-507a-11e9-a104-fa163e9854eb", ## "Client ID" "displayName": "my-temp-service-principal-for-fred", "name": "http://my-temp-service-principal-for-fred", "password": "bc9f571e-fe4c-43d5-909d-4bc66796eb41", ## "Client secret" "tenant": "8cb88849-6e18-46d6-b0fa-551a47a31681" ## Tenant ID } ``` The newly-created service principal is added to the desired subscription as a contributor. The value of `appID` must be used as the value to the `--assignee` option. This gives the application sufficient permissions for the cluster to run: **Example** ```bash azure@Azure:~$ az role assignment create --assignee dcf8151e-507a-11e9-a104-fa163e9854eb --role\ Contributor --subscription b9e22a88-507a-11e9-9352-fa163e9854eb { "canDelegate": null, "id": "/subscriptions/b9e22a88-507a-11e9-9352-fa163e9854eb/providers/Microsoft.Authorization\ /roleAssignments/1094f75c-507b-11e9-ac8f-fa163e9854eb", "name": "1094f75c-507b-11e9-ac8f-fa163e9854eb", "principalId": "2f899334-507b-11e9-b312-fa163e9854eb", "roleDefinitionId": "/subscriptions/b9e22a88-507a-11e9-9352-fa163e9854eb/providers\ /Microsoft.Authorization/roleDefinitions/b24988ac-6180-42a0-ab88-20f7382dd24c", "scope": "/subscriptions/b9e22a88-507a-11e9-9352-fa163e9854eb", "type": "Microsoft.Authorization/roleAssignments" } ``` The Microsoft documentation suggests that using the name instead of the subscription ID should also work. However in the version that was used at the time of writing (May 2019) this did not work, and an `error 400` was displayed. ### 2.2.2 Credentials For AWS To use COD AWS, an AWS subscription is needed from Amazon. COD-AWS requires the following associated AWS credentials: * Secret Access Key: Available only once, when first generated, as described at [https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys) * Access Key ID: as described at [https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys) * AWS Account ID, as described at [https://docs.aws.amazon.com/general/latest/gr/acct-identifiers.html](https://docs.aws.amazon.com/general/latest/gr/acct-identifiers.html) * AWS Username: This can be either the AWS account root user (the e-mail address of the root user), or it can be an IAM username with sufficient permissions to launch the cluster. These credentials allow the COD command line tool (section [2.3](#23-cod-using-the-cod-command-line-tool)) to launch a COD with AWS. ### 2.2.3 Credentials For OCI To use COD on OCI, an OCI account subscription is needed from Oracle. COD-OCI requires the following OCI credentials: * API signing Key: This key can be either an uploaded public key from cod machine, or a downloaded private key, as described at [https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm#two](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm#two), in the section [`How to Generate an API Signing Key`](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm#two). After it is added, a configuration file is displayed. * User’s OCID: This shows up in the configuration file with a format such as:\ `user=ocid1.user.oc1..` \<*unique ID*>\ where \<*unique ID*> may have a format such as: ```bash aaaaaaaax5552rxvs6ar4qm7kur6jq2kmhrvzmk6coqwq6b7vxoclltr64ba ``` * Fingerprint: This shows up in the configuration file with a format such as:\ `fingerprint=12:34:56:78:90:ab:cd:ef:12:34:56:78:90:ab:cd:ef` * Tenancy’s OCID: This shows up in the configuration file with a format such as:\ `tenancy=ocid1.tenancy.oc1..` \<*unique ID*> * Compartment ID: In the OCI navigation menu, the compartment ID can be found using the navigation path:\ `Identity & Security` > `Identity` > `Compartments`\ Further information on finding the OCID of a compartment can be found at: [https://docs.oracle.com/en-us/iaas/Content/GSG/Tasks/contactingsupport_topic-Finding_the_OCID_of_a_Compartment.htm](https://docs.oracle.com/en-us/iaas/Content/GSG/Tasks/contactingsupport_topic-Finding_the_OCID_of_a_Compartment.htm). To use Oracle’s `oci-cli` utility, the configuration file has to be placed in `~/.oci/config`. For the COD client, the credentials should be added to the COD configuration file as described in [2.6.1](#261-minimal-configuration-file-for-cod-on-oci). These credentials allow the COD command line tool (section [2.3](#23-cod-using-the-cod-command-line-tool)) to launch a COD with OCI. ### 2.2.4 Credentials For GCP To use COD on GCP, a GCP account ([https://cloud.google.com](https://cloud.google.com)) is needed. COD-GCP requires the following GCP credentials: * `project_id`: The GCP project ID * `head_node_service_account` The head node service account ID * `ssh_pub_key_path`: The public SSH key of the user. * `license_product_key`: The product key of the BCM COD cluster. * `cod_prefix`: An arbitrary prefix used to associate the cluster with the user. * `head_node_service_account`: The Google service account. * `inbound_rule`= IP address, port, and protocol specification for Ingress. * `ingress_icmp`= Ingress IP ping address. These credentials allow the COD command line tool (section [2.3](#23-cod-using-the-cod-command-line-tool)) to launch a COD with GCP (section [2.7.2](#272-cluster-creation-run-with-cm-cod-gcp)). ### 2.2.5 Mandatory Inbound Rules By default, a COD cluster is no longer created open to the world. Unless an inbound rule is configured for the cluster, cluster creation fails validation. The user **must** specify one of the following for the cluster: * `inbound_rule` (and, for AWS, `inbound_network_acl_rule`) opening port 22 (SSH), or * `inbound_rule` (and, for AWS, `inbound_network_acl_rule`) opening port 8081 (web management) At least one of these two ports must be open. The port does not have to be open specifically to the IP address of the machine that runs the COD client. For example, creating a cluster on behalf of another user is a valid use case. The value `{CLIENT_IP}` can be used in `inbound_rule` or `inbound_network_acl_rule` instead of a hardcoded IP address, and resolves to the public IP address of the machine running the COD client. Examples for each cloud are given in the minimal configuration file examples in sections [2.4.1](#241-minimal-configuration-file-for-cod-on-azure), [2.5.1](#251-minimal-configuration-file-for-cod-on-aws), [2.6.1](#261-minimal-configuration-file-for-cod-on-oci), and [2.7.1](#271-minimal-configuration-file-for-cod-on-gcp). ## 2.3 COD Using The COD Command Line Tool There are two options for running the COD command line tool: * The COD command line tool can be run from a Docker image: * The Docker instance that runs from the image can be hosted on a standalone PC that is not part of a BCM cluster. It can also be hosted by a BCM node. * Section [2.3.1](#231-cod-using-a-docker-image) describes the procedure for installing and configuring COD on Docker on a BCM node in more detail. * The COD command line tool can be run after installing a Python package in a local (virtual) environment: * The Python package can be installed on a standalone PC. It can also be installed on a BCM node. * Section [2.3.2](#232-cod-using-a-python-package) describes the procedure for installing and configuring the COD client using a Python package on a BCM node in more detail. Once the COD client is installed and configured, the next stage is to launch the COD inside the cloud service. The following table shows the 4 cloud services that can run a COD launched from Docker, along with the utility that launches the COD:
**Cloud Service** **CLI utility in Docker instance that launches the COD**
`Azure` `cm-cod-azure` ( [here](#24-cod-on-azure-using-the-command-line) )
`AWS` `cm-cod-aws` ( [here](#25-cod-on-aws-using-the-command-line) )
`OCI` `cm-cod-oci` ( [here](#26-cod-on-oci-using-the-command-line) )
`GCP` `cm-cod-gcp` ( [here](#27-cod-on-gcp-using-the-command-line) )
### 2.3.1 COD Using A Docker Image The following steps are an overview of how to start up the head node and regular nodes of the COD: * On a standalone machine or on a BCM node, a Docker host is used to pull a BCM COD (Cluster On Demand) image from the Docker registry. * The image is run in a Docker container. * From within the instance of the image running in the container, a cloudbursting request for a COD is carried out to a cloud service provider. * When the request completes successfully, a cluster that is running in the cloud is ready for use. The COD Docker image is typically run from a standalone machine, such as, for example, a laptop that is not part of a BCM cluster. However, launching a COD Docker image from BCM is possible. In more detail, the procedure can be as follows: * Typically Docker on BCM is installed by the cluster administrator using `cm-docker-setup` or using the Base View Docker setup wizard ([section 2.1 of the *Containerization Manual*](/containerization/dockerengine#21-docker-setup)). Installing the distribution version of Docker should not be done, since it leaves out some integrations with BCM. * After installation, the administrator should start the Docker service with:\ `systemctl start docker.service`. * To allow a regular user to run Docker, the user can be added to the `docker` system group. For example, if BCM user management ([section 6.2 of the *Administrator Manual*](/admin/usermanagement#62-managing-users-and-groups-with-cmsh)) has created a user `joe`, then the cluster admiinistrator can add the user to the Docker system group with: **Example** ```bash root@basecm11:~# usermod -a -G docker joe #just a regular joe ``` * User `joe` can then load the environment module ([section 2.3 of the *User Manual*](/user/clusterusage#23-environment-modules)) for Docker, and pull a COD client: **Example** ```bash joe@basecm11:~$ module load docker joe@basecm11:~$ docker pull brightcomputing/cod:latest latest: Pulling from brightcomputing/cod ... cac15ba059ef: Pull complete Digest: sha256:19b263033090e1a70043989decdf3c3870d3def8c2e69b2a85ac293fd7d149ab Status: Downloaded newer image for brightcomputing/cod:latest ``` The command line COD tools installed in the container image all have several command line arguments. Instead of the user having to specify many arguments on every invocation of a COD client, the options can conveniently be set in configuration files. To provide access to such configuration files it is recommended that the home directory of the user running the COD client is mounted into the container instance. The following shell aliases are recommended for each COD client: **Example** ```bash alias cm-cod-aws='docker run --rm -it --network host -v ~:/root brightcomputing/cod cm-cod-aws' alias cm-cod-azure='docker run --rm -it --network host -v ~:/root brightcomputing/cod cm-cod-azure' alias cm-cod-oci='docker run --rm -it --network host -v ~:/root brightcomputing/cod cm-cod-oci' alias cm-cod-gcp='docker run --rm -it --network host -v ~/.config/gcloud:/root/.config/gcloud \ -v ~/.config/cm-cluster-on-demand/:/root/.config/cm-cluster-on-demand/ -v ~/.ssh:/root/.ssh \ -e LOGNAME=$LOGNAME \ brightcomputing/cod cm-cod-gcp' ``` In the preceding aliases: * the `--rm` option ensures that the container instance created to run a command is also cleaned up on completion * the `-it` flags are required because some commands may present interactive prompts to the user * the `--network host` option gives the container network access * the argument to the `-v` flag has the form of \<*source*>`:<`*destination*>, where * \<*source*> points to the home directory of the user invoking the command, and * \<*destination*> is the path where the directory is mounted inside the running container instance. The path of \<*destination*> should always be `/root`. * for the `cm-cod-gcp` alias, the extra GCP mounts are needed for authentication becuase the GCP boot procedure differs from the others, and requires `gcloud`. Once a COD cluster has been created, it is typically accessed using SSH. It is strongly recommended to use key-based authentication instead of password-based authentication. Since the home directory for users is mounted into the container, any existing SSH key pairs (stored in `~/.ssh`) can be used. If no key pair exists, then `ssh-keygen` can be used to create one. To ensure the highest level of compatibility it is recommended to use RSA as the key type and a minimum of 2048 bits. ### 2.3.2 COD Using A Python Package Instead of running the COD client in a Docker container it is also possible to install and run the client directly on the host system in a Python virtual environment. A minimum Python version of 3.12 is required for this. Loading a module as the default for users ([section 2.2.3 of the *Administrator Manual*](/admin/clustermanagement#223-setting-up-a-default-environment-for-all-users)) for the Python version may be helpful. The client can be run from BCM as follows: * A virtual environment is created to install the COD client. For example, on a regular node: **Example** ```bash joe@basecm11:~$ ssh node001 joe@node001:~$ python -V Python 3.10.12 <---version not high enough joe@node001:~$ module load python312 Python 3.12.10 <---can be set for loading automatically joe@node001:~$ python -m venv .venvs/cod joe@node001:~$ source .venvs/cod/bin/activate (cod) python@node001:~$ ``` * Next, the required COD clients are installed. Possible packages that can be installed, and their associated executables, are: | **Pip package** | **Executable** | | :--------------------------- | :------------- | | `cm-cluster-on-demand-aws` | `cm-cod-aws` | | `cm-cluster-on-demand-azure` | `cm-cod-azure` | | `cm-cluster-on-demand-oci` | `cm-cod-oci` | | `cm-cluster-on-demand-gcp` | `cm-cod-gcp` | For example, the COD client tool for AWS can be installed as follows: ```bash (cod) joe@node001:~$ pip install cm-cluster-on-demand-aws ... ``` More than one client can be installed. * After the COD client has been installed, it can be run whenever the virtual environment is activated, without having to install it again: ```bash root@basecm11:~# python -V Python 3.10.12 root@basecm11:~# su joe Loading gcc/14.2.0 Loading requirement: gmp/6.3.0 mpfr/4.2.1 mpc/1.3.1 joe@basecm11:/root$ cd joe@basecm11:~$ python -V Python 3.10.12 joe@basecm11:~$ source .venvs/cod/bin/activate (cod) joe@basecm11:~$ python -V Python 3.12.10 (cod) joe@node001 ~$ cm-cod-aws usage: cm-cod-aws [-h] [--config CONFIG] [--no-system-config] [-v] [--show-configuration] [--log-file LOG_FILE] [--version] cluster,c,cluster create,cc,cluster list,cl,cluster delete,cd,cr,cremove,image,i,image list,il,instancetype,config ... cm-cod-aws: error: the following arguments are required: group ``` ## 2.4 COD On Azure Using The Command Line Launching a COD inside Azure is carried out by running `cm-cod-azure` from either the Docker image or the Python package. The `cm-cod-azure` command can take configuration options from one or more configuration files, environment variables, or command line options. More details on this are given in section [2.8](#28-cod-client-configuration-and-command-line-options). Typically some of the more static options are configured in a configuration file, while some of the more variable options are specified on the command line. Online help is also available: for example, running `cm-cod-azure cluster create -h` displays a list of options and explanations for the `cluster create` command. ### 2.4.1 Minimal Configuration File For COD On Azure It is recommended to create a minimal configuration in `~/cm-cluster-on-demand.d/config.ini` The credentials and product key in the following example are dummy credentials, and must be replaced with the appropriate values for the Azure instance used by the cluster administrator. The value for `ssh_pub_key_path` is the key for connecting to the head node, which can be generated using `ssh-keygen -b 4096`. **Example** ```bash [azure.credentials] azure_subscription_id=5e519b1e-aff9-e839-bc3a-c5d87e9d0d5 azure_tenant_id=2c89c8dd-6b8b-5393-60f4-d876cbe188f azure_client_id=afe13723-c80a-68e2-9cd0-e95a657106a azure_client_secret=aiVohwi7OhJ6igim= azure_location=westeurope [azure.cluster.create] inbound_rule={CLIENT_IP},22:tcp [cluster.create.license] license_product_key=123456-123456-123456-123456-123456 [cluster.create.password] ssh_pub_key_path=/root/.ssh/id_rsa.pub ``` The `inbound_rule` setting shown opens SSH access to the machine running the COD client (section [2.2.5](#225-mandatory-inbound-rules)). The credentials allow the COD command line tool (section [2.3](#23-cod-using-the-cod-command-line-tool)) to launch a COD with Azure (section [2.4.2](#242-cluster-creation-run-with-cm-cod-azure)). ### 2.4.2 Cluster Creation Run With `cm-cod-azure` An example of a `cm-cod-azure` command that can be run is then: **Example** ```bash cm-cod-azure cluster create \ --head-node-type Standard_D1_v2 --head-node-root-volume-size 50 \ --nodes 3 --node-type Standard_D1_v2 \ --name testcluster ``` The BCM Azure integration makes use of several node-installer images which are published in the Azure marketplace. In order to use those images, Azure requires the administrator to accept the legal terms for those images. The code automatically detects if the terms were already accepted in the past, in which case it proceeds and creates the cluster. If the terms still need to be accepted, then several links to the legal terms are presented, along with a prompt to accept them. The output looks similar to the following (some output ellipsized): **Example** ```bash To use Azure VM images, you need to accept the following terms: - License (bcmni-azure-9-2:bcm-ni-azure-9-2-v1): https://download.microsoft.com/download/F/D/8/FD8BA8F2... - Privacy Policy (bcmni-azure-9-2:bcm-ni-azure-9-2-v1): https://www.brightcomputing.com/privacy-policy - Marketplace Terms (bcmni-azure-9-2:bcm-ni-azure-9-2-v1): https://mpcprodsa.blob.core.windows.net/market... - License (bcmni-azure-9-2-free:bcm-ni-azure-9-2-v2): https://download.microsoft.com/download/F/D/8/FD8BA... - Privacy Policy (bcmni-azure-9-2-free:bcm-ni-azure-9-2-v2): https://www.brightcomputing.com/privacy-policy - Marketplace Terms (bcmni-azure-9-2-free:bcm-ni-azure-9-2-v2): https://mpcprodsa.blob.core.windows.net/mark... Accept (y/n)? ``` If all is well, then something similar to the following is displayed (some output elided): **Example** ```bash [root@cod-client ~]# cm-cod-azure cluster create \ --head-node-type Standard_D1_v2 --head-node-root-volume-size 50 \ --nodes 3 --node-type Standard_D1_v2 \ --name testcluster 09:35:01: INFO: No custom cluster password specified. Using a random password. Use --log-cluster-password to see it. If you are going to use the Bright View you have to change this password by using cm-change-passwd script (change root password of head node) or simply 'passwd root' command. If this cluster has a high-availability setup with 2 head nodes, be sure to change the password on both head nodes. 09:35:05: INFO: Temporary azure resources will be created in order to verify that the API credentials have the required permissions, this can take a few minutes. You can skip this step by specifying the following flag: '--skip-permission-verifications' 09:35:05: INFO: Cluster Create 09:35:05: INFO: ------------------------------------------------- 09:35:05: INFO: Cluster: testcluster 09:35:05: INFO: ------------------------------------------------- 09:35:05: INFO: Image name: bcm-cod-image-9.0-8 09:35:05: INFO: Image date: 2020-08-21 15:10 (38d 18h ago) 09:35:05: INFO: Package groups: none 09:35:05: INFO: Version: 9.0 09:35:05: INFO: Head nodes: 1 (Standard_D1_v2) 09:35:05: INFO: Nodes: 3 (Standard_D1_v2) 09:35:05: INFO: Resource Group: testcluster_cod_resource_group 09:35:05: INFO: Region: westeurope 09:35:05: INFO: ------------------------------------------------- 09:35:05: INFO: Press ENTER to continue and create the cluster. 09:35:05: INFO: Press ctrl+c (or type 'a') to abort. Type 'i' for more info. 09:35:45: INFO: Credentials are valid and have read/write authorizations. 09:38:17: INFO: ## Progress: 2 09:38:17: INFO: #### stage: Creating resource group testcluster_cod_resource_group 09:38:18: INFO: Creating storage account testclusterstoragerg68u3 09:38:36: INFO: ## Progress: 7 09:38:36: INFO: #### stage: Copying head node image from https://brightimages.blob.core.windows.net/images/bcm-cod-image-9.0-8.vhd 09:38:36: INFO: Copying https://brightimages.blob.core.windows.net/images/bcm-cod-image-9.0-8.vhd to testclusterstoragerg68u3/images/testcluster-head-node-os-disk.vhd 09:38:37: INFO: ## Progress: 12.0 09:38:37: INFO: #### stage: Server side copy 09:38:59: INFO: ## Progress: 12.13 09:38:59: INFO: #### stage: Server side copy 09:39:03: INFO: ## Progress: 12.26 ... 09:49:40: INFO: #### stage: Server side copy 09:49:41: INFO: ## Progress: 51.85 09:49:41: INFO: #### stage: Server side copy 09:49:43: INFO: ## Progress: 51.98 09:49:43: INFO: #### stage: Server side copy 09:49:45: INFO: Elapsed: 11:08 min 09:49:45: INFO: Creating Azure Image resource 'testcluster-head-node-os-disk' from blob https://testclusterstoragerg68u3.blob.core.windows.net/images/testcluster-head-node-os-disk.vhd 09:50:21: INFO: Deleting blob testclusterstoragerg68u3/images/testcluster-head-node-os-disk.vhd 09:50:21: INFO: ## Progress: 5 09:50:21: INFO: #### stage: Building deployment template 09:50:21: INFO: generating cloud-init script 09:50:21: INFO: ## Progress: 85 09:50:21: INFO: #### stage: Creating and deploying Head node 09:52:22: INFO: Head node IP: 51.145.169.45 09:52:22: INFO: Waiting for sshd to start (ssh root@51.145.169.45). 09:52:22: INFO: Waiting for CMDaemon to start. 09:55:05: INFO: Waiting for CMDaemon to initialize. 09:56:07: INFO: ## Progress: 100 09:56:07: INFO: #### stage: Deployment finished successfully. 09:56:07: INFO: ----------------------------------------------------- 09:56:07: INFO: Cluster: testcluster 09:56:07: INFO: ----------------------------------------------------- 09:56:07: INFO: Head node ID: 7f03d713-71ad-4eb6-88ff-045f82a41990 09:56:07: INFO: Public IP: 51.145.169.45 09:56:07: INFO: Key path: /root/.ssh/id_rsa.pub 09:56:07: INFO: ----------------------------------------------------- [root@cod-client ~]# ``` The cluster can now be logged into by using the SSH keys generated earlier, and using the public IP address shown in the last few lines of the preceding output. The regular nodes running on Azure are configured for use, but are not powered on. They can be powered on using the `cmsh` or Base View front ends, just as in a regular cluster. From `cmsh`, the first cloud node can be powered on with, for example: ```bash [testcluster->device]% power on cnode001 ``` More details can be found in section [2.12](#212-cod-cloud-node-start-up). ### 2.4.3 Listing Clusters With `cm-cod-azure` The clusters that are running can be listed with the `cluster list` command option: **Example** ```bash [root@cod-client ~]# cm-cod-azure cluster list +--------------+-----------------------+---------------+------------+-... | Cluster Name | Head node name | Public IP | Location | ... +--------------+-----------------------+---------------+------------+-... | testcluster | testcluster-head-node | 51.145.169.45 | westeurope | ... +--------------+-----------------------+---------------+------------+-... ``` Since all of the credentials are specified in the configuration file, no further command line options are needed. ### 2.4.4 Cluster Removal With `cm-cod-azure` A cluster can be removed with a command in the form of: `cluster delete` \<*cluster name*>: **Example** ```bash [root@cod-client ~]# cm-cod-azure cluster delete testcluster 10:10:47: INFO: Deleting resource group(s): testcluster_cod_resource_group 10:10:47: INFO: Proceed? [yes/no] y 10:10:52: INFO: Started deleting resources for Resource Group 'testcluster_cod_resource_group' [root@cod-client ~]# ``` ## 2.5 COD On AWS Using The Command Line Launching a COD inside AWS is carried by running `cm-cod-aws` from either the Docker image or the Python package. The `cm-cod-aws` command can take configuration options from one or more configuration files, environment variables, or command line options. More details on this are given in section [2.8](#28-cod-client-configuration-and-command-line-options). Typically some of the more static options are configured in a configuration file, while some of the more variable options are specified on the command line. Online help is also available. For example, running `cm-cod-aws cluster create -h` displays a list of options and explanations for the `cluster create` command. ### 2.5.1 Minimal Configuration File For COD On AWS Creation of a minimal configuration file in `~/cm-cluster-on-demand.d/config.ini` is recommended, with the following contents: **Example** ```bash [aws.credentials] aws_access_key_id=AKIAJAICHAZI7OHH1EEGO aws_secret_key=Oocei6eiieshahG7IiJe9Quoud0oo aws_region=eu-west-1 [aws.cluster.create] inbound_rule={CLIENT_IP},22:tcp inbound_network_acl_rule={CLIENT_IP},22:tcp,ALLOW,100 [cluster.create.license] license_product_key=123456-123456-123456-123456-123456 [cluster.create.password] ssh_pub_key_path=/root/.ssh/id_rsa.pub ``` The `inbound_rule` and `inbound_network_acl_rule` settings shown open SSH access to the machine running the COD client (section [2.2.5](#225-mandatory-inbound-rules)). The example credentials and product key should be replaced with the appropriate values. The value for `ssh_pub_key_path` is the key for connecting to the head node, which can be generated using `ssh-keygen -b 4096`. The credentials allow the COD command line tool (section [2.3](#23-cod-using-the-cod-command-line-tool)) to launch a COD with AWS (section [2.5.2](#252-cluster-creation-run-with-cm-cod-aws)). ### 2.5.2 Cluster Creation Run With `cm-cod-aws` An example of a `cm-cod-aws` command that can be run is then: **Example** ```bash cm-cod-aws cluster create \ --head-node-type t3.medium --head-node-root-volume-size 50 \ --nodes 3 --node-type t3.medium \ --name testcluster ``` If all is well, then something similar to the following is displayed (some output elided): **Example** ```bash [root@cod-client ~]# cm-cod-aws cluster create \ --head-node-type t3.medium --head-node-root-volume-size 50 \ --nodes 3 --node-type t3.medium \ --name testcluster 13:39:59: INFO: No custom cluster password specified. Using a random password. Use --log-cluster-password to see it. If you are going to use the Bright View you have to change this password by using cm-change-passwd script (change root password of head node) or simply 'passwd root' command. If this cluster has a high-availability setup with 2 head nodes, be sure to change the password on both head nodes. 13:40:01: INFO: ----------------------------------------------------------------------------- 13:40:01: INFO: Cluster: testcluster 13:40:01: INFO: ----------------------------------------------------------------------------- 13:40:01: INFO: Image name: brightheadnode-9.0-centos7u7-hvm-9(ami-07ef81b0e2d680131:9) 13:40:01: INFO: Image date: 2020-08-21 13:44 (39d 0h ago) 13:40:01: INFO: Package groups: none 13:40:01: INFO: Version: 9.0 13:40:01: INFO: Distro: centos7u7 13:40:01: INFO: Head nodes: 1 (t3.medium) 13:40:01: INFO: Head node IP: (set with --head-node-internal-ip) 13:40:01: INFO: Nodes: 3 (t3.medium) 13:40:01: INFO: Region: eu-west-1 13:40:01: INFO: Key path: /root/.ssh/id_rsa.pub 13:40:01: INFO: ------------------------------------------------------------------------------ 13:40:01: INFO: Press ENTER to continue and create the cluster. 13:40:01: INFO: Press ctrl+c (or type 'a') to abort. Type 'i' for more info. 13:40:02: INFO: ## Progress: 2 13:40:02: INFO: #### stage: Setting up VPC 13:40:02: INFO: ## Progress: 5 13:40:02: INFO: #### stage: Creating VPC for on-demand testcluster in eu-west-1... 13:40:03: INFO: ## Progress: 10 13:40:03: INFO: #### stage: Adding internet connectivity... 13:40:04: INFO: ## Progress: 12 13:40:04: INFO: #### stage: Creating public subnet... 13:40:05: INFO: ## Progress: 14 13:40:05: INFO: #### stage: Creating private subnet... 13:40:06: INFO: Created security group for the head node: sg-057c11f4d9857a025 13:40:06: INFO: Created security group for the compute node: sg-06249aedeffcb0d2d 13:40:07: INFO: ## Progress: 16 13:40:07: INFO: #### stage: Creating head node (t3.medium)... 13:40:07: INFO: Public key specified. 13:40:07: INFO: Creating the head node VM instance. 13:40:08: INFO: Created VM i-0d92887ef735f87b1. 13:40:09: INFO: Waiting for head node to be running 13:40:25: INFO: ## Progress: 22 13:40:25: INFO: #### stage: Done setting up VPC for on-demand testcluster. 13:40:25: INFO: ## Progress: 22 13:40:25: INFO: #### stage: Starting the head node 13:40:25: INFO: ## Progress: 30 13:40:25: INFO: #### stage: Waiting for the head node to start 13:40:25: INFO: ## Progress: 32 13:40:25: INFO: #### stage: Assigning public IP to the head node (use '--no-head-node-assign-public-ip' to skip) 13:40:27: INFO: Elastic IP assigned: 54.217.104.57. 13:40:27: INFO: ## Progress: 30 13:40:27: INFO: #### stage: Waiting for cloud-init to start (use '--cloud-init-timeout 0' to skip) 13:40:27: INFO: Waiting for cloud-init to finish on the cluster testcluster (by trying to connect to 54.217.104.57:8081). 13:44:04: INFO: ## Progress: 100 13:44:04: INFO: #### stage: Deployment of cluster: testcluster finished successfully. 13:44:14: INFO: Script completed. 13:44:14: INFO: --------------------------------------------------------------------- 13:44:14: INFO: Time it took: 04:12 13:44:14: INFO: SSH string: 'ssh root@54.217.104.57' 54.217.104.57 testcluster 13:44:14: INFO: Head node ID: i-0d92887ef735f87b1 [root@cod-client ~]# ``` At this point, the head node running on AWS is ready for use. It can be accessed using SSH, as suggested in the SSH string line of the preceding output. The regular nodes running on AWS are configured for use, but are not powered on. They can be powered on using the `cmsh` or Base View front ends, just as in a regular cluster. From `cmsh`, the first cloud node can be powered on with, for example: ```bash [testcluster->device]% power on cnode001 ``` More details can be found at section [2.12](#212-cod-cloud-node-start-up). ### 2.5.3 Listing Clusters With `cm-cod-aws` The clusters that are running can be listed with the command option `cluster list`: **Example** ```bash [root@cod-client ~]# cm-cod-aws cluster list 14:16:44: INFO: Listing clusters in region eu-west-1 +----------------+-----------------------+---------------------+---------------+-... | Cluster Name | VPC ID | Head node ID | Public IP | ... +----------------+-----------------------+---------------------+---------------+-... | testcluster | vpc-019dacc0877bfd71a | i-0d92887ef735f87b1 | 54.217.104.57 | ... +----------------+-----------------------+---------------------+---------------+-... ``` Since all of the credentials are specified in the configuration file, no further command line options are needed. ### 2.5.4 Cluster Removal With `cm-cod-aws` A cluster can be removed with a command in the form of: `cluster delete` \<*cluster name*>: **Example** ```bash [root@cod-client ~]# cm-cod-aws cluster delete testcluster 14:18:57: INFO: This will destroy VPCs for testcluster, continue? 14:18:57: INFO: Proceed? [yes/no] y 14:19:00: INFO: Stopping instances for VPCs on-demand testcluster 14:19:00: INFO: Listing instances... 14:19:00: INFO: Issuing instances termination requests... 14:19:00: INFO: Waiting until instances terminated... 14:20:31: INFO: Destroying VPC on-demand testcluster 14:20:31: INFO: Deleting subnets... 14:20:32: INFO: Deleting route tables... 14:20:32: INFO: Detaching and deleting gateways... 14:20:33: INFO: Flushing permissions... 14:20:34: INFO: Deleting security groups... 14:20:35: INFO: Deleting VPC... 14:20:35: INFO: Done destroying VPC on-demand testcluster [root@cod-client ~]# ``` ### 2.5.5 ARM64 Images Support In COD COD supports `arch64` images. This allows workloads to run on ARM-based infrastructure. A cluster with `arch64` images can be created with the `--arch=aarch64` CLI parameter. An instance type that supports `aarch64` images must be chosen for this to work. **Example** ```bash cm-cod-aws cc --name arm64-cluster --arch=aarch64 --head-node-type t4g.medium ...[other options]... ``` The text `...[`*`other options`*`]...` in the preceding example should be replaced with additional parameters required for the specific cluster configuration. #### 2.5.5.1 ARM64 Images Support Limitations For COD: * `aarch64` image support is available only for COD-AWS, COD-OCI, and COD-GCP. Support for other COD providers may be added in future releases. * Mixed-architecture clusters are not supported. The head node and all compute nodes must use the same architecture-—either all `x86_64` or all `aarch64`. ## 2.6 COD On OCI Using The Command Line Launching a COD inside OCI is carried out by running `cm-cod-oci` from either the Docker image or the Python package. The `cm-cod-oci` command can take configuration options from one or more configuration files, environment variables, or command line options. More details on this are given in section [2.8](#28-cod-client-configuration-and-command-line-options). Typically some of the more static options are configured in a configuration file, while some of the more variable options are specified on the command line. Online help is also available. For example, running `cm-cod-oci cluster create -h` displays a list of options and explanations for the `cluster create` command. ### 2.6.1 Minimal Configuration File For COD On OCI The creation of a minimal configuration file in `~/cm-cluster-on-demand.d/config.ini` is recommended, with the following contents: **Example** ```bash [oci.credentials] oci_user=ocid1.user.oc1.. oci_fingerprint=12:34:56:78:90:ab:cd:ef:12:34:56:78:90:ab:cd:ef oci_tenancy=ocid1.tenancy.oc1.. oci_region=us-sanjose-1 oci_key_file=~/.oci/oci_api_key.pem [oci.cluster.common] oci_compartment_id = "ocid1.compartment.oc1.. [oci.cluster.create] inbound_rule={CLIENT_IP},22:tcp [cluster.create.password] ssh_pub_key_path=/root/.ssh/id_rsa.pub ``` The `inbound_rule` setting shown opens SSH access to the machine running the COD client (section [2.2.5](#225-mandatory-inbound-rules)). The credentials should be replaced with the OCI credentials from [2.2](#22-requirements-for-cod-cloudbursting). The value for `ssh_pub_key_path` is the key for connecting to the head node, which can be generated using `ssh-keygen -b 4096`. The credentials allow the COD command line tool (section [2.3](#23-cod-using-the-cod-command-line-tool)) to launch a COD with OCI (section [2.6.2](#262-cluster-creation-run-with-cm-cod-oci)). ### 2.6.2 Cluster Creation Run With `cm-cod-oci` An example of a `cm-cod-oci` command that can be run is then: **Example** ```bash cm-cod-oci cluster create \ --head-node-number-cpus 2 --head-node-root-volume-size 50 \ --head-node-shape VM.Standard.E4.Flex --nodes 2 \ --node-shape VM.Standard3.Flex --node-number-cpus 2 --node-root-volume-size 50 \ --name testcluster ``` Note: Bare metal shapes with no local disks, such as `BM.Standard3.64`, are not supported yet. If all is well, then something similar to the following is displayed (some output elided): **Example** ```bash [root@cod-client ~]# cm-cod-oci cluster create \ --head-node-number-cpus 2 --head-node-root-volume-size 50 \ --head-node-shape VM.Standard.E4.Flex --nodes 2 \ --node-shape VM.Standard3.Flex --node-number-cpus 2 --node-root-volume-size 50 \ --name testcluster 05:35:54: INFO: No custom cluster password specified. Using a random password. Use --log-cluster -password to see it. If you are going to use the Bright View you have to change this password by us- ing cm-change-passwd script (change root password of head node) or simply 'passwd root' command. If this cluster has a high-availability setup with 2 head nodes, be sure to change the password on both head nodes. 05:35:56: INFO: Oracle Cloud Infrastructure (OCI) requires acceptance of terms & conditions ass- ociated with relevant images whenever a new version of these images is made available. The following terms & conditions / end user license agreements must be accepted in order to use this software: Head node terms: (already accepted) Compute node terms: - https://objectstorage.eu-marseille-1.oraclecloud.com/n/axogvvqxyiae/b/community-application /o/eula.html - https://cloudmarketplace.oracle.com/marketplace/content?contentId=50511634&render=inline - https://www.oracle.com/legal/privacy/privacy-policy.html Do you grant permission for Bright Cluster Manager to accept the required end user license agreements and terms & conditions on your behalf when creating machine instances in OCI? 05:35:56: INFO: Proceed? [yes/no] yes 05:35:58: INFO: ------------------------------------------------------------------------- 05:35:58: INFO: Cluster: testcluster 05:35:58: INFO: ------------------------------------------------------------------------- 05:35:58: INFO: Image name: bcmh-ubuntu2004-10.0-1021(ubuntu2004-10.0:1021) 05:35:58: INFO: Image date: 2023-05-14 07:09 (22h 26m ago) 05:35:58: INFO: Package groups: none 05:35:58: INFO: Version: 10.0 05:35:58: INFO: Distro: ubuntu2004 05:35:58: INFO: Head node: 1 (VM.Standard.E4.Flex) 05:35:58: INFO: Nodes: 2 (VM.Standard3.Flex) 05:35:58: INFO: Region: us-sanjose-1 05:35:58: INFO: Compartment: bright 05:35:58: INFO: Key path: /root/.ssh/id_rsa.pub 05:35:58: INFO: Cloud authentication: Head node will inherit COD credentials for cloud authentication 05:35:58: INFO: ----------------------------------------------------------------------- 05:35:58: INFO: Press ENTER to continue and create the cluster. 05:35:58: INFO: Press ctrl+c (or type 'a') to abort. Type 'i' for more info. 05:36:00: INFO: No availability domain configured; selected 'exbx:US-SANJOSE-1-AD-1' from those in the configured compartment 05:36:01: INFO: Creating VCN 05:36:05: INFO: Creating subnet 05:36:07: INFO: Creating network security groups 05:36:09: INFO: Creating head node 05:37:18: INFO: Waiting for sshd to start (ssh root@155.248.204.211). 05:38:26: INFO: Waiting for CMDaemon to start. 05:39:28: INFO: Waiting for CMDaemon to initialize. 05:41:38: INFO: ----------------------------------------------------------------------- 05:41:38: INFO: Cluster: testcluster 05:41:38: INFO: ----------------------------------------------------------------------- 05:41:38: INFO: Head node ID: ocid1.instance.oc1.us-sanjose-1.anzwuljricxcl6acdujsuhn 3yw6uze3h3c44jgbgitwwlqceqpdqkebzij3q 05:41:38: INFO: Public IP: 155.248.204.211 05:41:38: INFO: Key path: /root/.ssh/id_rsa.pub 05:41:38: INFO: ----------------------------------------------------------------------- ``` At this point, the head node running on OCI is ready for use. It can be accessed using SSH, as suggested in the SSH string line of the preceding output. The regular nodes running on OCI are configured for use, but are not powered on. They can be powered on using the `cmsh` or Base View front ends, just as in a regular cluster. In `cmsh`, the first cloud node can be powered on with: **Example** ```bash [testcluster->device]% power on cnode001 ``` More details can be found in section [2.12](#212-cod-cloud-node-start-up). ### 2.6.3 Listing Clusters With `cm-cod-oci` The clusters that are running can be listed with the `cluster list` command line option: **Example** ```bash [root@cod-client ~]# cm-cod-oci cluster list 05:47:06: INFO: Listing clusters in region us-sanjose-1 +-------------+--------------------------+-----------------------+---------+--------------- | Cluster Name| VCN Name | Head node name | State | Shape +-------------+--------------------------+-----------------------+---------+--------------- | testcluster | on-demand testcluster-01 | testcluster-head-node | RUNNING | VM.Standard.E4 +-------------+--------------------------+-----------------------+---------+--------------- ``` Since all of the credentials are specified in the configuration file, no further command line options are needed. ### 2.6.4 Cluster Removal With `cm-cod-oci` A cluster `testcluster` can be removed with a command such as: **Example** ```bash [root@cod-client ~]# cm-cod-oci cluster delete -- filters testcluster 05:50:45: INFO: This will delete all OCI resources tagged as 'testcluster'; continue? 05:50:45: INFO: Proceed? [yes/no] yes 05:50:49: INFO: Cluster testcluster: No cluster networks 'BCM_Cluster=testcluster'; skipping 05:50:49: INFO: Cluster testcluster: No instance pools found matching 'BCM_Cluster=testcluster'; skipping 05:50:49: INFO: Cluster testcluster: Terminating testcluster-head-node (instance state: RUNNING)... 05:51:54: INFO: Cluster testcluster: Terminated 1 instance in 64.48 seconds 05:51:54: INFO: Cluster testcluster: No volumes found matching 'BCM_Cluster=testcluster'; skipping 05:51:55: INFO: Cluster testcluster: Deleting subnet 'testcluster-01' 05:51:58: INFO: Cluster testcluster: Deleting route table 'Default Route Table for on-demand testcluster-01' 05:52:00: INFO: Cluster testcluster: Deleting internet gateway 'testcluster-01' 05:52:02: INFO: Cluster testcluster: No Instance Configurations found matching 'BCM_Cluster=test- cluster'; skipping 05:52:02: INFO: Cluster testcluster: Deleting VCN 'on-demand testcluster-01' [root@cod-client ~]# ``` ## 2.7 COD On GCP Using The Command Line Launching a COD inside GCP is carried out by running `cm-cod-gcp` from either the Docker image or the Python package. The `cm-cod-gcp` command can take configuration options from one or more configuration files, environment variables, or command line options. More details on this are given in section [2.8](#28-cod-client-configuration-and-command-line-options). Typically some of the more static options are configured in a configuration file, while some of the more variable options are specified on the command line. Online help is also available. For example, running `cm-cod-gcp cluster create -h` displays a list of options and explanations for the `cluster create` command. #### 2.7.0.1 The `gcloud` Application The `gcloud` application is a CLI application provided by Google to run its GCP services. The GCP services can be integrated with BCM by using a `gcp.ini` file to set initial values. This file can be read by `gcloud` and by the `cm-cod-gcp` client. The application can be installed on a head node, or it can be installed within a regular node image. The cluster administrator can set the credentials for COD in GCP in `gcp.ini` How to set up `gcloud` and have it work with BCM using credentials is described next. **Installing `gcloud`:** The `gcloud` application is distributed by Google from [https://cloud.google.com/sdk/docs/install](https://cloud.google.com/sdk/docs/install). A general Linux tarball is available at that URL. This can be picked up, unpacked, and installed by a non-root user: **Example** ```bash joe@head:~$ curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/ google-cloud-cli-linux-x86_64.tar.gz ... joe@head:~$ tar xvzf google-cloud-cli-linux-x86_64.tar.gz ... joe@head:~$ cd google-cloud-sdk ``` A directory `~/google-cloud-sdk` is created by the tarball extraction. The `gcloud` application is found in the `bin` directory inside that directory. An installation procedure is run in the main directory with `install.sh` script: ```bash joe@head:~/google-cloud-sdk$ ./install.sh ... Welcome to the Google Cloud CLI! ... Modify profile to update your $PATH and enable shell command completion? Do you want to continue (Y/n)?Y ... ``` If the defaults are used during the preceding run, then the user’s `.bashrc` file is slightly modified to allow the `gcloud` binary to be included in `$PATH`. The `gcloud` binary can then be used if the user, `joe` in this case, logs into the system again. ### 2.7.1 Minimal Configuration File For COD On GCP The `gcp.ini` file should be created by the user within the directory `~/.config/cm-cluster-on-demand/` **Example** ```bash joe@head:~/.config/cm-cluster-on-demand$ cat gcp.ini [gcp.common] project_id = "nv-bcmcodgcp-20240216" [gcp.cluster.create] head_node_service_account = ci-202@nv-bcmcodgcp-20240216.iam.gserviceaccount.com head_node_type = e2-medium head_node_zone = europe-west4-c node_type = e2-medium image_storage_location = eu inbound_rule="87.233.215.97/32,22:tcp", "87.233.215.97/32,8081:tcp", "82.168.163.140/32,22:tcp", "87.213.230.195/32,22:tcp" ingress_icmp = 87.233.215.97/32 [cluster.create] #ssh_password_authentication = true #log_cluster_password = true [cluster.create.password] ssh_pub_key_path = ~/.ssh/id_ecdsa.pub [cluster.create.license] license_product_key = 123456-123456-123456-123456-123456 [cluster.prefix] cod_prefix = joe ``` The project ID and service account are specific to the GCP account. The machine types available for GCP are listed at [https://cloud.google.com/compute/docs/machine-resource](https://cloud.google.com/compute/docs/machine-resource). The commented-out password authentication and logging lines are not needed if certificate-based SSH authentication is used, with `ssh_pub_key_path`. The inbound rules are Ingress rules, and are specific for the end user, which means that they should be replaced by appropriate rules for the user. The format is a CIDR IP address, with a port number and a protocol. The value `{CLIENT_IP}` can be used instead of a hardcoded IP address, and at least one of port 22 or port 8081 must be opened for cluster creation to succeed (section [2.2.5](#225-mandatory-inbound-rules)). The ICMP rule is to allow pings at that server IP address. A value for `cod_prefix` is not needed, but is almost always useful. #### 2.7.1.1 Getting The Authentication Token With `gcp.ini` The `gcp.ini` file is read by running an authentication session with a `gcloud` authentication command as follows: ```bash joe@head:~$ gcloud auth application-default login ``` The user is prompted to go to a long URL that uses the Google Auth Library. The library should be allowed access to what it requests. A verification code is provided by the library. The code should be entered in the `gcloud` authentication session. This then sets up an authentication token that is valid for 24 hours. After the token expires, then the `gcloud` authentication command can be run again to allow authentication again. Authentication allows the COD command line tool (section [2.3](#23-cod-using-the-cod-command-line-tool)) to launch a COD with GCP (section [2.7.2](#272-cluster-creation-run-with-cm-cod-gcp)) ### 2.7.2 Cluster Creation Run With `cm-cod-gcp` An example of a `cm-cod-gcp` command that can be run is then: **Example** ```bash joe@head:~$ cm-cod-gcp cluster create \ --head-node-root-volume-size 50 \ --head-node-type e2-medium --nodes 2 \ --node-type e2-medium --node-root-volume-size 50 \ --name testcluster ``` ### 2.7.3 Cluster Removal With `cm-cod-gcp` A cluster `testcluster` can be removed with a command such as: **Example** ```bash joe@head:~$ cm-cod-gcp cluster delete -- filters testcluster 05:50:45: INFO: This will delete all GCP resources tagged as 'testcluster'; continue? 05:50:45: INFO: Proceed? [yes/no] ... ``` ## 2.8 COD Client Configuration And Command Line Options All COD clients (`cm-cod-azure` (section [2.4](#24-cod-on-azure-using-the-command-line)), `cm-cod-aws` (section [2.5](#25-cod-on-aws-using-the-command-line)), and `cm-cod-oci` (section [2.6](#26-cod-on-oci-using-the-command-line)) `cm-cod-gcp` (section [2.7](#27-cod-on-gcp-using-the-command-line)) ) have a similar command line structure. They can take many command line arguments. All of those arguments can also be specified in configuration files, or as environment variables. Options specified as environment variables override options specified in configuration files. Options specified on the command line override options defined as environment variables as well as options defined in configuration files. ### 2.8.1 Command Line Structure The command line structure is similar for all the different COD clients. Each client offers a number of top-level commands, which each offer a number of sub-level commands. The top-level and sub-level commands are always the first two positional arguments. Each top-level as well as sub-level commands can have several optional arguments. The syntax is indicated by: ```bash cm-cod-{aws,azure,oci,gcp} [top-level command [sub-level command]] options ``` #### 2.8.1.1 Command Line Help To find out which commands and options are available, the `--help`|`-h` option can be used. The following example shows how to display the top-level commands and options for the `cm-cod-aws` client: **Example** ```bash [root@cod-client ~]# cm-cod-aws --help usage: cm-cod-aws [-h] [--config CONFIG] [--no-system-config] [-v] [--show-configuration] [--log-file LOG_FILE] [--version] {cluster,c,cluster create,cc,cluster list,cl,cluster delete,cd,cr,cremove,image,i,image list,il,instancetype,config} ... Cluster-on-demand by Bright Computing positional arguments: {cluster,c,cluster create,cc,cluster list,cl,cluster delete,cd,cr,cremove,image,i,image list, il,instancetype,config} cluster (c) Manage clusters cluster create (cc) Create a new cluster cluster list (cl) List all of the recognized clusters (VPCs) cluster delete (cd, cr, cremove) Delete all resources in a cluster image (i) Manage images image list (il) List available Bright head node images instancetype Instance types config Configuration operations optional arguments: -h, --help show this help message and exit --config CONFIG, -c CONFIG --no-system-config -v, -vv, -vvv --show-configuration --log-file LOG_FILE --version ``` #### 2.8.1.2 Command Line Help At Other Levels The `--help` option can also be used to get the available options for a specific top-level command. For example: `cm-cod-aws cluster --help` The `--help` option can also be used to get the available options for a specific sub-level command. For example: `cm-cod-aws cluster create --help`. #### 2.8.1.3 Command Line Parameter Explanations With `--explain` More detailed information about a specific parameter can be obtained using the `--explain flag`. The following example shows how to find out more about the `--nodes` parameter of the `cluster create` command of `cm-cod-aws`. The last parameter is specified as `nodes`, not `--nodes`. **Example** ```bash [root@cod-client ~]# cm-cod-aws cluster create --explain nodes NAME nodes (cm-cod-aws cluster create --nodes | -n ) DESCRIPTION The amount of cloud nodes to configure for a cluster. The nodes are not powered on automatically. Its default value is 5. NAMESPACES AND COMMANDS It is used in the following namespaces and commands -[cluster.create.nodes:nodes] '-[cluster.create:nodes] '-[aws.cluster.create:nodes] (command: cm-cod-aws cluster create) HOW TO CONFIGURE On the Command Line: cm-cod-aws cluster create --nodes 5 [..] cm-cod-aws cluster create -n 5 [..] As Environment Variable: COD_NODES=5 cm-cod-aws cluster create [..] In a Config File: [cluster.create.nodes] nodes=5 ``` The detailed information includes what the default for the parameter is (if applicable), how to specify the parameter as an environment variable, and how to include it in a configuration file. ### 2.8.2 Configuration Files By default, configuration files for setting up the COD are searched for in the following locations, and in the following sequence. Settings found earlier in the sequence are overwritten by settings later on in the sequence. * `/etc/cm-cluster-on-demand.ini` * `/etc/cm-cluster-on-demand.conf` * `/etc/cm-cluster-on-demand.d/*` * `~/.cm-cluster-on-demand.ini` * `~/.cm-cluster-on-demand.conf` * `~/.cm-cluster-on-demand/*` * a file location specified on the command line. For example, a file `mycodsettings` can be accessed using the `--config` option of the COD client: **Example** ```bash [fred@basecm11 ~]$ cm-cod-azure --config mycodsettings ``` Typically, the administrator sets up configuration options in one of the first 3 locations, and the regular user modifies the options or adds other options in one of the last 4 locations. The configuration files are formatted as `.ini` files. #### 2.8.2.1 Viewing The Configuration File Options A dump of the existing configuration can be viewed using the `config dump` command, for example: `cm-cod-azure config dump` To check what options have been applied, and their sequence, the log to STDOUT can be viewed if the `-v|--verbose` option has been applied. A list of configuration options for a command can be seen with the `--show-configuration` option, for example (output elided): **Example** ```bash [fred@basecm11 ~]$ cm-cod-azure cluster create --show-configuration +---------------------------------+--------------------------------------------------------------------+ | option | value | +---------------------------------+--------------------------------------------------------------------+ | accept_eula | False (default) | | access_validation | True (default) | | advanced_help | False (default) | | ask_to_confirm_cluster_creation | True (default) | | azure_client_id | '12345678-1234-1234-1234-123456789012' (file: /etc/.../config.ini) | | azure_client_secret | XXXXXXXXXXXXXXX (file: /etc/.../config.ini) # use ... to uncover | | azure_location | 'westeurope' (file: /etc/cm-cluster-on-demand.d/config.ini) | | azure_subscription_id | '12345678-1234-1234-1234-123456789012' (file: /etc/.../config.ini) | | azure_tenant_id | '12345678-1234-1234-1234-123456789012' (file: /etc/.../config.ini) | | cluster_password | None (default) | ... ``` The table also indicates from which file a specific setting was taken. Arguments to the `cm-cod-{aws,azure}` commands override the equivalent configuration file settings. This means that the configuration file settings of a working configuration can be used as a default template, and modifications to the template can conveniently be carried out using the command line. #### 2.8.2.2 Setting Configuration File Options And Corresponding Arguments The configuration files are formatted as `.ini` files, which can contain multiple sections. The `--explain` flag can be used to find out how a certain command line parameter could be defined in a configuration file. Thus in the earlier example, in the case of the `cm-cod-aws cluster create --node` parameters, defining it as a configuration file would end up with a file looking like this: ```bash [cluster.create.nodes] nodes=5 ``` ## 2.9 Using The AWS EC2 Management Console The recommended way of managing COD is using Base View with its public IP address, or by using `cmsh` with the head node (section [2.9.5](#295-security-group-configuration-to-allow-access-to-the-head-node-using-cmsh-or-base-view)). This section (section [2.9](#29-using-the-aws-ec2-management-console)) describes how the Amazon management console can also be used, to manage some AWS aspects of the instance. A login is possible from [https://console.aws.amazon.com/console/](https://console.aws.amazon.com/console/) by using the e-mail address and password of the associated AWS root account, or AWS IAM user account. The head node instance can be watched and managed without BCM in the following ways. ### 2.9.1 Status Checking Using Instance Selection From Instances List Clicking the `Instances` menu resource item from the navigation pane opens up the “`Instances`” pane. This lists instances belonging to the account owner. An instance can be marked by ticking its checkbox. Information for the selected instance is then displayed in the lower main pane (figure [2.1](#fig-amazon-ec2-instances-list-selection)). ![The EC2 Instances List](/base-command-manager/product-manuals/_fern-img/5e5bfa141f9aa8357f5e8e39db3cdbb3dc5047ae254d742899293638029dff1e.webp) *Figure 2.1: The EC2 Instances List* System (Amazon machine infrastructure) and instance (instance running under the infrastructure) reachabilities are similarly shown under the neighboring “`Status Checks`” tab (figure [2.2](#fig-amazon-ec2-instances-list-system-instance-reachabilities)). ![Reachability Status For An EC2 Instance](/base-command-manager/product-manuals/_fern-img/5dc59b5b2bc600a17dd58849ee6730666f2278a6ce970579755701462487699e.webp) *Figure 2.2: Reachability Status For An EC2 Instance* ### 2.9.2 Acting On An Instance From The AWS EC2 Management Console An instance can be marked by clicking on it. Clicking the `Actions` button near the top of the main center pane, or equivalently from a right-mouse-button click in the pane, brings up a menu of possible actions. These actions can be executed on the marked instance, and include the options to `Start`, `Stop` or `Terminate` the instance. ### 2.9.3 Connecting To An Instance From The AWS EC2 Management Console A marked and running instance can have an SSH connection made to it. As in the Azure case, for most users this means running `ssh` to the public IP address as suggested at the end of the AWS COD cluster creation run (section [2.5.2](#252-cluster-creation-run-with-cm-cod-aws)). This is assuming the private ssh key that was generated by the `ssh-keygen` command in the container is used for the ssh connection, which may mean that copying it out of the container is needed. ### 2.9.4 Viewing The Head Node Console The head node takes about 2 minutes to start up. If, on following the instructions, an SSH connection cannot be made, then it can be worth checking the head node system log to check if the head node has started up correctly. The log is displayed on right-clicking on the “`Actions`” button, selecting the `Instance Settings` sub-menu, and selecting the “`Get System Log`” menu item (figure [2.3](#fig-amazon-ec2-instances-selection-action-system-log)). A successful start of the system generates a log with a tail similar to that of figure [2.3](#fig-amazon-ec2-instances-selection-action-system-log). ![System Log Of The Checkboxed Instance](/base-command-manager/product-manuals/_fern-img/2cfa975d9f3bf4ad281874db602c7fd22fc90f5266cba33dca1ac8b66f7fc4f4.webp) *Figure 2.3: System Log Of The Checkboxed Instance* A screenshot of the instance is also possible by right-clicking on the selected instance, then following the navigation path `Instance Settings` > `Get Instance Screenshot`. ### 2.9.5 Security Group Configuration To Allow Access To The Head Node Using `cmsh` Or Base View Amazon provides a security group to each instance, which controls network access according to the `inbound_rule` setting configured at cluster creation (section [2.2.5](#225-mandatory-inbound-rules)); the `inbound_network_acl_rule` setting configures the subnet’s network ACL separately. A new security group can be configured, or an existing one modified, using the `Edit details` button in figure [2.4](#fig-Security-Group-Network-And-Port-Access-Restriction). Security groups can also be accessed from the navigation menu on the left side of the EC2 Management Console. #### 2.9.5.1 COD With AWS: Access With Base View: The security group defined by Amazon for the head node can be modified by the administrator to allow remote connections to CMDaemon running on the head node (figure [2.4](#fig-Security-Group-Network-And-Port-Access-Restriction)). ![Security Group Network And Port Access Restriction](/base-command-manager/product-manuals/_fern-img/fa8025f66d4a3683d9b552b01e99506785287a5293e3189ec9982b4d28c4f3e9.webp) *Figure 2.4: Security Group Network And Port Access Restriction* * To allow only a specific network block to access the instance, the network from which remote connections are allowed can be specified in CIDR format. * If the cluster was created with an inbound rule opening port 8081, that allows Base View ([section 2.4 of the *Administrator Manual*](/admin/clustermanagement#24-base-view-gui)) to connect to the head node. This is because the Base View back end, which is CMDaemon, communicatesusing port 8081. #### 2.9.5.2 COD With AWS: Access With A Local `cmsh`: If the cluster was created with an inbound rule opening port 22, the security group created by Amazon allows inbound SSH connections from outside the cloud to the instance running in the cloud, even if the incoming port 8081 is blocked. Launching a `cmsh` session within an SSH connection running to the head node is therefore possible, and should work without lag. ## 2.10 Using The Azure Dashboard For Azure, as is the case for AWS, the cluster is normally expected to be managed using Base View or `cmsh`. Sometimes when carrying out some special configurations, there may be a need to manage the cluster objects directly using the portal at [https://portal.azure.com](https://portal.azure.com). This requires an Azure login and password, which opens up the Azure dashboard, and allows the objects to be viewed and managed. For example, the virtual machines of the COD can be viewed and managed using the `Virtual machines` resource (figure [2.5](#fig-azure-virtual-machines)), and the virtual networks associated with the COD nodes can similarly be viewed and managed using the `Virtual networks` resource. ![Viewing Virtual Machines In A Cluster On Demand In Azure](/base-command-manager/product-manuals/_fern-img/bdfa2b03cd84f8bdde614548b296730c1592306d62470a0d222c2fb5057f2ea2.webp) *Figure 2.5: Viewing Virtual Machines In A Cluster On Demand In Azure* Some items that may be useful when accessing the portal: * An overview of the COD head node is possible using the navigation path: `Home` > `Virtual machines` > \<*head node*> > `Overview` * Boot process diagnostics can be viewed using the navigation path: `Home` > `Virtual machines` > \<*head node*> > `Boot diagnostics` > `Serial log` * A serial console can be accessed the navigation path: `Home` > `Virtual machines` > \<*head node*> > `Serial console` The file `/etc/ssh/sshd_config` should have the parameter `ChallengeResponseAuthentication` set to `yes`, and the service restarted, for serial console login to work. ## 2.11 Using the OCI Console Dashboards For OCI, the cluster is expected to be managed using `cmsh`. Sometimes when the cluster are corrupted or it needs special configurations, cluster objects can be directly controlled using the portal at [https://www.oracle.com/cloud/](https://www.oracle.com/cloud/). This requires cloud account, or SSO to sign in. For example, all the instances can be viewed using `Left Navigation` > `Compute` > `Instances` (figure [2.6](#fig-oci-list-all-instances)) ![Viewing All Instances In A Cluster On Demand In OCI](/base-command-manager/product-manuals/_fern-img/b72dd8f8180769fd4150f3f5cbfee9a2116158212ff46d4a637baaa5e3a42722.webp) *Figure 2.6: Viewing All Instances In A Cluster On Demand In OCI* ## 2.12 COD: Cloud Node Start-up Cloud nodes must be explicitly started up. This is done by powering them up, assuming the associated cloud node objects exist. The cloud node objects are typically specified in the `cm-cod-aws` or `cm-cod-azure` script which is run in the Docker instance. In the preceding example the cloud node objects are `cnode001` and `cnode002`. However, more cloud node objects can be created if needed after the scripts have run. The maximum number that may be created is set by the license purchased. Large numbers of cloud node objects can be created with BCM as follows: * In Base View they can conveniently be cloned using the navigation path: `Devices` > `Cloud Nodes` > `<`*cloud node* > `Clone` > \<*number*> * In `cmsh` a large number of cloud node objects can conveniently be created with the “`foreach --clone`” command instead, as described in section [4.2](/cloudbursting/clicloud#42-launching-the-cloud-nodes). After creation, an individual cloud node, \<*cloud node*>, can be powered up from within Base View using the navigation path: `Devices` > `Cloud Nodes` > `<`*cloud node* > `Power` > `On` As with regular non-cloud nodes, cloud nodes can also be powered up from within the `device` mode of `cmsh`. The initial power status ([section 4.1 of the *Administrator Manual*](/admin/power#41-configuring-power-parameters)) of cloud nodes is `FAILED`, because they cannot be communicated with. As they start up, their power status changes to `OFF`, and then to `ON`. Some time after that they are connected to the cluster and ready for use. The device status (as opposed to the power status) remains `DOWN` until it is ready for use, at which point it switches to `UP`: **Example** ```bash [head1->device]% power status cloud ................. [ FAILED ] cnode001 (Cloud instance ID not set) cloud ................. [ FAILED ] cnode002 (Cloud instance ID not set) No power control ...... [ UNKNOWN ] head1 [head1->device]% power on -n cnode001 cloud ................. [ ON ] cnode001 [head1->device]% power status cloud ................. [ OFF ] cnode001 (pending) cloud ................. [ FAILED ] cnode002 (Cloud instance ID not set) No power control ...... [ UNKNOWN ] head1 [head1->device]% power on -n cnode002 cloud ................. [ ON ] cnode002 [head1->device]% power status cloud ................. [ ON ] cnode001 (running) cloud ................. [ OFF ] cnode002 (pending) No power control ...... [ UNKNOWN ] head1 [head1->device]% !ping -c1 cnode001 ping: unknown host cnode001 [head1->device]% status head1 .................... [ UP ] node001 .................. [ UP ] node002 .................. [ DOWN ] [head1->device]% !ping -c1 cnode001 PING cnode001.cm.cluster (10.234.226.155) 56(84) bytes of data. 64 bytes from cnode001.cm.cluster (10.234.226.155): icmp_seq=1 ttl=63 t\ ime=3.94 ms ``` Multiple cloud nodes can be powered up at a time in `cmsh` with the “`power on`” command using ranges and other options ([section 4.2.3 of the *Administrator Manual*](/admin/power#423-power-operations-through-cmsh)). ### 2.12.1 COD: IP Addresses In The Cloud * The IP addresses assigned to cloud nodes on powering them up are arbitrarily scattered over the 10.0.0.0/8 network * No pattern should therefore be relied upon in the addressing scheme of cloud nodes * Shutting down and starting up head and regular cloud nodes can cause their IP address to change. * However, BCM managing the nodes means that a regular cloud node re-establishes its connection to the cluster when it comes up, and will have the same node name as before. ## 2.13 COD With AWS: Optimizing AWS For High Performance Computing (HPC) Optimization of cloud nodes for HPC in AWS is discussed in section [3.5](/cloudbursting/cexcloud#35-cluster-extension-with-aws-optimizing-aws-for-high-performance-computing-hpc). ## 2.14 COD And High Availability ### 2.14.1 Introduction A COD head node, like a physical on-premises head node, can crash or somehow become non-operational. The possible reasons for this are pretty much the same in the case of a COD head node or a physical head node. The solution to reduce downtime due to these reasons is also the same: that is, to provide high availability (HA). HA has always been available as a feature for BCM running on on-premises clusters ([Chapter 15 of the *Administrator Manual*](/admin/ha)). Since 9.2, HA is also available for edge directors ([section 2.1.1.1 of the *Edge Manual*](/edge/edge#2111-running-cm-edge-setup-in-interactive-mode)) and for COD (this section [2.14](#214-cod-and-high-availability)). The cluster administrator should have some familiarity with how HA for physical head nodes is implemented ([Chapter 15 of the *Administrator Manual*](/admin/ha)) before implementing it on a COD. HA on COD is implemented very similarly to on a physical cluster, with a passive COD head node that takes over from a failing active COD head node. COD with HA is deployed in a similar manner for AWS, Azure, and OCI. The COD head nodes use shared storage for `cm/shared` and `/home`, just like the physical head nodes do. For COD HA head nodes, the shared storage needs to be a special filesystem in the cloud, just like the physical head HA nodes use a special filesystem. Two shared IP addresses are also allocated––public and private. These shared IP addresses always point to the active head node, and are analogous to the external and internal IP address in the case of the physical head HA nodes ([figure 15.1 of the *Administrator Manual*](/admin/ha)). ### 2.14.2 Deployment #### 2.14.2.1 COD HA Deployment Overview A COD HA cluster is created by first creating a standard (that is: non-HA) COD cluster using `cm-cod-aws` (section [2.3](#23-cod-using-the-cod-command-line-tool)), `cm-doc-azure` (section [2.4](#24-cod-on-azure-using-the-command-line)), or `cm-cod-oci` (section [2.6](#26-cod-on-oci-using-the-command-line)). For AWS the following may be run: **Example** ```bash [root@basecm11 ~]# cm-cod-aws cluster create --version 11 --name basecm11-ha ``` The procedures for Azure (with `cm-cod-azure`) and OCI (with `cm-cod-oci`) are very similar. The standard COD cluster, which here has arbitrarily been given the name `basecm11-ha` in advance, can now be converted to an HA COD cluster. For AWS, after the deployment of the converted cluster is complete, the original head node automatically gets the suffix `-a`, and the new head node that is brought up gets suffix `-b`. #### 2.14.2.2 Configuration For Deployment The conversion is done by running the `cm-cloud-ha-setup` wizard, whch starts up a TUI for configuring the HA deployment. The `Deploy` item in the main menu starts the configuration (figure [2.7](#fig-cod-ha-cm-cloud-ha-setup-deploy)): ![Cluster On Demand TUI Wizard: Deploy Selection](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/e012630257c1872c074c7fe839422b22508dc88e204e20e31893fd9863e82c95/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/fig:cod-ha-cm-cloud-ha-setup-deploy.png) *Figure 2.7: Cluster On Demand TUI Wizard: Deploy Selection* The deployment process has the cluster administrator: * provide the product key, so that the MAC address of the secondary COD head node can be added to the license (figure [2.8](#fig-cod-ha-cm-cloud-ha-setup-prodkey1)): ![Cluster On Demand TUI wizard: deploy selection](/base-command-manager/product-manuals/_fern-img/7814ef6f08dd999163c641f31b870adeb211a60f1ffed382d6d94daa24d18f8e.webp) *Figure 2.8: Cluster On Demand TUI wizard: deploy selection* * decide using using a screen on whether to auto-allocate a private IP address to be shared between the head nodes, or whether to specify the private address (figure [2.9](#fig-cod-ha-cm-cloud-ha-setup-auto-allocate)): ![Cluster On Demand TUI wizard: decide on how private IP address is a created](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/b5948183c55867ae5156408adedae4dda45ad6d1260f5dfce260c20f0c28a502/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/fig:cod-ha-cm-cloud-ha-setup-auto-allocate.png) *Figure 2.9: Cluster On Demand TUI wizard: decide on how private IP address is a created* * if specifying the private address is chosen, then the private IP address can be set (figure [2.10](#fig-cod-ha-cm-cloud-ha-setup-private-public-ip)): ![Cluster On Demand TUI wizard: private IP address allocation](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/1fceab81a2b4a9bb141f972969386e83377a9cc5fbbe8a4c43878c36e482ea8e/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/fig:cod-ha-cm-cloud-ha-setup-private-public-ip.png) *Figure 2.10: Cluster On Demand TUI wizard: private IP address allocation* * decide on where to deploy shared storage (figure [2.11](#fig-cod-ha-cm-cloud-ha-setup-shared-storage)): ![Cluster On Demand TUI wizard: decide on where the shared storage should be](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/8aa6da0944c10d3208ed5a9716fad70ffee6e4ba311fb7b7f571b50286f208cd/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/fig:cod-ha-cm-cloud-ha-setup-shared-storage.png) *Figure 2.11: Cluster On Demand TUI wizard: decide on where the shared storage should be* * If `yes` is selected, then the shared storage is created by the wizard creating an AWS EFS instance and sharing the `/home` and `/cm/shared` partitions among the cluster nodes using NFS * If `no` is selected, then the shared storage should be specified manually, or a preconfigured storage can be used * decide on how to set up a NAT gateway (figure [2.12](#fig-cod-ha-cm-cloud-ha-setup-nat-gateway)): ![Cluster On Demand TUI wizard: decide on how to set up a NAT gateway](/base-command-manager/product-manuals/_fern-img/11eb5cfe68859391f329614f89f8a9fded5fe3c4cf94ccf53d191595c1b13c7e.webp) *Figure 2.12: Cluster On Demand TUI wizard: decide on how to set up a NAT gateway* * decide on setting up the shared IP address as public or private (figure [2.13](#fig-cod-ha-cm-cloud-ha-setup-shared-ip)): ![Cluster On Demand TUI wizard: decide on how to set up the shared IP address](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/143b27314a5e8d7d87b5bb868a1ef1a8abe286864f955f18da02891f637822bb/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/fig:cod-ha-cm-cloud-ha-setup-shared-ip.png) *Figure 2.13: Cluster On Demand TUI wizard: decide on how to set up the shared IP address* * save the configuration and deploy the conversion (figure [2.14](#fig-cod-ha-cm-cloud-ha-setup-save-and-deploy)): ![Cluster On Demand TUI wizard: saving the configuration, as part of deployment](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/df1d289fbb271c2e32e395eeb2cd729e5ec648b680a209e9a9fe0f0270c2c9b4/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/fig:cod-ha-cm-cloud-ha-setup-save-and-deploy.png) *Figure 2.14: Cluster On Demand TUI wizard: saving the configuration, as part of deployment* #### 2.14.2.3 Conversion To HA The deployment that converts the cluster to HA then takes place by bringing up a second HA node in the cloud. Deployment typically takes about 50 minutes. The progress can be followed on the console, and in the logs `/var/log/cmha.log` and `/var/log/failoversetup.log`. If the deployment is not yet complete, then the `cmha setup` command indicates that with, for example: ```bash root@basecm11-ha-a:~# cmha status No HA status available. Unable to get passive head node, is failover setup complete? ``` or later on, during the `Finalizing HA stage`, with: ```bash root@basecm11-ha-a:~# cmha status Unable to connect to cluster Internal error, unable to connect to cluster: [Errno 111] Connection refused Internal error, unable to connect at local cluster: [Errno 111] Connection refused ``` ### 2.14.3 COD HA Checks And Recovery #### 2.14.3.1 COD HA Head Status Check Once the deployment using `cm-cloud-ha-setup` is complete, the HA status of the cluster can be checked: ```bash [root@basecm11-ha-b ~]# cmha status Node Status: running in active mode basecm11-ha-b* -> basecm11-ha-a mysql [ OK ] ping [ OK ] status [ OK ] basecm11-ha-a -> basecm11-ha-b* mysql [ OK ] ping [ OK ] status [ OK ] [root@basecm11-ha-b ~]# ``` #### 2.14.3.2 COD Failover Session If the active head node fails, then the passive head node is activated automatically by reassigning the shared public and private IP addresses to the passive head node, and marking the passive head node as active. This process is called failover and can be triggered automatically or manually ([section 15.1.7 of the *Administrator Manual*](/admin/ha#1517-automatic-vs-manual-failover)). A manual failover can be carried out with `cmha makeactive`: ```bash [root@basecm11-ha-b ~]# cmha makeactive =========================================================================== This is the passive head node. Please confirm that this node should become the active head node. After this operation is complete, the HA status of the head nodes will be as follows: basecm11-ha-b will become active head node (current state: passive) basecm11-ha-a will become passive head node (current state: active) =========================================================================== Continue(c)/Exit(e)? c Initiating failover.............................. [ OK ] basecm11-ha-b is now active head node, makeactive successful [root@basecm11-ha-b ~]# cmha status Node Status: running in active mode basecm11-ha-b* -> basecm11-ha-a mysql [ OK ] ping [ OK ] status [ OK ] basecm11-ha-a -> basecm11-ha-b* mysql [ OK ] ping [ OK ] status [ OK ] [root@basecm11-ha-b ~]# ``` Managing HA and manual failover is described in more detail in [section 15.4 of the *Administrator Manual*](/admin/ha#154-managing-ha). ### 2.14.4 Recovery Of The Passive COD Head Unlike with physical HA head nodes, the recovery of a passive head node in the cloud is automated. Recovery can be carried out by running `cm-cloud-ha-setup` wizard, and selecting the `Recovery` item in the main menu (figure [2.15](#fig-cod-ha-cm-cloud-ha-setup-recovery)): ![Cluster On Demand TUI Wizard: Recovery Selection](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/027c773fead2bb00ece8d6a4ec772f15ba1b772b082e515c00ce7f9ea4874bd5/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-wPoXzKKBLZez/latextofern/fern/assets/figures/fig:cod-ha-cm-cloud-ha-setup-recovery.png) *Figure 2.15: Cluster On Demand TUI Wizard: Recovery Selection* The product key is needed to license the new head node (figure [2.16](#fig-cod-ha-cm-cloud-ha-setup-prodkey2)): ![Cluster On Demand TUI Wizard: Product Key Selection](/base-command-manager/product-manuals/_fern-img/7814ef6f08dd999163c641f31b870adeb211a60f1ffed382d6d94daa24d18f8e.webp) *Figure 2.16: Cluster On Demand TUI Wizard: Product Key Selection* The recovery process warns that all resources associated with the old head node are to be deleted (virtual machine, disks, IP addresses) (figure [2.17](#fig-cod-ha-cm-cloud-ha-setup-cleanup)): ![Cluster On Demand TUI Wizard: Cleanup](/base-command-manager/product-manuals/_fern-img/a7ca2678a489bf02125bb9be1e56f6730ebed13bdfe40fb839ee57bc6434e98c.webp) *Figure 2.17: Cluster On Demand TUI Wizard: Cleanup* It then clones the active head node, configuring the clone to become the new passive head node, and the cluster then resumes normal HA operation. When recovery is completed, the new passive head node elastic IP for the cluster called \<*cluster_name*> can be obtained by running: `cm-cod-aws cluster list [` \<*cluster_name*>`]` The `cm-cod-aws` command in the preceding text can be replaced ty `cm-cod-azure` or `cm-cod-oci` in order to get a similar listing for COD Azure or COD OCI clusters. The output from `cmha status` can be used to confirm recovery is complete. ### 2.14.5 COD HA Deletion The COD clusters for AWS listing might show something like (some output truncated): ```bash [root@basecm11 ~]# cm-cod-aws cluster list 20:10:08: INFO: Listing clusters in region eu-west-1 +--------------+-----------------------+-------------------------+---------------------+---------+----------- | Cluster Name | VPC ID | Head node ID | Cluster IP | State | Type +--------------+-----------------------+-------------------------+---------------------+---------+----------- | basecm11-ha | vpc-01d547e708c9eec9c | i-0224f8b38fe0b8658 (A) | 54.155.161.116 (A) | running | t3.medium | | | i-0f906293daa687d20 (B) | 52.213.160.8 (B) | | | | | | 52.211.148.112 (HA) | | +--------------+-----------------------+-------------------------+---------------------+---------+----------- ``` The COD cluster can be specified for removal with the `delete` option. For example, with AWS: **Example** ```bash [root@basecm11 ~]# cm-cod-aws cluster delete basecm11-ha 20:11:43: WARNING: Some clusters have a Cloud HA EFS storage. It will also be deleted 20:11:43: INFO: This will destroy resources associated with clusters: 'basecm11-ha'. Would you like to continue? 20:11:43: INFO: Proceed? [yes/no] yes 20:11:53: INFO: Deleting EFS fs-0dee13aa1d1a2d337... 20:12:16: INFO: Finding instances for cluster 'basecm11-ha'... 20:12:16: INFO: Issuing termination requests for 2 instances for cluster 'basecm11-ha'... 20:12:17: INFO: Waiting until instances of cluster 'basecm11-ha' are terminated... ... 20:16:40: INFO: Done destroying cluster 'basecm11-ha' resources in VPC 'on-demand basecm11-ha' ```