Cluster On Demand Cloudbursting With Azure, AWS, Or OCI

2.1 Introduction

Cluster On Demand (COD) cloudbursting is when a separate cluster is started up in a cloud, with the cluster head node that manages the cluster also in that cloud. A COD cluster is regarded as an independent virtual cluster (sometimes described as a ’pure’ cloud cluster), and not an extension of an existing physical cluster. This chapter describes how COD can run in:

• Azure (COD-Azure)

• AWS (COD-AWS)

• Oracle Cloud Infrastructure (COD-OCI)

When the requirements outlined in the sections that follow have been met, COD clusters can be managed (created, deleted, etc.) by using the COD command line tool via a Docker image or a Python package (all cloud platforms) (section 2.3). After a COD head node has been launched in the cloud, then compute nodes are managed by using the various management interfaces provided by NVIDIA Base Command Manager running on the head node (section 2.11).

2.2 Requirements For COD Cloudbursting

The high level requirements for COD are:

• a BCM product key. This key is later activated when the license is installed (Chapter 4 of the Installation Manual) on the head node.

• Credentials for the cloud environment in which the COD cluster is created. This is covered in the following sections.

2.2.1 Credentials For Azure

To use Azure, an Azure account subscription is needed from Microsoft. COD cloudbursting requires the following associated Azure credentials to launch:

• tenant ID

• subscription ID

• Client ID

• Client Secret

A CLI-centric way to obtain these credentials requires logging into the Azure web portal using an account that has sufficient privileges. The Azure web bash console is then opened, and the subscription ID can then be listed for the account with:

Example

azure@Azure:~$ az account list -o table
Name          CloudName    SubscriptionId                        State    IsDefault
------------  -----------  ------------------------------------  -------  -----------
anne          AzureCloud   23748c3e-507b-11e9-a994-fa163e9854eb  Enabled  False
nerds         AzureCloud   b9e22a88-507a-11e9-9352-fa163e9854eb  Enabled  False

A service principal (sp) is now created for role-based access control create-for-rbac in Active Directory (ad), and the remaining 3 credentials can then be seen:

Example

azure@Azure:~$ az ad sp create-for-rbac --name my-temp-service-principal-for-fred
Changing "my-temp-service-principal-for-fred" to a valid URI of
"http://my-temp-service-principal-for-fred", which is the required format used for service
principal names
Retrying role assignment creation: 1/36
Retrying role assignment creation: 2/36
{
  "appId": "dcf8151e-507a-11e9-a104-fa163e9854eb",    ## "Client ID"
  "displayName": "my-temp-service-principal-for-fred",
  "name": "http://my-temp-service-principal-for-fred",
  "password": "bc9f571e-fe4c-43d5-909d-4bc66796eb41",   ## "Client secret"
  "tenant": "8cb88849-6e18-46d6-b0fa-551a47a31681"    ## Tenant ID
}

The newly-created service principal is added to the desired subscription as a contributor. The value of appID must be used as the value to the --assignee option. This gives the application sufficient permissions for the cluster to run:

Example

azure@Azure:~$ az role assignment create --assignee dcf8151e-507a-11e9-a104-fa163e9854eb --role\
 Contributor --subscription b9e22a88-507a-11e9-9352-fa163e9854eb
{
  "canDelegate": null,
  "id": "/subscriptions/b9e22a88-507a-11e9-9352-fa163e9854eb/providers/Microsoft.Authorization\
/roleAssignments/1094f75c-507b-11e9-ac8f-fa163e9854eb",
  "name": "1094f75c-507b-11e9-ac8f-fa163e9854eb",
  "principalId": "2f899334-507b-11e9-b312-fa163e9854eb",
  "roleDefinitionId": "/subscriptions/b9e22a88-507a-11e9-9352-fa163e9854eb/providers\
/Microsoft.Authorization/roleDefinitions/b24988ac-6180-42a0-ab88-20f7382dd24c",
  "scope": "/subscriptions/b9e22a88-507a-11e9-9352-fa163e9854eb",
  "type": "Microsoft.Authorization/roleAssignments"
}

The Microsoft documentation suggests that using the name instead of the subscription ID should also work. However in the version that was used at the time of writing (May 2019) this did not work, and an error 400 was displayed.

2.2.2 Credentials For AWS

To use AWS, an AWS subscription is needed from Amazon. COD cloudbursting requires the following associated AWS credentials for launch:

• Secret Access Key: Available only once, when first generated, as described at https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys

• Access Key ID: as described at https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys

• AWS Account ID, as described at https://docs.aws.amazon.com/general/latest/gr/acct-identifiers.html

• AWS Username: This can be either the AWS account root user (the e-mail address of the root user), or it can be an IAM username with sufficient permissions to launch the cluster.

2.2.3 Credentials For OCI

To use OCI, an OCI account subscription is needed from Oracle. COD cloudbursting requires the following OCI credentials to launch:

• API signing Key: This key can be either an uploaded public key from cod machine, or a downloaded private key, as described at https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm#two, in the section How to Generate an API Signing Key. After it is added, a configuration file is displayed.

• User’s OCID: This shows up in the configuration file with a format such as:
  user=ocid1.user.oc1.. <unique ID>
  where <unique ID> may have a format such as:

          aaaaaaaax5552rxvs6ar4qm7kur6jq2kmhrvzmk6coqwq6b7vxoclltr64ba

• Fingerprint: This shows up in the configuration file with a format such as:
  fingerprint=12:34:56:78:90:ab:cd:ef:12:34:56:78:90:ab:cd:ef

• Tenancy’s OCID: This shows up in the configuration file with a format such as:
  tenancy=ocid1.tenancy.oc1.. <unique ID>

• Compartment ID: In the OCI navigation menu, the compartment ID can be found via the navigation path:
  Identity & Security > Identity > Compartments
  Further information on finding the OCID of a compartment can be found at:

  https://docs.oracle.com/en-us/iaas/Content/GSG/Tasks/contactingsupport_topic-Finding_the_OCID_of_a_Compartment.htm.

To use Oracle’s oci-cli utility, the configuration file has to be placed in ~/.oci/config. For the COD client, the credentials should be added to the COD configuration file as described in 2.6.1.

2.2.4 Mandatory Inbound Rules

By default, a COD cluster is no longer created open to the world. Unless an inbound rule is configured for the cluster, cluster creation fails validation. The user must specify one of the following for the cluster:

• inbound_rule (and, for AWS, inbound_network_acl_rule) opening port 22 (SSH), or

• inbound_rule (and, for AWS, inbound_network_acl_rule) opening port 8081 (web management)

At least one of these two ports must be open. The port does not have to be open specifically to the IP address of the machine that runs the COD client. For example, creating a cluster on behalf of another user is a valid use case.

The value {CLIENT_IP} can be used in inbound_rule or inbound_network_acl_rule instead of a hardcoded IP address, and resolves to the public IP address of the machine running the COD client.

Examples for each cloud are given in the minimal configuration file examples in sections 2.4.1, 2.5.1, and 2.6.1.

2.3 COD Via The COD Command Line Tool

There are two options for running the COD command line tool on a standalone PC:

• The COD command line tool can be run from a Docker image:

  • The Docker instance that runs from the image can be hosted on a standalone PC that is not part of a BCM cluster, or it can be hosted on a BCM node.

  • Section 2.3.1 describes the procedure for installing and configuring COD on Docker on a BCM node in more detail.

• The COD command line tool can be run after installing a Python package in a local (virtual) environment:

  • The Python package can also be installed on a standalone PC or it can be installed on a BCM node.

  • Section 2.3.2 describes the procedure for installing and configuring the COD client using a Python package on a BCM in more detail.

Once the COD client is installed and configured, the next stage is to launch the COD inside the cloud service. There are three cloud services that a COD from Docker can be launched into. These are:

• AWS (Amazon Web Services), made available by Amazon. The CLI utility in the Docker instance that launches the COD is then cm-cod-aws (here).

• Azure, made available by Microsoft. The CLI utility in the Docker instance that launches the COD is then cm-cod-azure (here).

• OCI (Oracle Cloud Infrastructure). The CLI utility in the Docker instance that launches the COD is then cm-cod-oci (here).

2.3.1 COD Via Docker Image

The following steps are carried out to start up the head node and regular nodes of the COD:

• A Docker host is used to pull a BCM COD (Cluster On Demand) image from the Docker registry.

• The image is run in a Docker container.

• From within the instance of the image running in the container, a cloudbursting request for a COD is carried out to a cloud service provider.

• When the request completes successfully, a cluster that is running in the cloud is ready for use.

The COD Docker image is typically run from a standalone machine, such as, for example, a laptop that is not part of a BCM cluster. However, launching a COD from BCM is also possible, and can be carried out as follows:

• Docker is installed in BCM with cm-docker-setup (section 2.1 of the Containerization Manual), or using the Base View Docker setup wizard.

• The node that is to run the Docker image is used to pull the image to the node

  Example

  [root@basecm10 ~]# ssh node001
  [root@node001 ~]# module load docker
  [root@node001 ~]# docker pull brightcomputing/cod:latest
  ...
  cac15ba059ef: Pull complete
  Digest: sha256:19b263033090e1a70043989decdf3c3870d3def8c2e69b2a85ac293fd7d149ab
  Status: Downloaded newer image for brightcomputing/cod:latest

• The command line COD tools installed in the container image all have several command line arguments. Instead of the user having to specify many arguments on every invocation of a tool, the options can conveniently be set in configuration files. To provide access to such configuration files it is recommended that the home directory of the user running the tool is mounted into the container instance. It is also recommended to set up the following shell aliases for each of the tools:

  Example

    alias cm-cod-aws='docker run --rm -it --network host -v /root:/root brightcomputing/cod\
  :latest cm-cod-aws'
    alias cm-cod-azure='docker run --rm -it --network host -v /root:/root brightcomputing\
  /cod:latest cm-cod-azure'
    alias cm-cod-oci='docker run --rm -it --network host -v /root:/root brightcomputing\
  /cod:latest cm-cod-oci'

  In the preceding aliases:

  • the --rm option ensures that the container instance created to run a command is also cleaned up on completion

  • the -it flags are required because some commands may present interactive prompts to the user

  • the --network host option gives the container network access

  • the argument to the -v flag has the form of <source>:<destination>, where

    • <source> points to the home directory of the user invoking the command, and

    • <destination> is the path where the directory is mounted inside the running container instance. The path of <destination> should always be /root.

• Once a COD cluster has been created, it is typically accessed using SSH. It is strongly recommended to use key-based authentication instead of password-based authentication. Since the home directory for users is mounted into the container, any existing SSH key pairs (stored in ~/.ssh) can be used. If no key pair exists, then ssh-keygen can be used to create one. To ensure the highest level of compatibility it is recommended to use RSA as the key type and a minimum of 2048 bits.

2.3.2 COD Via A Python Package

Instead of running the COD client in a Docker container it is also possible to install and run the client directly on the host system in a Python virtual environment. The client can be run from BCM as follows:

• A virtual environment is created to install the COD client:

  [root@basecm10 ~]# ssh node001
  [root@node001 ~]# module load python3
  [root@node001 ~]# python3 -m venv .venvs/cod
  [root@node001 ~]# source .venvs/cod/bin/activate
  (cod) [root@node001 ~]#

• Next, the required COD clients are installed. Possible clients are:

  • cm-cluster-on-demand-aws

  • cm-cluster-on-demand-azure

  • cm-cluster-on-demand-oci

  For example, the COD client tool for AWS can be installed as follows:

  (cod) [root@node001 ~]# pip install cm-cluster-on-demand-aws
  ...
  Successfully installed boto3-1.7.75 botocore-1.10.84 cm-cluster-on-demand-aws-9.1 ...

  More than one client can be installed.

• After the COD client has been installed, it can be run whenever the virtual environment is activated, without having to install it again:

  [root@node001 ~]# source .venvs/cod/bin/activate
  (cod) [root@node001 ~]# cm-cod-aws
  usage: cm-cod-aws [-h] [--config CONFIG] [--no-system-config] [-v]
                    [--show-configuration] [--log-file LOG_FILE] [--version]
                    cluster,c,cluster create,cc,cluster list,cl,cluster
                    delete,cd,cr,cremove,image,i,image
                    list,il,instancetype,config ...
  cm-cod-aws: error: the following arguments are required: group

2.4 COD On Azure Via Command Line

Launching a COD inside Azure is carried out by running cm-cod-azure from either the Docker image or the Python package. The cm-cod-azure command can take configuration options from one or more configuration files, environment variables, or command line options. More details on this are given in section 2.7.

Typically some of the more static options are configured in a configuration file, while some of the more variable options are specified on the command line.

Online help is also available: for example, running cm-cod-azure cluster create -h displays a list of options and explanations for the cluster create command.

2.4.1 Minimal Configuration File For COD On Azure

It is recommended to create a minimal configuration in ~/cm-cluster-on-demand.d/config.ini

The credentials and product key in the following example are dummy credentials, and must be replaced with the appropriate values for the Azure instance used by the cluster administrator. The value for ssh_pub_key_path is the key for connecting to the head node, which can be generated using ssh-keygen -b 4096.

Example

[azure.credentials]
azure_subscription_id=5e519b1e-aff9-e839-bc3a-c5d87e9d0d5
azure_tenant_id=2c89c8dd-6b8b-5393-60f4-d876cbe188f
azure_client_id=afe13723-c80a-68e2-9cd0-e95a657106a
azure_client_secret=aiVohwi7OhJ6igim=
azure_location=westeurope
[azure.cluster.create]
inbound_rule={CLIENT_IP},22:tcp
[cluster.create.license]
license_product_key=123456-123456-123456-123456-123456
[cluster.create.password]
ssh_pub_key_path=/root/.ssh/id_rsa.pub

The inbound_rule setting shown opens SSH access to the machine running the COD client (section 2.2.4).

2.4.2 Cluster Creation Run With cm-cod-azure

An example of a cm-cod-azure command that can be run is then:

Example

cm-cod-azure cluster create \
    --head-node-type Standard_D1_v2 --head-node-root-volume-size 50 \
    --nodes 3 --node-type Standard_D1_v2 \
    --name testcluster

The BCM Azure integration makes use of several node-installer images which are published in the Azure marketplace. In order to use those images, Azure requires the administrator to accept the legal terms for those images. The code automatically detects if the terms were already accepted in the past, in which case it proceeds and creates the cluster. If the terms still need to be accepted, then several links to the legal terms are presented, along with a prompt to accept them. The output looks similar to the following (some output ellipsized):

Example

To use Azure VM images, you need to accept the following terms:
- License (bcmni-azure-9-2:bcm-ni-azure-9-2-v1): https://download.microsoft.com/download/F/D/8/FD8BA8F2...
- Privacy Policy (bcmni-azure-9-2:bcm-ni-azure-9-2-v1): https://www.brightcomputing.com/privacy-policy
- Marketplace Terms (bcmni-azure-9-2:bcm-ni-azure-9-2-v1): https://mpcprodsa.blob.core.windows.net/market...
- License (bcmni-azure-9-2-free:bcm-ni-azure-9-2-v2): https://download.microsoft.com/download/F/D/8/FD8BA...
- Privacy Policy (bcmni-azure-9-2-free:bcm-ni-azure-9-2-v2): https://www.brightcomputing.com/privacy-policy
- Marketplace Terms (bcmni-azure-9-2-free:bcm-ni-azure-9-2-v2): https://mpcprodsa.blob.core.windows.net/mark...
Accept (y/n)?

If all is well, then something similar to the following is displayed (some output elided):

Example

  [root@cod-client ~]# cm-cod-azure cluster create \
  --head-node-type Standard_D1_v2 --head-node-root-volume-size 50 \
  --nodes 3 --node-type Standard_D1_v2 \
  --name testcluster
  09:35:01:     INFO: No custom cluster password specified. Using a random password.
  Use --log-cluster-password to see it. If you are going to use the Bright View you have
  to change this password by using cm-change-passwd script (change root password of head
  node) or simply 'passwd root' command. If this cluster has a high-availability setup
  with 2 head nodes, be sure to change the password on both head nodes.
  09:35:05:     INFO: Temporary azure resources will be created in order to verify that
  the API credentials have the required permissions, this can take a few minutes. You can
  skip this step by specifying the following flag: '--skip-permission-verifications'
  09:35:05:     INFO: Cluster Create
  09:35:05:     INFO: -------------------------------------------------
  09:35:05:     INFO:         Cluster:  testcluster
  09:35:05:     INFO: -------------------------------------------------
  09:35:05:     INFO:      Image name:  bcm-cod-image-9.0-8
  09:35:05:     INFO:      Image date:  2020-08-21 15:10 (38d 18h ago)
  09:35:05:     INFO:  Package groups:  none
  09:35:05:     INFO:         Version:  9.0
  09:35:05:     INFO:      Head nodes:  1 (Standard_D1_v2)
  09:35:05:     INFO:           Nodes:  3 (Standard_D1_v2)
  09:35:05:     INFO:  Resource Group:  testcluster_cod_resource_group
  09:35:05:     INFO:          Region:  westeurope
  09:35:05:     INFO: -------------------------------------------------
  09:35:05:     INFO: Press ENTER to continue and create the cluster.
  09:35:05:     INFO: Press ctrl+c (or type 'a') to abort. Type 'i' for more info.
  09:35:45:     INFO: Credentials are valid and have read/write authorizations.
  09:38:17:     INFO: ## Progress: 2
  09:38:17:     INFO: #### stage: Creating resource group testcluster_cod_resource_group
  09:38:18:     INFO: Creating storage account testclusterstoragerg68u3
  09:38:36:     INFO: ## Progress: 7
  09:38:36:     INFO: #### stage: Copying head node image from
  https://brightimages.blob.core.windows.net/images/bcm-cod-image-9.0-8.vhd
  09:38:36:     INFO: Copying https://brightimages.blob.core.windows.net/images/bcm-cod-image-9.0-8.vhd
  to testclusterstoragerg68u3/images/testcluster-head-node-os-disk.vhd
  09:38:37:     INFO: ## Progress: 12.0
  09:38:37:     INFO: #### stage: Server side copy
  09:38:59:     INFO: ## Progress: 12.13
  09:38:59:     INFO: #### stage: Server side copy
  09:39:03:     INFO: ## Progress: 12.26
  ...
  09:49:40:     INFO: #### stage: Server side copy
  09:49:41:     INFO: ## Progress: 51.85
  09:49:41:     INFO: #### stage: Server side copy
  09:49:43:     INFO: ## Progress: 51.98
  09:49:43:     INFO: #### stage: Server side copy
  09:49:45:     INFO: Elapsed: 11:08 min
  09:49:45:     INFO: Creating Azure Image resource 'testcluster-head-node-os-disk' from blob
  https://testclusterstoragerg68u3.blob.core.windows.net/images/testcluster-head-node-os-disk.vhd
  09:50:21:     INFO: Deleting blob testclusterstoragerg68u3/images/testcluster-head-node-os-disk.vhd
  09:50:21:     INFO: ## Progress: 5
  09:50:21:     INFO: #### stage: Building deployment template
  09:50:21:     INFO: generating cloud-init script
  09:50:21:     INFO: ## Progress: 85
  09:50:21:     INFO: #### stage: Creating and deploying Head node
  09:52:22:     INFO: Head node IP: 51.145.169.45
  09:52:22:     INFO: Waiting for sshd to start (ssh root@51.145.169.45).
  09:52:22:     INFO: Waiting for CMDaemon to start.
  09:55:05:     INFO: Waiting for CMDaemon to initialize.
  09:56:07:     INFO: ## Progress: 100
  09:56:07:     INFO: #### stage: Deployment finished successfully.
  09:56:07:     INFO: -----------------------------------------------------
  09:56:07:     INFO:       Cluster:  testcluster
  09:56:07:     INFO: -----------------------------------------------------
  09:56:07:     INFO:  Head node ID:  7f03d713-71ad-4eb6-88ff-045f82a41990
  09:56:07:     INFO:     Public IP:  51.145.169.45
  09:56:07:     INFO:      Key path:  /root/.ssh/id_rsa.pub
  09:56:07:     INFO: -----------------------------------------------------
  [root@cod-client ~]#

The cluster can now be logged into by using the SSH keys generated earlier, and using the public IP address shown in the last few lines of the preceding output. The regular nodes running on Azure are configured for use, but are not powered on. They can be powered on using the cmsh or Base View front ends, just as in a regular cluster.

From cmsh, the first cloud node can be powered on with, for example:

[testcluster->device]% power on cnode001

More details can be found in section 2.11.

2.4.3 Listing Clusters With cm-cod-azure

The clusters that are running can be listed with the cluster list command option:

Example

  [root@cod-client ~]# cm-cod-azure cluster list
  +--------------+-----------------------+---------------+------------+-...
  | Cluster Name | Head node name        | Public IP     | Location   | ...
  +--------------+-----------------------+---------------+------------+-...
  | testcluster  | testcluster-head-node | 51.145.169.45 | westeurope | ...
  +--------------+-----------------------+---------------+------------+-...

Since all of the credentials are specified in the configuration file, no further command line options are needed.

2.4.4 Cluster Removal With cm-cod-azure

A cluster can be removed with a command in the form of: cluster delete <cluster name>:

Example

  [root@cod-client ~]# cm-cod-azure cluster delete testcluster
  10:10:47:     INFO: Deleting resource group(s): testcluster_cod_resource_group
  10:10:47:     INFO: Proceed? [yes/no]
  y
  10:10:52:     INFO: Started deleting resources for Resource Group 'testcluster_cod_resource_group'
  [root@cod-client ~]#

2.5 COD On AWS Via Command Line

Launching a COD inside AWS is carried by running cm-cod-aws from either the Docker image or the Python package.

The cm-cod-aws command can take configuration options from one or more configuration files, environment variables, or command line options. More details on this are given in section 2.7.

Typically some of the more static options are configured in a configuration file, while some of the more variable options are specified on the command line.

Online help is also available. For example, running cm-cod-aws cluster create -h displays a list of options and explanations for the cluster create command.

2.5.1 Minimal Configuration File For COD On AWS

Creation of a minimal configuration file in ~/cm-cluster-on-demand.d/config.ini is recommended, with the following contents:

Example

  [aws.credentials]
  aws_access_key_id=AKIAJAICHAZI7OHH1EEGO
  aws_secret_key=Oocei6eiieshahG7IiJe9Quoud0oo
  aws_region=eu-west-1
  [aws.cluster.create]
  inbound_rule={CLIENT_IP},22:tcp
  inbound_network_acl_rule={CLIENT_IP},22:tcp,ALLOW,100
  [cluster.create.license]
  license_product_key=123456-123456-123456-123456-123456
  [cluster.create.password]
  ssh_pub_key_path=/root/.ssh/id_rsa.pub

The inbound_rule and inbound_network_acl_rule settings shown open SSH access to the machine running the COD client (section 2.2.4).

The example credentials and product key should be replaced with the appropriate values. The value for ssh_pub_key_path is the key for connecting to the head node, which can be generated using ssh-keygen -b 4096.

2.5.2 Cluster Creation Run With cm-cod-aws

An example of a cm-cod-aws command that can be run is then:

Example

cm-cod-aws cluster create \
    --head-node-type t3.medium --head-node-root-volume-size 50 \
    --nodes 3 --node-type t3.medium \
    --name testcluster

If all is well, then something similar to the following is displayed (some output elided):

Example

[root@cod-client ~]# cm-cod-aws cluster create \
--head-node-type t3.medium --head-node-root-volume-size 50 \
--nodes 3 --node-type t3.medium \
--name testcluster
13:39:59:     INFO: No custom cluster password specified. Using a random password. Use
--log-cluster-password to see it. If you are going to use the Bright View you have to change this
password by using cm-change-passwd script (change root password of head node) or simply
'passwd root' command. If this cluster has a high-availability setup with 2 head nodes, be sure
to change the password on both head nodes.
13:40:01:     INFO: -----------------------------------------------------------------------------
13:40:01:     INFO:         Cluster:  testcluster
13:40:01:     INFO: -----------------------------------------------------------------------------
13:40:01:     INFO:      Image name:  brightheadnode-9.0-centos7u7-hvm-9(ami-07ef81b0e2d680131:9)
13:40:01:     INFO:      Image date:  2020-08-21 13:44 (39d 0h ago)
13:40:01:     INFO:  Package groups:  none
13:40:01:     INFO:         Version:  9.0
13:40:01:     INFO:          Distro:  centos7u7
13:40:01:     INFO:      Head nodes:  1 (t3.medium)
13:40:01:     INFO:   Head node IP:   <auto> (set with --head-node-internal-ip)
13:40:01:     INFO:           Nodes:  3 (t3.medium)
13:40:01:     INFO:          Region:  eu-west-1
13:40:01:     INFO:        Key path:  /root/.ssh/id_rsa.pub
13:40:01:     INFO: ------------------------------------------------------------------------------
13:40:01:     INFO: Press ENTER to continue and create the cluster.
13:40:01:     INFO: Press ctrl+c (or type 'a') to abort. Type 'i' for more info.
13:40:02:     INFO: ## Progress: 2
13:40:02:     INFO: #### stage: Setting up VPC
13:40:02:     INFO: ## Progress: 5
13:40:02:     INFO: #### stage: Creating VPC for on-demand testcluster in eu-west-1...
13:40:03:     INFO: ## Progress: 10
13:40:03:     INFO: #### stage: Adding internet connectivity...
13:40:04:     INFO: ## Progress: 12
13:40:04:     INFO: #### stage: Creating public subnet...
13:40:05:     INFO: ## Progress: 14
13:40:05:     INFO: #### stage: Creating private subnet...
13:40:06:     INFO: Created security group for the head node: sg-057c11f4d9857a025
13:40:06:     INFO: Created security group for the compute node: sg-06249aedeffcb0d2d
13:40:07:     INFO: ## Progress: 16
13:40:07:     INFO: #### stage: Creating head node (t3.medium)...
13:40:07:     INFO: Public key specified.
13:40:07:     INFO: Creating the head node VM instance.
13:40:08:     INFO: Created VM i-0d92887ef735f87b1.
13:40:09:     INFO: Waiting for head node to be running
13:40:25:     INFO: ## Progress: 22
13:40:25:     INFO: #### stage: Done setting up VPC for on-demand testcluster.
13:40:25:     INFO: ## Progress: 22
13:40:25:     INFO: #### stage: Starting the head node
13:40:25:     INFO: ## Progress: 30
13:40:25:     INFO: #### stage: Waiting for the head node to start
13:40:25:     INFO: ## Progress: 32
13:40:25:     INFO: #### stage: Assigning public IP to the head node (use
'--no-head-node-assign-public-ip' to skip)
13:40:27:     INFO: Elastic IP assigned: 54.217.104.57.
13:40:27:     INFO: ## Progress: 30
13:40:27:     INFO: #### stage: Waiting for cloud-init to start (use
'--cloud-init-timeout 0' to skip)
13:40:27:     INFO: Waiting for cloud-init to finish on the cluster testcluster (by trying to
connect to 54.217.104.57:8081).
13:44:04:     INFO: ## Progress: 100
13:44:04:     INFO: #### stage: Deployment of cluster: testcluster finished successfully.
13:44:14:     INFO: Script completed.
13:44:14:     INFO: ---------------------------------------------------------------------
13:44:14:     INFO: Time it took:   04:12
13:44:14:     INFO:   SSH string:   'ssh root@54.217.104.57'
54.217.104.57  testcluster
13:44:14:     INFO: Head node ID:   i-0d92887ef735f87b1
[root@cod-client ~]#

At this point, the head node running on AWS is ready for use. It can be accessed via SSH, as suggested in the SSH string line of the preceding output.

The regular nodes running on AWS are configured for use, but are not powered on. They can be powered on using the cmsh or Base View front ends, just as in a regular cluster.

From cmsh, the first cloud node can be powered on with, for example:

[testcluster->device]% power on cnode001

More details can be found at section 2.11.

2.5.3 Listing Clusters With cm-cod-aws

The clusters that are running can be listed with the command option cluster list:

Example

  [root@cod-client ~]# cm-cod-aws cluster list
14:16:44:     INFO: Listing clusters in region eu-west-1
+----------------+-----------------------+---------------------+---------------+-...
| Cluster Name   | VPC ID                | Head node ID        | Public IP     | ...
+----------------+-----------------------+---------------------+---------------+-...
| testcluster    | vpc-019dacc0877bfd71a | i-0d92887ef735f87b1 | 54.217.104.57 | ...
+----------------+-----------------------+---------------------+---------------+-...

Since all of the credentials are specified in the configuration file, no further command line options are needed.

2.5.4 Cluster Removal With cm-cod-aws

A cluster can be removed with a command in the form of: cluster delete <cluster name>:

Example

  [root@cod-client ~]# cm-cod-aws cluster delete testcluster
  14:18:57:     INFO: This will destroy VPCs for testcluster, continue?
  14:18:57:     INFO: Proceed? [yes/no]
  y
  14:19:00:     INFO: Stopping instances for VPCs on-demand testcluster
  14:19:00:     INFO: Listing instances...
  14:19:00:     INFO: Issuing instances termination requests...
  14:19:00:     INFO: Waiting until instances terminated...
  14:20:31:     INFO: Destroying VPC on-demand testcluster
  14:20:31:     INFO: Deleting subnets...
  14:20:32:     INFO: Deleting route tables...
  14:20:32:     INFO: Detaching and deleting gateways...
  14:20:33:     INFO: Flushing permissions...
  14:20:34:     INFO: Deleting security groups...
  14:20:35:     INFO: Deleting VPC...
  14:20:35:     INFO: Done destroying VPC on-demand testcluster
  [root@cod-client ~]#

2.5.5 ARM64 Images Support In COD

COD supports arch64 images. This allows workloads to run on ARM-based infrastructure.

A cluster with arch64 images can be created with the --arch=aarch64 CLI parameter. An instance type that supports aarch64 images must be chosen for this to work.

Example

    cm-cod-aws cc --name arm64-cluster --arch=aarch64 --head-node-type t4g.medium ...[other options]...

The text

...[other options]...

in the preceding example should be replaced with additional parameters required for the specific cluster configuration.

2.5.5.1 ARM64 Images Support Limitations

For COD:

• aarch64 image support is currently available only for COD-AWS. Support for other COD providers may be added in future releases.

• Mixed-architecture clusters are not supported. The head node and all compute nodes must use the same architecture-—either all x86_64 or all aarch64.

2.6 COD On OCI Via Command Line

Launching a COD inside OCI is carried out by running cm-cod-oci from either the Docker image or the Python package.

The cm-cod-oci command can take configuration options from one or more configuration files, environment variables, or command line options. More details on this are given in section 2.7.

Typically some of the more static options are configured in a configuration file, while some of the more variable options are specified on the command line.

Online help is also available. For example, running cm-cod-oci cluster create -h displays a list of options and explanations for the cluster create command.

2.6.1 Minimal Configuration File For COD On OCI

The creation of a minimal configuration file in ~/cm-cluster-on-demand.d/config.ini is recommended, with the following contents:

Example

[oci.credentials]
  oci_user=ocid1.user.oc1..<unique_ID>
  oci_fingerprint=12:34:56:78:90:ab:cd:ef:12:34:56:78:90:ab:cd:ef
  oci_tenancy=ocid1.tenancy.oc1..<unique_ID>
  oci_region=us-sanjose-1
  oci_key_file=~/.oci/oci_api_key.pem
[oci.cluster.common]
  oci_compartment_id = "ocid1.compartment.oc1..<unique_ID>
[oci.cluster.create]
  inbound_rule={CLIENT_IP},22:tcp
[cluster.create.password]
  ssh_pub_key_path=/root/.ssh/id_rsa.pub

The inbound_rule setting shown opens SSH access to the machine running the COD client (section 2.2.4).

The credentials should be replaced with the OCI credentials from 2.2. The value for ssh_pub_key_path is the key for connecting to the head node, which can be generated using ssh-keygen -b 4096.

2.6.2 Cluster Creation Run With cm-cod-oci

An example of a cm-cod-oci command that can be run is then:

Example

  cm-cod-oci cluster create \
    --head-node-number-cpus 2 --head-node-root-volume-size 50  \
    --head-node-shape VM.Standard.E4.Flex --nodes 2 \
    --node-shape VM.Standard3.Flex --node-number-cpus 2 --node-root-volume-size 50 \
    --name testcluster

Note: Bare metal shapes with no local disks, such as BM.Standard3.64, are not supported yet.

If all is well, then something similar to the following is displayed (some output elided):

Example

[root@cod-client ~]# cm-cod-oci cluster create \
  --head-node-number-cpus 2 --head-node-root-volume-size 50  \
  --head-node-shape VM.Standard.E4.Flex --nodes 2 \
  --node-shape VM.Standard3.Flex --node-number-cpus 2 --node-root-volume-size 50 \
  --name testcluster
05:35:54:     INFO: No custom cluster password specified. Using a random password. Use --log-cluster
-password to see it. If you are going to use the Bright View you have to change this password by us-
ing cm-change-passwd script (change root password of head node) or simply 'passwd root' command. If
this cluster has a high-availability setup with 2 head nodes, be sure to change the password on both
head nodes.
05:35:56:     INFO: Oracle Cloud Infrastructure (OCI) requires acceptance of terms & conditions ass-
ociated with relevant images whenever a new version of these images is made available. The following
terms & conditions / end user license agreements must be accepted in order to use this software:
Head node terms:
(already accepted)
Compute node terms:
- https://objectstorage.eu-marseille-1.oraclecloud.com/n/axogvvqxyiae/b/community-application
/o/eula.html
- https://cloudmarketplace.oracle.com/marketplace/content?contentId=50511634&render=inline
- https://www.oracle.com/legal/privacy/privacy-policy.html
Do you grant permission for Bright Cluster Manager to accept the required end user license
agreements and terms & conditions on your behalf when creating machine instances in OCI?
05:35:56:     INFO: Proceed? [yes/no]
yes
05:35:58:     INFO: -------------------------------------------------------------------------
05:35:58:     INFO:               Cluster:  testcluster
05:35:58:     INFO: -------------------------------------------------------------------------
05:35:58:     INFO:            Image name:  bcmh-ubuntu2004-10.0-1021(ubuntu2004-10.0:1021)
05:35:58:     INFO:            Image date:  2023-05-14 07:09 (22h 26m ago)
05:35:58:     INFO:        Package groups:  none
05:35:58:     INFO:               Version:  10.0
05:35:58:     INFO:                Distro:  ubuntu2004
05:35:58:     INFO:             Head node:  1 (VM.Standard.E4.Flex)
05:35:58:     INFO:                 Nodes:  2 (VM.Standard3.Flex)
05:35:58:     INFO:                Region:  us-sanjose-1
05:35:58:     INFO:           Compartment:  bright
05:35:58:     INFO:              Key path:  /root/.ssh/id_rsa.pub
05:35:58:     INFO:  Cloud authentication:  Head node will inherit COD credentials for cloud
authentication
05:35:58:     INFO: -----------------------------------------------------------------------
05:35:58:     INFO: Press ENTER to continue and create the cluster.
05:35:58:     INFO: Press ctrl+c (or type 'a') to abort. Type 'i' for more info.
05:36:00:     INFO: No availability domain configured; selected 'exbx:US-SANJOSE-1-AD-1'
from those in the configured compartment
05:36:01:     INFO: Creating VCN
05:36:05:     INFO: Creating subnet
05:36:07:     INFO: Creating network security groups
05:36:09:     INFO: Creating head node
05:37:18:     INFO: Waiting for sshd to start (ssh root@155.248.204.211).
05:38:26:     INFO: Waiting for CMDaemon to start.
05:39:28:     INFO: Waiting for CMDaemon to initialize.
05:41:38:     INFO: -----------------------------------------------------------------------
05:41:38:     INFO:       Cluster:  testcluster
05:41:38:     INFO: -----------------------------------------------------------------------
05:41:38:     INFO:  Head node ID:  ocid1.instance.oc1.us-sanjose-1.anzwuljricxcl6acdujsuhn
3yw6uze3h3c44jgbgitwwlqceqpdqkebzij3q
05:41:38:     INFO:     Public IP:  155.248.204.211
05:41:38:     INFO:      Key path:  /root/.ssh/id_rsa.pub
05:41:38:     INFO: -----------------------------------------------------------------------

At this point, the head node running on OCI is ready for use. It can be accessed via SSH, as suggested in the SSH string line of the preceding output. The regular nodes running on OCI are configured for use, but are not powered on. They can be powered on using the cmsh or Base View front ends, just as in a regular cluster.

In cmsh, the first cloud node can be powered on with:

Example

[testcluster->device]% power on cnode001

More details can be found in section 2.11.

2.6.3 Listing Clusters With cm-cod-oci

The clusters that are running can be listed with the cluster list command line option:

Example

[root@cod-client ~]# cm-cod-oci cluster list
05:47:06:     INFO: Listing clusters in region us-sanjose-1
+-------------+--------------------------+-----------------------+---------+---------------
| Cluster Name| VCN Name                 | Head node name        | State   | Shape
+-------------+--------------------------+-----------------------+---------+---------------
| testcluster | on-demand testcluster-01 | testcluster-head-node | RUNNING | VM.Standard.E4
+-------------+--------------------------+-----------------------+---------+---------------

Since all of the credentials are specified in the configuration file, no further command line options are needed.

2.6.4 Cluster Removal With cm-cod-oci

A cluster testcluster can be removed with a command such as:

Example

[root@cod-client ~]# cm-cod-oci cluster delete -- filters testcluster
05:50:45:     INFO: This will delete all OCI resources tagged as 'testcluster'; continue?
05:50:45:     INFO: Proceed? [yes/no]
yes
05:50:49:     INFO: Cluster testcluster: No cluster networks 'BCM_Cluster=testcluster'; skipping
05:50:49:     INFO: Cluster testcluster: No instance pools found matching 'BCM_Cluster=testcluster';
skipping
05:50:49:     INFO: Cluster testcluster: Terminating testcluster-head-node (instance state:
RUNNING)...
05:51:54:     INFO: Cluster testcluster: Terminated 1 instance in 64.48 seconds
05:51:54:     INFO: Cluster testcluster: No volumes found matching 'BCM_Cluster=testcluster';
skipping
05:51:55:     INFO: Cluster testcluster: Deleting subnet 'testcluster-01'
05:51:58:     INFO: Cluster testcluster: Deleting route table 'Default Route Table for on-demand
testcluster-01'
05:52:00:     INFO: Cluster testcluster: Deleting internet gateway 'testcluster-01'
05:52:02:     INFO: Cluster testcluster: No Instance Configurations found matching 'BCM_Cluster=test-
cluster'; skipping
05:52:02:     INFO: Cluster testcluster: Deleting VCN 'on-demand testcluster-01'
[root@cod-client ~]#

2.7 COD Client Configuration And Command Line Options

All COD clients (cm-cod-azure (section 2.4), cm-cod-aws (section 2.5), and cm-cod-oci (section 2.6) ) have a similar command line structure. They can take many command line arguments. All of those arguments can also be specified in configuration files, or as environment variables. Options specified as environment variables override options specified in configuration files. Options specified on the command line override options defined as environment variables as well as options defined in configuration files.

2.7.1 Command Line Structure

The command line structure is similar for all the different COD clients. Each client offers a number of top-level commands, which each offer a number of sub-level commands. The top-level and sub-level commands are always the first two positional arguments. Each top-level as well as sub-level commands can have several optional arguments.

The syntax is indicated by:

  cm-cod-{aws, azure, oci} [top-level command [sub-level command]] options

2.7.1.1 Command Line Help

To find out which commands and options are available, the --help|-h option can be used. The following example shows how to display the top-level commands and options for the cm-cod-aws client:

Example

[root@cod-client ~]# cm-cod-aws --help
usage: cm-cod-aws [-h] [--config CONFIG] [--no-system-config] [-v]
                  [--show-configuration] [--log-file LOG_FILE] [--version]
                  {cluster,c,cluster create,cc,cluster list,cl,cluster
                  delete,cd,cr,cremove,image,i,image
                  list,il,instancetype,config} ...
Cluster-on-demand by Bright Computing
positional arguments:
  {cluster,c,cluster create,cc,cluster list,cl,cluster delete,cd,cr,cremove,image,i,image list,
  il,instancetype,config}
    cluster (c)         Manage clusters
    cluster create (cc)
                        Create a new cluster
    cluster list (cl)   List all of the recognized clusters (VPCs)
    cluster delete (cd, cr, cremove)
                        Delete all resources in a cluster
    image (i)           Manage images
    image list (il)     List available Bright head node images
    instancetype        Instance types
    config              Configuration operations
optional arguments:
  -h, --help            show this help message and exit
  --config CONFIG, -c CONFIG
  --no-system-config
  -v, -vv, -vvv
  --show-configuration
  --log-file LOG_FILE
  --version

2.7.1.2 Command Line Help At Other Levels

The --help option can also be used to get the available options for a specific top-level command. For example:

cm-cod-aws cluster --help

The --help option can also be used to get the available options for a specific sub-level command. For example:

cm-cod-aws cluster create --help.

2.7.1.3 Command Line Parameter Explanations With --explain

More detailed information about a specific parameter can be obtained using the --explain flag.

The following example shows how to find out more about the --nodes parameter of the cluster create command of cm-cod-aws. The last parameter is specified as nodes, not --nodes.

Example

  [root@cod-client ~]# cm-cod-aws cluster create --explain nodes
  NAME
    nodes (cm-cod-aws cluster create --nodes <value> | -n <value>)
  DESCRIPTION
    The amount of cloud nodes to configure for a cluster. The nodes are not powered on
    automatically.
    Its default value is 5.
  NAMESPACES AND COMMANDS
    It is used in the following namespaces and commands
    -[cluster.create.nodes:nodes]
      '-[cluster.create:nodes]
         '-[aws.cluster.create:nodes]    (command: cm-cod-aws cluster create)
  HOW TO CONFIGURE
    On the Command Line:
      cm-cod-aws cluster create --nodes 5 [..]
      cm-cod-aws cluster create -n 5 [..]
    As Environment Variable:
      COD_NODES=5 cm-cod-aws cluster create [..]
    In a Config File:
      [cluster.create.nodes]
      nodes=5

The detailed information includes what the default for the parameter is (if applicable), how to specify the parameter as an environment variable, and how to include it in a configuration file.

2.7.2 Configuration Files

By default, configuration files for setting up the COD are searched for in the following locations, and in the following sequence. Settings found earlier in the sequence are overwritten by settings later on in the sequence.

• /etc/cm-cluster-on-demand.ini

• /etc/cm-cluster-on-demand.conf

• /etc/cm-cluster-on-demand.d/*

• ~/.cm-cluster-on-demand.ini

• ~/.cm-cluster-on-demand.conf

• ~/.cm-cluster-on-demand/*

• a file location specified on the command line. For example, a file mycodsettings can be accessed using the --config option of the COD client:

  Example

  [fred@basecm10 ~]$ cm-cod-azure --config mycodsettings

Typically, the administrator sets up configuration options in one of the first 3 locations, and the regular user modifies the options or adds other options in one of the last 4 locations. The configuration files are formatted as .ini files.

2.7.2.1 Viewing The Configuration File Options

A dump of the existing configuration can be viewed using the config dump command, for example:

cm-cod-azure config dump

To check what options have been applied, and their sequence, the log to STDOUT can be viewed if the -v|--verbose option has been applied.

A list of configuration options for a command can be seen with the --show-configuration option, for example (output elided):

Example

[fred@basecm10 ~]$ cm-cod-azure cluster create --show-configuration
+---------------------------------+--------------------------------------------------------------------+
| option                          | value                                                              |
+---------------------------------+--------------------------------------------------------------------+
| accept_eula                     | False (default)                                                    |
| access_validation               | True (default)                                                     |
| advanced_help                   | False (default)                                                    |
| ask_to_confirm_cluster_creation | True (default)                                                     |
| azure_client_id                 | '12345678-1234-1234-1234-123456789012' (file: /etc/.../config.ini) |
| azure_client_secret             | XXXXXXXXXXXXXXX (file: /etc/.../config.ini)  # use ... to uncover  |
| azure_location                  | 'westeurope' (file: /etc/cm-cluster-on-demand.d/config.ini)        |
| azure_subscription_id           | '12345678-1234-1234-1234-123456789012' (file: /etc/.../config.ini) |
| azure_tenant_id                 | '12345678-1234-1234-1234-123456789012' (file: /etc/.../config.ini) |
| cluster_password                | None (default)                                                     |
...

The table also indicates from which file a specific setting was taken.

Arguments to the cm-cod-{aws,azure} commands override the equivalent configuration file settings. This means that the configuration file settings of a working configuration can be used as a default template, and modifications to the template can conveniently be carried out via command line.

2.7.2.2 Setting Configuration File Options And Corresponding Arguments

The configuration files are formatted as .ini files, which can contain multiple sections. The --explain flag can be used to find out how a certain command line parameter could be defined in a configuration file. Thus in the earlier example, in the case of the cm-cod-aws cluster create --node parameters, defining it as a configuration file would end up with a file looking like this:

[cluster.create.nodes]
nodes=5

2.8 Using The AWS EC2 Management Console

The recommended way of managing COD is using Base View via its public IP address, or by using cmsh via the head node (section 2.8.5). This section (section 2.8) describes how the Amazon management console can also be used, to manage some AWS aspects of the instance.

A login is possible from https://console.aws.amazon.com/console/ by using the e-mail address and password of the associated AWS root account, or AWS IAM user account.

The head node instance can be watched and managed without BCM in the following ways.

2.8.1 Status Checking Via Instance Selection From Instances List

Clicking the Instances menu resource item from the navigation pane opens up the “Instances” pane. This lists instances belonging to the account owner. An instance can be marked by ticking its checkbox. Information for the selected instance is then displayed in the lower main pane (figure 2.1).

Figure 2.1: The EC2 Instances List

System (Amazon machine infrastructure) and instance (instance running under the infrastructure) reachabilities are similarly shown under the neighboring “Status Checks” tab (figure 2.2).

Figure 2.2: Reachability Status For An EC2 Instance

2.8.2 Acting On An Instance From The AWS EC2 Management Console

An instance can be marked by clicking on it. Clicking the Actions button near the top of the main center pane, or equivalently from a right-mouse-button click in the pane, brings up a menu of possible actions. These actions can be executed on the marked instance, and include the options to Start, Stop or Terminate the instance.

2.8.3 Connecting To An Instance From The AWS EC2 Management Console

A marked and running instance can have an SSH connection made to it.

As in the Azure case, for most users this means running ssh to the public IP address as suggested at the end of the AWS COD cluster creation run (section 2.5.2). This is assuming the private ssh key that was generated by the ssh-keygen command in the container is used for the ssh connection, which may mean that copying it out of the container is needed.

2.8.4 Viewing The Head Node Console

The head node takes about 2 minutes to start up. If, on following the instructions, an SSH connection cannot be made, then it can be worth checking the head node system log to check if the head node has started up correctly. The log is displayed on right-clicking on the “Actions” button, selecting the Instance Settings sub-menu, and selecting the “Get System Log” menu item (figure 2.3). A successful start of the system generates a log with a tail similar to that of figure 2.3.

Figure 2.3: System Log Of The Checkboxed Instance

A screenshot of the instance is also possible by right-clicking on the selected instance, then following the navigation path Instance Settings > Get Instance Screenshot.

2.8.5 Security Group Configuration To Allow Access To The Head Node Via cmsh Or Base View

Amazon provides a security group to each instance, which controls network access according to the inbound_rule setting configured at cluster creation (section 2.2.4); the inbound_network_acl_rule setting configures the subnet’s network ACL separately. A new security group can be configured, or an existing one modified, using the Edit details button in figure 2.4. Security groups can also be accessed from the navigation menu on the left side of the EC2 Management Console.

2.8.5.1 COD With AWS: Access With Base View:

The security group defined by Amazon for the head node can be modified by the administrator to allow remote connections to CMDaemon running on the head node (figure 2.4).

Figure 2.4: Security Group Network And Port Access Restriction

• To allow only a specific network block to access the instance, the network from which remote connections are allowed can be specified in CIDR format.

• If the cluster was created with an inbound rule opening port 8081, that allows Base View (section 2.4 of the Administrator Manual) to connect to the head node. This is because the Base View back end, which is CMDaemon, communicates via port 8081.

2.8.5.2 COD With AWS: Access With A Local cmsh:

If the cluster was created with an inbound rule opening port 22, the security group created by Amazon allows inbound SSH connections from outside the cloud to the instance running in the cloud, even if the incoming port 8081 is blocked. Launching a cmsh session within an SSH connection running to the head node is therefore possible, and should work without lag.

2.9 Using The Azure Dashboard

For Azure, as is the case for AWS, the cluster is normally expected to be managed via Base View or cmsh. Sometimes when carrying out some special configurations, there may be a need to manage the cluster objects directly via the portal at https://portal.azure.com. This requires an Azure login and password, which opens up the Azure dashboard, and allows the objects to be viewed and managed.

For example, the virtual machines of the COD can be viewed and managed via the Virtual machines resource (figure 2.5), and the virtual networks associated with the COD nodes can similarly be viewed and managed via Virtual networks resource.

Figure 2.5: Viewing Virtual Machines In A Cluster On Demand In Azure

Some items that may be useful when accessing the portal:

• An overview of the COD head node is possible via the navigation path:

  Home > Virtual machines > <head node> > Overview

• Boot process diagnostics can be viewed using the navigation path:

  Home > Virtual machines > <head node> > Boot diagnostics > Serial log

• A serial console can be accessed the navigation path:

  Home > Virtual machines > <head node> > Serial console

  The file /etc/ssh/sshd_config should have the parameter ChallengeResponseAuthentication set to yes, and the service restarted, for serial console login to work.

2.10 Using the OCI Console Dashboards

For OCI, the cluster is expected to be managed via cmsh. Sometimes when the cluster are corrupted or it needs special configurations, cluster objects can be directly controlled via the portal at https://www.oracle.com/cloud/. This requires cloud account, or SSO to sign in.

For example, you can view all your instances via Left Navigation > Compute > Instances (figure 2.6)

Figure 2.6: Viewing All Instances In A Cluster On Demand In OCI

2.11 COD: Cloud Node Start-up

Cloud nodes must be explicitly started up. This is done by powering them up, assuming the associated cloud node objects exist. The cloud node objects are typically specified in the cm-cod-aws or cm-cod-azure script which is run in the Docker instance. In the preceding example the cloud node objects are cnode001 and cnode002.

However, more cloud node objects can be created if needed after the scripts have run. The maximum number that may be created is set by the license purchased.

Large numbers of cloud node objects can be created with BCM as follows:

• In Base View they can conveniently be cloned via the navigation path:

  Devices > Cloud Nodes > <cloud node > Clone > <number>

• In cmsh a large number of cloud node objects can conveniently be created with the “foreach --clone” command instead, as described in section 4.2.

After creation, an individual cloud node, <cloud node>, can be powered up from within Base View via the navigation path:

Devices > Cloud Nodes > <cloud node > Power > On

As with regular non-cloud nodes, cloud nodes can also be powered up from within the device mode of cmsh. The initial power status (section 4.1 of the Administrator Manual) of cloud nodes is FAILED, because they cannot be communicated with. As they start up, their power status changes to OFF, and then to ON. Some time after that they are connected to the cluster and ready for use. The device status (as opposed to the power status) remains DOWN until it is ready for use, at which point it switches to UP:

Example

[head1->device]% power status
cloud ................. [  FAILED ] cnode001 (Cloud instance ID not set)
cloud ................. [  FAILED ] cnode002 (Cloud instance ID not set)
No power control ...... [ UNKNOWN ] head1
[head1->device]% power on -n cnode001
cloud ................. [   ON    ] cnode001
[head1->device]% power status
cloud ................. [   OFF   ] cnode001 (pending)
cloud ................. [  FAILED ] cnode002 (Cloud instance ID not set)
No power control ...... [ UNKNOWN ] head1
[head1->device]% power on -n cnode002
cloud ................. [   ON    ] cnode002
[head1->device]% power status
cloud ................. [   ON    ] cnode001 (running)
cloud ................. [   OFF   ] cnode002 (pending)
No power control ...... [ UNKNOWN ] head1
[head1->device]% !ping -c1 cnode001
ping: unknown host cnode001
[head1->device]% status
head1 .................... [   UP   ]
node001 .................. [   UP   ]
node002 .................. [  DOWN  ]
[head1->device]% !ping -c1 cnode001
PING cnode001.cm.cluster (10.234.226.155) 56(84) bytes of data.
64 bytes from cnode001.cm.cluster (10.234.226.155): icmp_seq=1 ttl=63 t\
ime=3.94 ms

Multiple cloud nodes can be powered up at a time in cmsh with the “power on” command using ranges and other options (section 4.2.3 of the Administrator Manual).

2.11.1 COD: IP Addresses In The Cloud

• The IP addresses assigned to cloud nodes on powering them up are arbitrarily scattered over the 10.0.0.0/8 network

  • No pattern should therefore be relied upon in the addressing scheme of cloud nodes

• Shutting down and starting up head and regular cloud nodes can cause their IP address to change.

  • However, BCM managing the nodes means that a regular cloud node re-establishes its connection to the cluster when it comes up, and will have the same node name as before.

2.12 COD With AWS: Optimizing AWS For High Performance Computing (HPC)

Optimization of cloud nodes for HPC in AWS is discussed in section 3.5.

2.13 COD And High Availability

2.13.1 Introduction

A COD head node, like a physical on-premises head node, can crash or somehow become non-operational. The possible reasons for this are pretty much the same in the case of a COD head node or a physical head node. The solution to reduce downtime due to these reasons is also the same: that is, to provide high availability (HA).

HA has always been available as a feature for BCM running on on-premises clusters (Chapter 17 of the Administrator Manual). Since 9.2, HA is also available for edge directors (section 2.1.1.1 of the Edge Manual) and for COD (this section 2.13).

The cluster administrator should have some familiarity with how HA for physical head nodes is implemented (Chapter 17 of the Administrator Manual) before implementing it on a COD. HA on COD is implemented very similarly to on a physical cluster, with a passive COD head node that takes over from a failing active COD head node.

COD with HA is deployed in a similar manner for AWS, Azure, and OCI.

The COD head nodes use shared storage for cm/shared and /home, just like the physical head nodes do. For COD HA head nodes, the shared storage needs to be a special filesystem in the cloud, just like the physical head HA nodes use a special filesystem.

Two shared IP addresses are also allocated––public and private. These shared IP addresses always point to the active head node, and are analogous to the external and internal IP address in the case of the physical head HA nodes (figure 17.1 of the Administrator Manual).

2.13.2 Deployment

2.13.2.1 COD HA Deployment Overview

A COD HA cluster is created by first creating a standard (that is: non-HA) COD cluster using cm-cod-aws (section 2.3), cm-doc-azure (section 2.4), or cm-cod-oci (section 2.6).

For AWS the following may be run:

Example

[root@basecm10 ~]# cm-cod-aws cluster create --version 10.0 --name basecm10-ha

The procedures for Azure (with cm-cod-azure) and OCI (with cm-cod-oci) are very similar.

The standard COD cluster, which here has arbitrarily been given the name basecm10-ha in advance, can now be converted to an HA COD cluster.

For AWS, after the deployment of the converted cluster is complete, the original head node automatically gets the suffix -a, and the new head node that is brought up gets suffix -b.

2.13.2.2 Configuration For Deployment

The conversion is done by running the cm-cloud-ha-setup wizard, whch starts up a TUI for configuring the HA deployment. The Deploy item in the main menu starts the configuration (figure 2.7):

Figure 2.7: Cluster On Demand TUI Wizard: Deploy Selection

The deployment process has the cluster administrator:

• provide the product key, so that the MAC address of the secondary COD head node can be added to the license (figure 2.8):

Figure 2.8: Cluster On Demand TUI wizard: deploy selection

• decide via a screen on whether to auto-allocate a private IP address to be shared between the head nodes, or whether to specify the private address (figure 2.9):

Figure 2.9: Cluster On Demand TUI wizard: decide on how private IP address is a created

• if specifying the private address is chosen, then the private IP address can be set (figure 2.10):

Figure 2.10: Cluster On Demand TUI wizard: private IP address allocation

• decide on where to deploy shared storage (figure 2.11):

Figure 2.11: Cluster On Demand TUI wizard: decide on where the shared storage should be

• If yes is selected, then the shared storage is created by the wizard creating an AWS EFS instance and sharing the /home and /cm/shared partitions among the cluster nodes via NFS

• If no is selected, then the shared storage should be specified manually, or a preconfigured storage can be used

• decide on how to set up a NAT gateway (figure 2.12):

Figure 2.12: Cluster On Demand TUI wizard: decide on how to set up a NAT gateway

• decide on setting up the shared IP address as public or private (figure 2.13):

Figure 2.13: Cluster On Demand TUI wizard: decide on how to set up the shared IP address

• save the configuration and deploy the conversion (figure 2.14):

Figure 2.14: Cluster On Demand TUI wizard: saving the configuration, as part of deployment

2.13.2.3 Conversion To HA

The deployment that converts the cluster to HA then takes place by bringing up a second HA node in the cloud. Deployment typically takes about 50 minutes. The progress can be followed on the console, and in the logs /var/log/cmha.log and /var/log/failoversetup.log.

If the deployment is not yet complete, then the cmha setup command indicates that with, for example:

root@basecm10-ha-a:~# cmha status
No HA status available.
Unable to get passive head node, is failover setup complete?

or later on, during the Finalizing HA stage, with:

root@basecm10-ha-a:~# cmha status
Unable to connect to cluster
Internal error, unable to connect to cluster: [Errno 111] Connection refused
Internal error, unable to connect at local cluster: [Errno 111] Connection refused

2.13.3 COD HA Checks And Recovery

2.13.3.1 COD HA Head Status Check

Once the deployment via cm-cloud-ha-setup is complete, the HA status of the cluster can be checked:

[root@basecm10-ha-b ~]# cmha status
Node Status: running in active mode
basecm10-ha-b* -> basecm10-ha-a
  mysql         [  OK  ]
  ping          [  OK  ]
  status        [  OK  ]
basecm10-ha-a -> basecm10-ha-b*
  mysql         [  OK  ]
  ping          [  OK  ]
  status        [  OK  ]
[root@basecm10-ha-b ~]#

2.13.3.2 COD Failover Session

If the active head node fails, then the passive head node is activated automatically by reassigning the shared public and private IP addresses to the passive head node, and marking the passive head node as active.

This process is called failover and can be triggered automatically or manually (section 17.1.7 of the Administrator Manual).

A manual failover can be carried out with cmha makeactive:

[root@basecm10-ha-b ~]# cmha makeactive
===========================================================================
This is the passive head node. Please confirm that this node should become
the active head node. After this operation is complete, the HA status of
the head nodes will be as follows:
basecm10-ha-b will become active head node (current state: passive)
basecm10-ha-a will become passive head node (current state: active)
===========================================================================
Continue(c)/Exit(e)? c
Initiating failover.............................. [  OK  ]
basecm10-ha-b is now active head node, makeactive successful
[root@basecm10-ha-b ~]# cmha status
Node Status: running in active mode
basecm10-ha-b* -> basecm10-ha-a
  mysql         [  OK  ]
  ping          [  OK  ]
  status        [  OK  ]
basecm10-ha-a -> basecm10-ha-b*
  mysql         [  OK  ]
  ping          [  OK  ]
  status        [  OK  ]
[root@basecm10-ha-b ~]#

Managing HA and manual failover is described in more detail in section 17.4 of the Administrator Manual.

2.13.4 Recovery Of The Passive COD Head

Unlike with physical HA head nodes, the recovery of a passive head node in the cloud is automated.

Recovery can be carried out by running cm-cloud-ha-setup wizard, and selecting the Recovery item in the main menu (figure 2.15):

Figure 2.15: Cluster On Demand TUI Wizard: Recovery Selection

The product key is needed to license the new head node (figure 2.16):

Figure 2.16: Cluster On Demand TUI Wizard: Product Key Selection

The recovery process warns that all resources associated with the old head node are to be deleted (virtual machine, disks, IP addresses) (figure 2.17):

Figure 2.17: Cluster On Demand TUI Wizard: Cleanup

It then clones the active head node, configuring the clone to become the new passive head node, and the cluster then resumes normal HA operation.

When recovery is completed, the new passive head node elastic IP for the cluster called <cluster_name> can be obtained by running:

cm-cod-aws cluster list [ <cluster_name>]

The cm-cod-aws command in the preceding text can be replaced ty cm-cod-azure or cm-cod-oci in order to get a similar listing for COD Azure or COD OCI clusters.

The output from cmha status can be used to confirm recovery is complete.

2.13.5 COD HA Deletion

The COD clusters for AWS listing might show something like (some output truncated):

[root@basecm10 ~]# cm-cod-aws cluster list
20:10:08:     INFO: Listing clusters in region eu-west-1
+--------------+-----------------------+-------------------------+---------------------+---------+-----------
| Cluster Name | VPC ID                | Head node ID            | Cluster IP          | State   | Type
+--------------+-----------------------+-------------------------+---------------------+---------+-----------
| basecm10-ha  | vpc-01d547e708c9eec9c | i-0224f8b38fe0b8658 (A) | 54.155.161.116 (A)  | running | t3.medium
|              |                       | i-0f906293daa687d20 (B) | 52.213.160.8 (B)    |         |
|              |                       |                         | 52.211.148.112 (HA) |         |
+--------------+-----------------------+-------------------------+---------------------+---------+-----------

The COD cluster can be specified for removal with the delete option. For example, with AWS:

Example

[root@basecm10 ~]# cm-cod-aws cluster delete basecm10-ha
20:11:43:  WARNING: Some clusters have a Cloud HA EFS storage. It will also be deleted
20:11:43:     INFO: This will destroy resources associated with clusters: 'basecm10-ha'.
              Would you like to continue?
20:11:43:     INFO: Proceed? [yes/no]
yes
20:11:53:     INFO: Deleting EFS fs-0dee13aa1d1a2d337...
20:12:16:     INFO: Finding instances for cluster 'basecm10-ha'...
20:12:16:     INFO: Issuing termination requests for 2 instances for cluster 'basecm10-ha'...
20:12:17:     INFO: Waiting until instances of cluster 'basecm10-ha' are terminated...
...
20:16:40:     INFO: Done destroying cluster 'basecm10-ha' resources in VPC 'on-demand basecm10-ha'