> This page is for version 10. > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # Cluster Extension Cloudbursting With Azure ## 5.1 Introduction To Cluster Extension Cloudbursting With Azure Cluster extension cloudbursting with AWS is covered in Chapter [3](/cloudbursting/cexcloud), where a GUI approach is described, and is also covered in section [4.1](/cloudbursting/clicloud#41-the-cm-cluster-extension-script-for-cluster-extension-clusters), where a text interface approach is described, Cluster extension cloudbursting with Microsoft Azure is covered in this chapter (Chapter 5). The procedure is somewhat similar to that for AWS: * The prerequisites to cloudburst into Azure are the same as those of cloudbursting into AWS, and are as previously described [here](/cloudbursting/cexcloud#3001-requirements). In summary, the prerequisites are as follows: * an activated cluster license should be ensured * an Azure account should exist * BCM must be registered as an Azure AD application ([here](#how-to-register-bright-as-an-azure-app)) * UDP port 1194 should be open * The techniques to cloudburst into Azure are also the same as that of cloudbursting into AWS, and are as previously described [here](/cloudbursting/cexcloud#3002-steps). In summary, the techniques are as follows: * a cloud director is set up in the cloud then started up * cloud nodes are then provisioned from the cloud director * The deployment of cluster extension cloudbursting for Azure is carried out in a similar way to how it is done for AWS. In summary, the tools used to deploy cluster extension cloudbursting for Azure are as follows: * the TUI `cm-cluster-extension` utility, run from the command line (section [5.2](#52-cluster-extension-into-azure)) * the `Azure Wizard`, run from Base View. ## 5.2 Cluster Extension Into Azure Section [4.1.1](/cloudbursting/clicloud#411-running-the-cm-cluster-extension-script-on-the-head-node-for-cluster-extension-clusters) introduces the `cm-cluster-extension` TUI wizard, which is run on the head node when configuring a cluster extension for Azure or AWS. After the cluster extension configuration is completed, the cluster is capable of extending into the cloud by having cloud nodes power up into the cloud. This section (section [5.2](#52-cluster-extension-into-azure)) covers how `cm-cluster-extension` can be run for Azure, and how clouds nodes can be deployed for Azure. There is also a Base View browser-based wizard for cluster extension into Azure. The browser wizard is accessible via the navigation path: `Cloud` > `Azure` > `Azure Wizard`. If the browser wizard is used, then the documentation here (section [5.2](#52-cluster-extension-into-azure)) for `cm-cluster-extension` can be followed since it is a very similar procedure. #### 5.2.0.1 Running The `cm-cluster-extension` Command Line Options As A Shell Dialog The `cm-cluster-extension` utility can be run as a dialog from the command line environment, as described in the section starting from [here](/cloudbursting/clicloud#4111-running-the-cm-cluster-extension-command-line-options-as-a-shell-dialog). Running it as a TUI session is however easier, and is described next. #### 5.2.0.2 Running The `cm-cluster-extension` TUI Dialog For Cluster Extension Into Azure The `cm-cluster-extension` utility can be run as a more user-friendly TUI session within the text environment. In a session described starting from [here](/cloudbursting/clicloud#4112-running-the-cm-cluster-extension-tui), an AWS cluster extension configuration is generated and deployed. In the session that is now described here, an Azure cluster extension configuration is generated and deployed instead: As in the AWS case, the `cm-cluster-extension` script is run without options, to bring up the TUI main screen (figure [5.1](#fig-cm-cluster-extension-configuration-processing-azure-selection)): ![Configuration Processing With cm-cluster-extension: Azure Selection](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/46c7ffb19ac5d61d7a2b24c2513998f8dcec669992ba0042a313174f5ac889f2/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-azure-selection.png) *Figure 5.1: Configuration Processing With cm-cluster-extension: Azure Selection* After Azure is selected, a new Azure provider can be set if none is already set up. If an Azure region has been configured previously, then its configuration can be removed. Testing only the network connectivity to the various Azure regions is also possible (figure [5.2](#fig-cm-cluster-extension-configuration-processing-azure-add-remove-test)). ![Configuration Processing With cm-cluster-extension: Add, Unconfigure, Or Test](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/54cc0d6fa279c9d7f8c8aebeaa23a1801a1eaa4ee7e60b521a898c38edbabaf2/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-azure-add-remove-test.png) *Figure 5.2: Configuration Processing With cm-cluster-extension: Add, Unconfigure, Or Test* If a new Azure provider is selected, then a screen comes up asking for the credentials for Azure (figure [5.3](#fig-cm-cluster-extension-configuration-processing-azure-credentials)): ![Configuration Processing With cm-cluster-extension: Azure Credentials](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/f9ddd30eaab8bbaac73e65bdb9f8980f1b3569df5c7d2bb3de34b09adde987e3/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-azure-credentials.png) *Figure 5.3: Configuration Processing With cm-cluster-extension: Azure Credentials* The inputs that are required here are Azure credentials, and exist for an activated Azure account which has had BCM registered as an application. **How BCM can be registered as an Azure application:** Registration of an Azure application can be carried out with the Azure portal at [https://portal.azure.com](https://portal.azure.com). The portal is accessible with an active Azure account. After logging in, navigating to locations via the navigation paths described next is possible. The steps to ensure a registration are then as follows: * User settings should first be checked to see if users can register applications. A navigation path to view this is: `Azure Active Directory` > `User settings` > `Users can register applications` The state should be set to `yes`. * The subscriptions should be checked for permissions. The navigation path to view subscriptions is simply: `Subscriptions` Within the subscription display the role can be viewed to determine if the account has adequate permissions to assign an AD application to a role. The account must have `Microsoft.Authorization/*/Write` access to assign an Azure AD application to a role. Assigning the `Contributor` role allows this access. Role-based access control (RBAC) is discussed at [https://docs.microsoft.com/en-us/azure/active-directory/role-based-access-control-manage-access-rest](https://docs.microsoft.com/en-us/azure/active-directory/role-based-access-control-manage-access-rest) * The user, for example, ``, should be checked for permissions. A suitable navigation path would be: `Azure Active Directory` > `Users and groups` > `` > `Azure resources` The Azure resources for the user, who is assigned the subscription, should show the role and assignment value. Suitable settings would be: ```bash ROLE: Contributor ASSIGNED TO: ``` * `Network`, `Compute`, and `Storage` namespaces must be registered. A convenient way to check that this is the case, is to use the Azure CLI tool from the head node. Instructions for installing it are available at [https://docs.microsoft.com/en-us/cli/azure/install-azure-cli](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli). After installation, a list of namespaces can be seen by running: ```bash az provider list --query "[].{Provider:namespace, Status:registrationState}" --out table ``` If the tool is run for the first time, then the tool gives the user a code and a URL. Using these, the user can authenticate the head node via a web browser. The required namespaces can be registered, if needed, with: ```bash az provider register --namespace Microsoft.Network --wait az provider register --namespace Microsoft.Compute --wait az provider register --namespace Microsoft.Storage --wait az provider register --namespace Microsoft.ADHybridHealthService --wait az provider register --namespace Microsoft.Authorization --wait az provider register --namespace Microsoft.Billing --wait az provider register --namespace Microsoft.ClassicSubscription --wait az provider register --namespace Microsoft.Commerce --wait az provider register --namespace Microsoft.Consumption --wait az provider register --namespace Microsoft.Features --wait az provider register --namespace Microsoft.MarketplaceOrdering --wait az provider register --namespace Microsoft.Resources --wait az provider register --namespace Microsoft.support --wait ``` * Create Azure Active Directory application: With all the permissions in place, the application can now be registered via the navigation path: `Azure Active Directory` > `App registrations` The application name and application URL values can be arbitrary since they are not actually used by `cm-cluster-extension`. The application type should be set to: `Web app / API` * The `Client ID` and `Client Secret` values are only available to Azure admin users, or the Azure application owners. Regular users cannot obtain these values. If the security settings allow Azure users to define their own Azure applications, then they can in theory create their own Azure application under the subscription, and use the data for that Azure application to get a `Client ID` and `Client Secret`. The credentials can now be picked up from the Azure portal account via the navigation paths shown in the following table:
**TUI** **navigation path From Azure Portal Menu After Azure Portal Login**
`Subscription ID` `Subscriptions` \> `SUBSCRIPTION ID`
`Tenant ID` `Azure Active Directory` \> `Properties` \> `Directory ID`
`Client ID` `Azure Active Directory` \> `App registrations` \> `APPLICATION ID`
`Client Secret` `Azure Active Directory` \> `App registrations` \> `APPLICATION ID` \> `Keys` \> `[` *the generated key must be noted* `]` \*
The `Provider Name` in figure [5.3](#fig-cm-cluster-extension-configuration-processing-azure-credentials) can be set to any user-defined value. For Azure, a sensible, if unimaginative value, is simply `azure`. To paste the credentials from the clipboard, the cluster administrator may find it helpful to know that a paste to the TUI can usually be carried out with a ``. After checking and accepting the credentials, legal terms may need to be accepted. The BCM Azure integration makes use of several node-installer images which are published in the Azure marketplace. In order to use those images, Azure requires the administrator to accept the legal terms for those images. The script automatically detects if the terms were already accepted in the past, in which case it proceeds to the next step. If the terms still need to be accepted, then several links to the legal terms are presented, along with a prompt to accept them. The output looks similar to the following (some output ellipsized): **Example** ```bash #### stage: cx_azure: Ask Terms Acceptance To use Azure VM images, you need to accept the following terms: - License (bcmni-azure-9-2:bcm-ni-azure-9-2-v1): https://download.microsoft.com/download/F/D/8/FD8BA8F... - Privacy Policy (bcmni-azure-9-2:bcm-ni-azure-9-2-v1): https://www.brightcomputing.com/privacy-policy - Marketplace Terms (bcmni-azure-9-2:bcm-ni-azure-9-2-v1): https://mpcprodsa.blob.core.windows.net/market... - License (bcmni-azure-9-2-free:bcm-ni-azure-9-2-v2): https://download.microsoft.com/download/F/D/8/FD8BA... - Privacy Policy (bcmni-azure-9-2-free:bcm-ni-azure-9-2-v2): https://www.brightcomputing.com/privacy-policy - Marketplace Terms (bcmni-azure-9-2-free:bcm-ni-azure-9-2-v2): https://mpcprodsa.blob.core.windows.net/mar... Accept (y/n)? ``` After checking and accepting the legal terms, the administrator is asked to choose a setup type for the wizard. This can be a `default` setup, or it can be an `advanced` setup (figure [5.4](#fig-cm-cluster-extension-configuration-processing-azure-advanced-setup)): ![Configuration Processing With cm-cluster-extension: Default Setup Or Advanced Setup](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/02ab1033275ce48a99a2cf21f47fd526b1a4581d5eb35f7ea14eed95c6bb82b2/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-advanced-setup.png) *Figure 5.4: Configuration Processing With cm-cluster-extension: Default Setup Or Advanced Setup* The `default` setup is recommended for most use cases. The `default` setup configures an OpenVPN network connection to the cluster extension. The `advanced` setup allows the network connection to the cluster extension to use ExpressRoute (for Azure). or another hardware-based VPN. After a `default` or `advanced` setup has been chosen, the initial number of cloud nodes to be set up in the cloud can be set (figure [5.5](#fig-cm-cluster-extension-configuration-processing-azure-number-of-cloudnodes.png)). A default of 3 is suggested. ![Configuration Processing With cm-cluster-extension: Setting The Initial Number Of Cloud Nodes](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/3b949d352af65849bc281113250ea0dafc2f623a36d2d2706fc39a5325b9e1dd/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-number-of-cloudnodes.png) *Figure 5.5: Configuration Processing With cm-cluster-extension: Setting The Initial Number Of Cloud Nodes* After setting an initial number of cloud nodes, the major regions into which these can be deployed are displayed (figure [5.6](#fig-cm-cluster-extension-configuration-processing-azure-major-regions)): ![Configuration Processing With cm-cluster-extension: Major Regions Selection](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/72493cd6c4667121d5622d7753ed1f43a798ccfee81c781bcafa218c0b77113c/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-azure-major-regions.png) *Figure 5.6: Configuration Processing With cm-cluster-extension: Major Regions Selection* After selecting a major region, the available regions under it into which the cloud nodes can be deployed are displayed (figure [5.7](#fig-cm-cluster-extension-configuration-processing-azure-regions)): ![Configuration Processing With cm-cluster-extension: Regions Selection](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/4756b7a68c4e1c0a0a1e2128fb72139af455a32e818c51aaa016a7def870a0f2/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-azure-regions.png) *Figure 5.7: Configuration Processing With cm-cluster-extension: Regions Selection* After selecting one or more regions, the default region into which the cloud nodes can be deployed can be set (figure [5.8](#fig-cm-cluster-extension-configuration-processing-azure-default-region)): ![Configuration Processing With cm-cluster-extension: Default Region Selection](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/18348a77ab3a17f3c145da12551018b797a15e759f923ed67a791ef57c511455/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-azure-default-region.png) *Figure 5.8: Configuration Processing With cm-cluster-extension: Default Region Selection* Azure regions are regional data centers, and the cloud director for a region helps manage the regular cloud nodes in that region when the cluster is extended into there. After setting the regions up, a default instance type must be set for the regular cloud nodes. The instance type family is first set. For example, the Dv3 type is an instance family type that can be set (figure [5.9](#fig-cm-cluster-extension-configuration-processing-azure-major-instance-type)): ![Configuration Processing With cm-cluster-extension: Instance Type Family](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/c2db6e6c348cc3d809a3f0766d88f6274a8673f4337d2a35ff10014da78afa7c/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-azure-major-instance-type.png) *Figure 5.9: Configuration Processing With cm-cluster-extension: Instance Type Family* **Instance types with Azure:** After selecting the instance type family, the specific default instance type can be set for the regular cloud nodes. The `Standard_D2_v3` type, (2 cores, 8GB RAM) is a common default that can be used (figure [5.10](#fig-cm-cluster-extension-configuration-processing-azure-default-instance-type)): ![Configuration Processing With cm-cluster-extension: Default Instance Type](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/83cfe15f347a7dd3d8fe5c0a1d1c4f25871b6d04e8c0a1652e98dc57ec66bbc1/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-azure-default-instance-type.png) *Figure 5.10: Configuration Processing With cm-cluster-extension: Default Instance Type* Azure intance types are documented online at * [https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes](https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes) and * [https://docs.microsoft.com/en-us/azure/virtual-machines/vm-naming-conventions](https://docs.microsoft.com/en-us/azure/virtual-machines/vm-naming-conventions). Eventually, when the wizard has finished deployment of the Azure instance, a summary listing of the supported types can be viewed in `cmsh` with: ```bash [root@b80 ~]# cmsh -c "cloud use azure ; types ; list" ``` or via the Base View navigation path: `Cloud` > `Azure` > `Azure VM Sizes` After setting the default instance type for the regular cloud nodes, screens (similar to figures [5.9](#fig-cm-cluster-extension-configuration-processing-azure-major-instance-type) and [5.10](#fig-cm-cluster-extension-configuration-processing-azure-default-instance-type)) are displayed to guide the administrator into setting a default instance type screen for the cloud director node type. The same default instance type suggested for regular cloud nodes earlier, `Standard_D2_v3`, is typically adequate for cloud director nodes too, for small clusters. **Accelerated networking with Azure:** NICs use hardware queues (ring buffers) to send and receive packets. Some NICs allow individual queues to be directly mapped from the physical host to a VM. The standard for doing this is called single root I/O virtualization (SR-IOV). This frees the host kernel from handling those packets, since, for example, network interrupts are delivered directly to the VM kernel. SR-IOV typically results in significantly lower latencies and ping times for the VM, similar to that of a bare metal machine. On Azure, *Accelerated Networking* enables SR-IOV, if the VM type supports it. The VM types that support Accelerated Networking can be found using the `az vm list-skus` command from the Azure CLI, as described at [https://learn.microsoft.com/en-us/azure/virtual-network/accelerated-networking-overview](https://learn.microsoft.com/en-us/azure/virtual-network/accelerated-networking-overview). **VPN configuration with Azure:** After having set the default instance types for the regular and cloud director nodes. the VPN connection to the cluster extension is configured. If the `advanced` setup (figure [5.4](#fig-cm-cluster-extension-configuration-processing-azure-advanced-setup)) was selected earlier in the wizard, then the administrator can select the type of VPN connection to the cluster extension (figure [5.11](#fig-cm-cluster-extension-configuration-processing-azure-custom-vpn)): ![Configuration Processing With cm-cluster-extension: Custom VPN Configuration](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/f4d47d4608512f3170620cc2f0d78284606d5eba0fc8f194b5cf926d9c0c37a2/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-azure-custom-vpn.png) *Figure 5.11: Configuration Processing With cm-cluster-extension: Custom VPN Configuration* * This can be the OpenVPN option that is automatically configured if the `advanced` setup is not chosen. * With the `advanced` setup it is also possible to configure * ExpressRoute (for Azure) * or another hardware-based VPN As part of VPN configuration, the wizard guides the administrator through the process of configuring the cluster extension so that the extension: * creates a resource group for each region by default. Alternatively, existing resource groups can be used instead. * creates a subnet per region by default. Alternatively, existing subnets can be used instead, but the cloud director must be able to contact the on-premises head node. * creates a storage account for each region by default. Alternatively, existing storage accounts can be used per region instead. * creates a security group for each region by default. Alternatively, existing security groups can be used instead. After the VPN connection is configured, the summary screen (figure [4.13](/cloudbursting/clicloud#4112-running-the-cm-cluster-extension-tui)) is displayed. This lets the administrator look things over before deployment: ![Configuration Processing With cm-cluster-extension: Azure Options Summary Screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/558c7616b72a5b998e2c8c992e0bebc6489f2d8344190e9b16403034061f9213/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-summary-screen.png) *Figure 5.12: Configuration Processing With cm-cluster-extension: Azure Options Summary Screen* The summary screen allows the following: * An administrator can just go ahead, save the configuration, and deploy the cluster extension. This is usually the expected action. * The configuration settings YAML file can be viewed. To scroll, the PageUp and PageDown keys are used. * An advanced configuration settings screen can be accessed in addition to the standard settings. The advanced settings are usually left alone. * The configuration file, \<*configuration file*>, can be saved and the TUI can be exited. By default, the value of \<*configuration file*> is set to `cm-cluster-extension.conf` in the home directory of the user. On exiting the TUI, deployment with that configuration can be carried out manually by running: ```bash cm-cluster-extension -c ``` #### 5.2.0.3 Deployment Of An Azure Configuration Created With `cm-cluster-extension` During configuration deployment, as the configuration is processed, text output indicates the progress. At the end of processing, the message ```bash Azure Cloud extension configuration finished ``` indicates that the cluster has been extended successfully. No nodes are activated yet within Azure. To start them up, the components of the cluster extension service for Azure must be started up by * powering up the cloud directors, as introduced for AWS in section [3.2](/cloudbursting/cexcloud#32-cluster-extension-with-aws-cloud-director-startup-from-scratch). The procedure for Azure is similar. * powering on the cloud nodes after the cloud directors are up. This may require first creating new cloud nodes, as introduced for AWS in section [3.3](/cloudbursting/cexcloud#33-cluster-extension-with-aws-cloud-node-startup-from-scratch). The procedure for Azure is similar. When powering up, the cloud director can be installed from scratch (section [3.2](/cloudbursting/cexcloud#32-cluster-extension-with-aws-cloud-director-startup-from-scratch)), or from a snapshot. For example, running the `power on` command from the `device` mode of `cmsh` on a head node shows amongst others the following states (some output elided or ellipsized): **Example** ```bash [basecm10->device]% power on westeurope-director cloud .................... [ PENDING ] westeurope-director ... [notice] basecm10: New certificate request with ID: 9 ... [notice] basecm10: westeurope-director [ INSTALLER REBOOTING ] ... [notice] basecm10: westeurope-director [ INSTALLING ] (node installer started) ... [notice] basecm10: New certificate request with ID: 10 (ldaps) ... [notice] basecm10: westeurope-director [ INSTALLER_CALLINGINIT ] (switching to local root) ... [notice] basecm10: westeurope-director [ UP ] ``` **Example** ```bash [root@basecm10 ~]# ps uww -C rsync | grep -o ' /cm/.*$' /cm/shared/ syncer@172.21.255.251::target//cm/shared/ ``` #### 5.2.0.4 Tracking Cloud Node Startup From `cmsh` The `provisioningstatus` command in the `softwareimage` mode of `cmsh` can be used to view the provisioning status of cloud directors from the head node, or of cloud nodes from the cloud director (some output elided): **Example** ```bash [root@basecm10 ~]# cmsh -c "softwareimage provisioningstatus" ... + westeurope-director ... Up to date images: none Out of date images: default-image Nodegroups: westeurope-director-dependents ``` In the preceding output, the absence of an entry for “`Up to date images`” shows that the cloud director does not yet have an image that it can provision to the cloud nodes. After some time, the last few lines of output should change to something like: **Example** ```bash + westeurope-director ... Up to date images: default-image Nodegroups: westeurope-director-dependents ``` This indicates the image for the cloud nodes is now ready. The image used for a device is defined by the value of `image` under `cloudsettings` for the cloud director, or cloud node: **Example** ```bash [root@basecm10 ~]# cmsh -c "device use westeurope-director; cloudsettings; get image" marketplace-free-node-installer-image ``` By default, this value is the Azure marketplace image for the current BCM version. With the `-a` option, the `provisioningstatus -a` command gives details that may be helpful. For example, while the cloud director is having the default software image placed on it for provisioning purposes, the source and destination paths are `/cm/images/default-image`: **Example** ```bash [root@basecm10 ~]# cmsh -c "softwareimage provisioningstatus -a" Request ID(s): 4 Source node: basecm10 Source path: /cm/images/default-image Destination node: westeurope-director Destination path: /cm/images/default-image ... ``` After some time, when the shared filesystem is being provisioned, then an indication of progress is shown by the `Request ID` incrementing, and the source and destination paths changing to the `/cm/shared` directory: ```bash [root@basecm10 ~]# cmsh -c "softwareimage provisioningstatus -a" Request ID(s): 5 Source node: basecm10 Source path: /cm/shared Destination node: westeurope-director Destination path: /cm/shared ... ``` After the shared directory and the cloud node software images are provisioned, the cloud director is fully up. Cloud node instances can then be powered up and provisioned from the cloud director. The instances can be started up from scratch (section [3.2](/cloudbursting/cexcloud#32-cluster-extension-with-aws-cloud-director-startup-from-scratch)), or from snapshot (section [3.4](/cloudbursting/cexcloud#34-cluster-extension-with-aws-cloud-director-and-cloud-node-startup-from-snapshots)). #### 5.2.0.5 Cluster Extension Cloudbursting Logging All Azure logging goes to the CMDaemon logs in `/var/log/cmdaemon`, where the `CLOUD` tag is used to indicate cloud-related operations. ## 5.3 Cluster Extension Into Azure: Cloud Node Startup From Scratch This section discusses the configuration of regular cloud node startup from scratch. **NOTE:** Cloud director and node startup from snapshot is also possible. Section [3.4](/cloudbursting/cexcloud#34-cluster-extension-with-aws-cloud-director-and-cloud-node-startup-from-snapshots) describes it for AWS. For Azure it is more involved, and is not covered here. To *configure* regular cloud nodes in Azure from scratch does not require a working cloud director. However to *boot up* the regular cloud nodes does require that the cloud director be up, and that the associated networks to the regular cloud nodes and to the head node be configured correctly. If needed, additional cloud provisioning nodes ([section 5.2 of the *Administrator Manual*](/admin/provisioning#52-provisioning-nodes)) can be configured by assigning the provisioning role to cloud nodes, along with appropriate nodegroups ([here](/admin/provisioning#521-provisioning-nodes-configuration-settings)) values, in order to create a provisioning hierarchy. Similarly to how it is done in AWS, the creation and configuration of regular cloud node objects in Azure is conveniently carried out by cloning another regular cloud node, from one of the default cloud nodes already created by the cluster extension wizard (section [5.2](#52-cluster-extension-into-azure)). A navigation path to do this cloning in Base View is: `Devices` > `Cloud Nodes` > `<`*cloud node hostname* > `Clone` Cloud node objects can also be created in `cmsh` as described in section [4.2](/cloudbursting/clicloud#42-launching-the-cloud-nodes). ## 5.4 Cluster Extension Into Azure: `shutdown` *Vs* `power off` An Azure cloud node has two stopped states: 1. `stopped`: A node running within Azure can be set to this state by running the `shutdown` command: * within the `device` mode of `cmsh` * with Base View, using the navigation path:\ `Devices` > `Cloud Nodes` > `<`*cloud node hostname* > `OS` > `Shutdown` * Or by clicking the `stop` button for that node within the Azure web portal, once. 2. `stopped (deallocated)`: A node running within Azure can be set to this state by running the `power off` command: * within the `device` mode of `cmsh` * with Base View, using the navigation path:\ `Devices` > `Cloud Nodes` > `<`*cloud node hostname* > `Power` > `Off` * Or by clicking on the `stop` button within the Azure web portal, twice. Carrying out a `power off` command is like a hard power off command, which can under some unusual conditions cause filesystem corruption. It is therefore safer to run the `shutdown` command first, wait for the node to shut down via the OS. After that, running the `power off` command ensures that the node is deallocated. From a financial point of view when using Azure, a node that is shut down but not deallocated continues to incur costs. However, a node that is deallocated does not continue to incur costs. ## 5.5 Submitting Jobs With `cmjob` And Cloud Storage Nodes, For Azure Cluster Extension Clusters The `cmjob` utility, a job submission wrapper for end users for cluster extension clusters, is introduced and documented for AWS cluster extension in section [4.3](/cloudbursting/clicloud#43-submitting-jobs-with-cmjob-and-cloud-storage-nodes-for-cluster-extension-clusters). As a summary, its configuration procedure consists of: * Making sure the `cmdaemon-cmjob` package is installed on the head node and the cloud image * Ensuring the users that are to use `cmjob` have the `createcmjobcertificate` property set to `yes` * Making sure that cloud storage nodes are set up. The administrator can run `cm-cloud-storage-setup` to configure the cloud storage nodes. The details of the configuration procedure for `cmjob` for Azure cluster extension is almost identical to that for AWS. The configuration procedure documented in section [4.3](/cloudbursting/clicloud#43-submitting-jobs-with-cmjob-and-cloud-storage-nodes-for-cluster-extension-clusters) can therefore also be followed for Azure. How the end user can use `cmjob` is documented in [section 4.7 of the *User Manual*](/user/workloadmanagement#47-running-jobs-in-cluster-extension-cloud-nodes-using-cmjob). ### 5.5.1 Integration Of `cmjob` With Azure NetApp Files *Azure NetApp Files* (ANF) is a high-performance highly scalable network file service that is available as a cloud service within most Azure regions. A BCM cluster that has burst into such a region can use ANF to create a shared storage volume that can be read from, and can be written to, by many compute nodes simultaneously. On a cluster that has burst into such a region, `cmjob` can provide workload managers with an ANF volume to store input and output data. Sequential workload manager jobs can use the same ANF volume. This means that the output of one job can immediately become the input of the next one without having to transmit the data to and from Azure cloud storage, or to and from the head node. Additionally, workload manager jobs that run on multiple compute nodes in parallel can obtain their shared input from the same ANF volume. Providing shared storage to cloud compute nodes via ANF is an alternative to providing it via via cloud storage nodes (section [4.3](/cloudbursting/clicloud#43-submitting-jobs-with-cmjob-and-cloud-storage-nodes-for-cluster-extension-clusters)). The storage node approach is not sufficiently scalable when there are more than several dozen compute nodes working simultaneously with the same storage node. ANF is significantly more scalable and far more performant, although at a higher cost. BCM currently offer three different methods of integrating `cmjob` with Azure NetApp Files: 1. *On-Demand ANF*: An ANF file system is created on a per-job basis and is deleted automatically when the job completes. 2. *User-managed ANF*: A user creates an ANF volume and then submits one or more jobs that use that same ANF volume. These jobs can execute in sequence or simultaneously. When the last job has finished, the user is responsible for deleting the ANF volume. 3. *Admin-managed ANF*: An administrator creates an ANF volume and shares it with one or more users. These users are able to use the volume in the same way as with user-managed ANF, but they cannot delete the volume or see each other’s files. #### 5.5.1.1 Prerequisite To be able to use the integration of `cmjob` with ANF NetApp Files, the cluster must be bursting into an Azure region that supports Azure NetApp Files. Azure does not offer this feature in all regions. The Azure documentation should be checked to see if a region offers ANF for Azure. #### 5.5.1.2 Enabling `cmjob` Integration With Azure NetApp The integration of `cmjob` with ANF can be accomplished in two ways, either: * through the `cm-cloud-storage-setup` wizard (section [4.3.1](/cloudbursting/clicloud#431-installation-and-configuration-of-cmjob-for-data-aware-scheduling-to-the-cloud)) or * manually **Enabling `cmjob` integration through the wizard:** During the `cm-cloud-storage-setup` wizard run, when the guidance for the Azure extension is being carried out, the administrator is asked if ANF support should be enabled. If the administrator agrees to this, then ANF support is set up. After the deployment has finished, all users that have the `createcmjobcertificate` property set to `yes` can then create user-managed and on-demand ANF volumes through `cmjob`. **Enabling `cmjob` integration manually:** The manual integration approach should be followed when `cmjob` has already been configured on a cluster. That is, if the `cm-cloud-storage-setup` tool has been run, but the cluster has been running without ANF support so far. To add `cmjob` integration with ANF manually, the following manual action is all that is needed: Every user that should be able to create ANF volumes should have their `createcmjobcertificate` property set to `yes`. Then, one or both of the following ANF tokens should be appended to the `cloudjob` profile: * `ON_DEMAND_ANF_TOKEN` * `USER_MANAGED_ANF_TOKEN`. **Example** A user, `fred` is already using cluster extension with Azure, but without ANF. Now the user would like to be able to launch and manage ANF storage on demand. The administrator should generate the `cmjob` certificate for the user: ```bash [basecm10->user[fred]]% set createcmjobcertificate yes; commit ``` The administrator should also add the desired tokens to the `cloudjob` profile: ```bash [basecm10]% profile use cloudjob [basecm10->profile*[cloudjob*]]% append tokens ON_DEMAND_ANF_TOKEN USER_MANAGED_ANF_TOKEN [basecm10->profile*[cloudjob*]]% get tokens SUBMIT_CLOUD_JOB_DESCRIPTION_TOKEN GET_CLOUD_JOB_DESCRIPTION_TOKEN ON_DEMAND_ANF_TOKEN USER_MANAGED_ANF_TOKEN [basecm10->profile*[cloudjob*]]% commit ``` The token assignment is global, and not per-user. No directors need to be updated or rebooted. All users with the `createcmjobcertificate` property set to `yes` should then be able to create and submit jobs with a user-managed ANF volume, or submit jobs with an on-demand ANF volume, depending on which tokens were assigned to the `cloudjob` profile. #### 5.5.1.3 Configuration Through `cmsh`, an administrator can use profile tokens ([section 6.4 of the *Administrator Manual*](/admin/usermanagement#64-tokens-and-profiles)) to control how users can work with ANF. An administrator can also set quotas that control how many ANF volumes a user can have simultaneously and how large these ANF volumes can be. The aim of this is to help manage costs. **Profile tokens:** * `ON_DEMAND_ANF_TOKEN`: allows a user to use the `--on-demand-anf` flag when submitting jobs with `cmjob`. * `USER_MANAGED_ANF_TOKEN`: allows a user to create, manage and delete ANF volumes. * `DELETE_ANY_ANF_VOLUME_TOKEN`: allows a user to delete ANF volumes that are owned by other users. Normally reserved for root only. * `LIST_ALL_ANF_VOLUMES_TOKEN`: allows a user to see ANF volumes owned by other users. Normally reserved for root only. * `SHARE_ANF_VOLUME_TOKEN`: allows a user to share ANF volumes with other users. Only the owner is allowed to share it. Normally reserved for root only. **Settings:** The following `cmsh` navigation path `cmsh` > `cmjob` > `storagenodepolicies` > `use` \<*policy*> > `intermediatestorage` leads to these storage parameters: * `Max ANF Volume Capacity (TiB)` * `Max ANF Volume Count Per User` which control quota limits for ANF volumes for the number and size of ANF volumes, for when a user or an administrator creates a user-managed or administrator-managed ANF volume. The count is combined. If a user has already reached the maximum number of user-managed ANF volumes, then the user cannot submit a job that uses an on-demand ANF volume. Also in the same navigation path is the storage parameter: * `Default ANF Volume Capacity (TiB)` which controls the default capacity when a user or admin creates a user-managed or admin-managed ANF volume. #### 5.5.1.4 Usage As listed earlier, the three methods that BCM provides for `cmjob` integration with ANF are: 1. on-demand ANF, 2. user-managed ANF, and 3. admin-managed ANF. How to run these methods using `cmjob` is covered in the next sections. **Using on-demand ANF:** When using `cmjob` to submit a job, a user can choose whether to use an Azure storage volume or an ANF file system to host the input data and output data. For a user there is little difference between the ways in which the two shared storage options are used, beside performance. The steps are: 1. an Azure storage volume or an ANF file system is created as shared storage, 2. the input data is copied from the head node to this shared storage 3. all compute nodes chosen by the workload manager mount this shared storage, and then run their jobs, 4. storing their output data on the same shared storage. 5. the output data is uploaded from the shared storage to the head node 6. the Azure volume or ANF file system is deleted. Submitting a job to run with Azure storage nodes is the default. If a user wants a job to use ANF, then the user must set the `--on-demand-anf` flag when submitting the job: **Example** ```bash cmjob submit --on-demand-anf job.sh ``` The capacity of the OnDemand ANF volume is determined by doubling the size of the input data. This calculation takes into account both the data uploaded from the head node, and labeled data downloaded from the Azure cloud storage. If the user thinks that the calculated capacity is not large enough to hold the output data (the output includes temporary data), then the user can increase the capacity with the `--expected-output-size` parameter. The specified size is combined with the size of the input data and rounded up to the nearest valid ANF volume size. **Using user-managed ANF** `cmjob` with a user-managed ANF file system operates differently from an on-demand system. With user-managed ANF, the user is responsible for creating and deleting the ANF file system. In some cases, this can be more expensive than using the on-demand approach, especially if the user forgets to delete the ANF volume after no longer needing it. However, with user-managed ANF a user can run multiple concurrent jobs with the same ANF file system. It is also possible to run multiple jobs in sequence, each using the output of the previous job as input, without having the overhead of the files being copied from the ANF file system to Azure cloud storage, and vice versa. Depending on the size of the workload, this option may be cheaper, or more desirable, than on-demand ANF. To create an ANF volume, the user can execute `cmjob` with the `create-anf-volume` option: **Example** ```bash cmjob create-anf-volume --capacity ``` Here: * \<*name or ID*> can be a sequence of alphanumeric characters and dashes. It does not have to be unique. * \<*capacity*> is the total size of the ANF file system in terabytes. If not specified, the default value is used. The default value can be managed via:\ `cmsh` > `cmjob` > `storagenodepolicies` > `use` \<*policy*> > `intermediatestorage`. To delete an ANF volume, a user can run `cmjob` with the `delete-anf-volume` option: **Example** ```bash cmjob delete-anf-volume ``` To obtain a list of all existing ANF volumes with their names and statuses, the user can run `cmjob` with the `list-anf-volumes` option: **Example** ```bash cmjob list-anf-volumes ``` Once the ANF volume has reached a status of `AVAILABLE` (which can take a few minutes), it is possible to submit a job with this volume name or ID with the `--anf-volume` option. **Example** ```bash cmjob submit job.sh --anf-volume ``` **Using admin-managed ANF** Using `cmjob` with an admin-managed file system is very similar to using it with a user-managed file system. The main difference is that the administrator is responsible for creating and deleting the file system, and that the file system of the administrator is shared with specific users. To share the file system, the `share-anf-volume` option can be run by the administrator, together with the `--with option`, as follows: **Example** ```bash cmjob share-anf-volume --with ``` With admin-managed ANF, users do not need any ANF-specific tokens to submit a job with an ANF volume. Using admin-managed ANF volumes instead of user-managed ANF volumes gives the administrator more control over the creation and deletion of ANF volumes. To stop sharing the ANF volume, the admin can run the `--stop-sharing` option: **Example** ```bash cmjob share-anf-volume --stop-sharing ``` **Tracking NetApp costs:** All Azure NetApp Files capacity pools created via `cmjob` have a `BCM Owner` tag assigned to them. This tag allows the cluster administrator to track the NetApp costs on a per-user basis using Azure’s cost management capabilities. ## 5.6 Creating An Azure Cluster Extension Using ExpressRoute Or A Hardware VPN For simple configurations, BCM recommends, and provides, OpenVPN by default for cluster extension cloudbursting VPN connectivity. If there is a wish to use Direct Connect or a hardware VPN (for example, an IPSec tunnel), then BCM can be configured to work with those. The Azure cluster extension always runs in an Azure Virtual Network. In the default deployment scenario, the head node communicates with the cloud nodes using an OpenVPN connection between the head node and the cloud director. In the case of an ExpressRoute connection or a hardware VPN, the head node can be configured to communicate directly with the cloud director and cloud nodes. Setting up ExpressRoute or a VPN for a cluster extension can be carried out according to these three steps: 1. Virtual network creation (section [5.6.1](#561-creating-a-virtual-network)). This step can be skipped if an existing virtual network is to be used. 2. Connecting the local network to the virtual network (section [5.6.2](#562-connecting-the-local-network-to-the-virtual-network)). The connection can be with, for example, ExpressRoute, or a hardware VPN. This step can be skipped if the local network is already connected to an existing virtual network via an ExpressRoute or a hardware VPN connection. 3. Configuring and deploying the cluster extension (section [5.6.3](#563-configuring-and-deploying-the-cluster-extension)). ### 5.6.1 Creating A Virtual Network The Azure virtual network is the fundamental building block for building private networks in Azure. A virtual network allows the cloud director and cloud nodes to communicate with each other securely, and can be extended to the on-premises networks. Documentation for creating a virtual network can be found at: [https://docs.microsoft.com/en-us/azure/virtual-network/quick-create-portal](https://docs.microsoft.com/en-us/azure/virtual-network/quick-create-portal) A new virtual network can be created and configured for the cluster extension as follows: 1. After logging in to the Azure portal, [https://portal.azure.com](https://portal.azure.com), from the home page, [https://portal.azure.com/#home](https://portal.azure.com/#home), the following navigation path can be followed: `+ Create a resource` > `Azure Marketplace` > `Networking` > `Virtual network` ![Azure Marketplace Networking Screen](/base-command-manager/product-manuals/_fern-img/9154d15dd88af51023401296c018c66cc18a5915e6ec9345c2625c4c5760d683.webp) *Figure 5.13: Azure Marketplace Networking Screen* 2. A resource group should be selected or a new resource group should be created for the virtual network in the `Basics` tab: ![Creating A Resource Group In The Basics Tab](/base-command-manager/product-manuals/_fern-img/cd05d9c4ebf6ef0fbfc03d2eaaca819cef45d599fe4e7256f7bfec0e3e2eb870.webp) *Figure 5.14: Creating A Resource Group In The Basics Tab* 3. An IPv4 CIDR block can be set to a desired range via the `IP Addresses` tab, reached by clicking on the `Next: IP Addresses` button: ![Creating IP Address Ranges In The IP Addresses Tab](/base-command-manager/product-manuals/_fern-img/fb297aa199cd7341189666ce467b523a8cb2f9993c2bbbd625c1196f85234dae.webp) *Figure 5.15: Creating IP Address Ranges In The IP Addresses Tab* This range should not conflict with the address space used on-premises. A possible range could be, for example, `10.77.0.0/16` 4. Once the virtual network range is set, a subnet can be entered via the `+ Add subnet` option, which opens up a dialog. ![Creating IP Address Ranges In The IP Addresses Tab: Subnet Addition](/base-command-manager/product-manuals/_fern-img/59d495b43267e05847172b03e2716476be37c21b974cf6a50129918b77e5948b.webp) *Figure 5.16: Creating IP Address Ranges In The IP Addresses Tab: Subnet Addition* A possible subnet could be, for example, `10.77.0.0/24`. The `Add` button then adds it. 5. The navigation path: `Review + create` > `Create` goes on to deploy the virtual network. ### 5.6.2 Connecting The Local Network To The Virtual Network Azure offers two virtual network gateway types to connect the local network to the virtual network: ExpressRoute and VPN. #### 5.6.2.1 Connecting Via ExpressRoute To connect the local network to the virtual network via ExpressRoute requires using an *ExpressRoute circuit connection*. Azure’s instructions for this are at: [`https://docs.microsoft.com/en-us/azure/expressroute/expressroute-howto-linkvnet-portal-resource-manager`](https://docs.microsoft.com/en-us/azure/expressroute/expressroute-howto-linkvnet-portal-resource-manager) #### 5.6.2.2 Connectng Via A Site-To-Site Connection Using A VPN Connecting an on-premises network to the virtual network using a VPN requires an Azure site-to-site connection. Azure’s instructions for configuring a site-to-site IPsec VPN connection are at: [`https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-howto-site-to-site-resource-manager-portal`](https://docs.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-howto-site-to-site-resource-manager-portal) In Azure this consists of three components: 1. a virtual network gateway (routes traffic from local network to virtual network) 2. a local network gateway (routes traffic from virtual network to local network) 3. a connection The local network gateway needs to define the address ranges of the local network. The default address ranges for the local network are: `10.141.0.0/16`, `10.2.0.0/16`, and `192.168.200.0/24`. The virtual network gateway requires a gateway subnet. The address range for the gateway subnet should be a subnet of the virtual network that does not overlap with the subnet address range for the cluster extension, for example: `10.77.1.0/24`. ### 5.6.3 Configuring And Deploying The Cluster Extension Once IP connectivity from on-prem to the Azure virtual network is running, the final step is creating the cluster extension using the newly created site-to-site connection. #### 5.6.3.1 Getting Through Shorewall First, the firewall rules on the head node must be adjusted to accept traffic from the virtual network. The file `/etc/shorewall/rules` can be edited so that the `net` section allows packets from the CIDR subnet. A quick-and-dirty way to do it is to append to the file with: ```bash [root@basecm10 ~]# echo "ACCEPT net: fw - -" >> /etc/shorewall/rules [root@basecm10 ~]# shorewall reload ``` #### 5.6.3.2 Cluster Extension Advanced Settings Configuration A cluster extension can now be created using the `cm-cluster-extension` utility in TUI mode as explained in section [5.2](#52-cluster-extension-into-azure). However, on reaching the summary screen (figure [5.12](#fig-cm-cluster-extension-configuration-processing-azure-summary-screen), [here](#fig-cm-cluster-extension-configuration-processing-azure-summary-screen)), the cluster administrator should not immediately select the `Save config & deploy` option, but should first go to the `Advanced settings` option, which opens up an `Advanced plugin configuration` screen (figure [5.17](#fig-cm-cluster-extension-configuration-processing-advanced-plugins-azure)): ![Cluster Extension Configuration Processing: Advanced Plugins Screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/53dd2cb40411d94845d35ded33c3aca39a58662cf74f378f372ecdd654b571f7/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-advanced-plugins.png) *Figure 5.17: Cluster Extension Configuration Processing: Advanced Plugins Screen* The `clusterextension` option should be selected, which opens up an advanced options settings screen for Azure (figure [5.18](#fig-cm-cluster-extension-configuration-processing-advanced-plugins-clusterextension-options-azure)): ![Cluster Extension Configuration Processing: Advanced Plugins For ’clusterextension’, Azure Options Screen](/base-command-manager/product-manuals/_fern-files/nvidia-bcm-manuals.docs.buildwithfern.com/b9f6f0e64eecade61d5d28bdcf9556fa5730d4c5584d8f2d846bbd39030767ec/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/_dot_dot_/tmp/tmp-465-AR0SE6FbhiT1/latextofern/fern/assets/v10/figures/cm-cluster-extension-configuration-processing-advanced-plugins-clusterextension-options-azure.png) *Figure 5.18: Cluster Extension Configuration Processing: Advanced Plugins For ’clusterextension’, Azure Options Screen* In this screen: * the `Create tunnel networks` option should be set to `False` * the `Subnet for region` \<*`region`*> should be set to the subnet that has been created for the cluster extension. The cluster extension is what is being connected to via ExpressRoute or via site-to-site connection The TUI advanced settings screens can be backed out of by selecting `Back` twice. The option `Save config & deploy` can then be selected to create the cluster extension. #### 5.6.3.3 Cloud Node Certificate Autosigning By default BCM does not issue certificates for nodes on the external network. This means that for cloud nodes the certificates need to be issued manually, once for every new cloud node. Alternatively, to automatically sign certificate requests by cloud nodes, autosign can be enabled by the administrator for external networks. Autosigning may be a security concern, as this allows anyone on the external network to request a node certificate. Autosign can be enabled on externalnet in `cmsh` as follows: ```bash [root@basecm10~]# cmsh [basecm10]% network [basecm10->network]% set externalnet allowautosign always [basecm10->network]% commit ``` The cluster extension can now be deployed as explained [here](#5203-deployment-of-an-azure-configuration-created-with-cm-cluster-extension). ## 5.7 Viewing And Setting Azure Generation 1 And Generation 2 Cloud Nodes ### 5.7.1 Introduction Azure introduced Generation 2 (Gen2) VMs in 2019. These boast a number of new features, including increased memory, Intel Software Guard Extensions (Intel SGX), and virtualized persistent memory (vPMEM). Gen2 VMs use the new UEFI-based boot architecture rather than the BIOS-based architecture used by Gen1 VMs. Compared to Gen1 VMs, Gen2 VMs might have improved boot and installation times. It is not possible to change the generation once a VM has been created. The generation to be used must therefore be specified before deploying a cloud instance. [More information on generation 2 VMs can be found at the feature documentation at](https://docs.microsoft.com/en-us/azure/virtual-machines/generation-2) [https://docs.microsoft.com/en-us/azure/virtual-machines/generation-2](https://docs.microsoft.com/en-us/azure/virtual-machines/generation-2). ### 5.7.2 Implementation Of Hyper-V Generation Choice In BCM Additional fields were introduced in NVIDIA Base Command Manager 9.2 to allow the generation to be specified, for the Azure VMSize, CloudProvider, and CloudSettings entities. Also introduced in 9.2 were new “hybrid” node-installer images, which support Gen1 (BIOS-based) and Gen2 (UEFI-based) VMs. The images are partitioned with GPT and contain an ESP partition for UEFI boot. The old, BIOS-only images cannot be used for Gen2 VMs. In the Azure Marketplace BCM provides separate SKUs for Gen1 and Gen2 images, although internally they are the same. If a cloud node instantiated from a non-Marketplace node-installer image VHD, then CMDaemon first creates a generation-specific Azure Image resource out of the VHD in the cluster extension resource group. That image resource is then used to create the VM. Some Azure VM sizes support both Gen1 and Gen2, while others are generation-specific. To see which VM size supports which generation of Hyper-V, cloud types can be listed in `cmsh` with the `hypervgenerations` parameter: **Example** ```bash [basecm10->cloud[azure]]% types [basecm10->cloud[azure]->types]% list -f name,hypervgenerations name (key) hypervgenerations -------------------- -------------------- Basic_A0 V1 Basic_A1 V1 Basic_A2 V1 Basic_A3 V1 Basic_A4 V1 Standard_A0 V1 Standard_A1 V1 … Standard_B12ms V1,V2 Standard_B16ms V1,V2 Standard_B1ls V1,V2 Standard_B1ms V1,V2 ``` Formats for the list command are discussed [here](/admin/clustermanagement#2536-working-with-objects-list-format-sort). Gen1 or Gen2 availability for a cloud is indicated by the tag `V1` or `V2` respectively. For historical reasons, and because of the many VM sizes available, BCM currently (September 2022) creates Gen1 VMs by default. This is not expected to remain the case. Switching to Gen2 from Gen1 is a matter of using a VM size that supports `V2`, and then changing the Hyper-V generation parameter. For the cloud provider the change can be carried out as follows, and all cloud nodes then become Gen2: **Example** ```bash [basecm10->cloud[azure]]% set defaulthyper-vgeneration v2 [basecm10->cloud*[azure*]]% commit [basecm10->cloud[azure]]% ``` Alternatively, each VM can have its Hyper-V generation setting changed within the `cloudsettings` submode: **Example** ```bash [basecm10->device[westeurope-director]->cloudsettings]% get hyper-vgeneration V1 (azure) [basecm10->device[westeurope-director]->cloudsettings]% set hyper-vgeneration v2; commit [basecm10->device[westeurope-director]->cloudsettings]% ``` ## 5.8 Running NVIDIA A100 GPUs On Cloud Nodes GPU hardware detection by the Linux kernel may be very delayed, or even be unsuccessful, on cloud nodes that run NVIDIA A100 GPUs. A100 hardware is made available on some instance types such as, for example, `Standard_NC96ads_A100_v4`. To work around this issue, the `pci-hyperv` and `pci-hyperv-intf` kernel modules must be loaded to the kernel used by cloud node images. In a default Azure setup, this can typically be carried out by adding the kernel modules to the existing kernel modules used by the cloud node category: **Example** ```bash [basecm10->category]% use centralus-cloud-node [basecm10->category[centralus-cloud-node]]% kernelmodules [basecm10->category[centralus-cloud-node]->kernelmodules]% add pci-hyperv [basecm10->category*[centralus-cloud-node*]->kernelmodules*[pci-hyperv*]]% add pci-hyperv-intf [basecm10->category*[centralus-cloud-node*]->kernelmodules*[pci-hyperv-intf*]]% commit [basecm10->category[centralus-cloud-node]->kernelmodules[pci-hyperv-intf]]% [basecm10->category[centralus-cloud-node]->kernelmodules[pci-hyperv-intf]]% exit [basecm10->category[centralus-cloud-node]->kernelmodules]% list Module (key) Parameters ------------------------ ------------------------------------------------ pci-hyperv pci-hyperv-intf [basecm10->category[centralus-cloud-node]->kernelmodules]% ... [notice] basecm10: Initial ramdisk for category centralus-cloud-node based on image default-image is being generated ... [notice] basecm10: Initial ramdisk for category centralus-cloud-node based on image default-image was generated successfully ``` Booting the cloud nodes with these kernel modules configured allows the GPUs to be detected correctly.