NVSM Multinode Provisioning (BCM DGX SuperPOD)

This section describes the steps to provision a NVSM multinode on a BCM managed DGX SuperPOD.

All DGX systems belong under the same category, using the same software image.

NVSM Multinode Overview

In this configuration, NVSM Aggregator runs on BCM headnode, connects to DGX nodes managed by BCM.

DGX Compute Node Config

Setting Exclude List

During NVSM multinode provisioning, NVSM node provisioner will push mutlinode configuration files to all compute nodes. To make sure these configurations are persistent across reboots, it’s required to add these configuration files to BCM exclude list.

$ cmsh -c "category use <category>; set excludelistfullinstall; commit

An editor will open, please add the following to the existing exclude list:

- /etc/nvsm/mn/*
no-new-files: - /etc/nvsm/mn/*

Repeat the same procedure for excludelistsyncinstall and excludelistupdate

$ cmsh -c "category use <category>; set excludelistsyncinstall; commit
$ cmsh -c "category use <category>; set excludelistupdate; commit

NVSM Version Requirements

NVSM Multinode feature is supported by NVSM >= 24.03.05, however we recommand version 26.03.01 or above.

Please update the software image used by BCM to include NVSM >= 26.03.01

$ cm-chroot-sw-img /cm/images/<software_image_name>

# Now we have chroot into the software image
$ apt update
$ apt install nvsm
$ dpkg -l nvsm           # make sure nvsm version is 25.06.01 or above
$ exit                   # exit from chroot

After updating the software image, make sure to restart DGX nodes to apply updated software image.

BCM Head Node Config

NVSM Aggregator run as docker containers, orchestrated by docker-compose. It uses port 8883 for MQTT server which communicates with DGX nodes; and optinally port 9123 for nvsm-exporter, port 3000 for grafana dashboard.

Install Docker and Docker-Compose

$ sudo apt update
$ sudo apt install docker
$ sudo apt install docker-compose

Optional, Open Required Ports on BCM Headnode

These steps are required only if BCM headnode has firewall (Shorewall) enabled, if firewall is disabled, please skip to next section.

Check if shorewall is running:

$ sudo shorewall status

If Shorewall is not running, this section should be skipped.

Ensure Docker support is enabled in Shorewall by editing /etc/shorewall/shorewall.conf, setting DOCKER=Yes, and restarting Shorewall:

$ sudo shorewall restart

Assign firewall role to BCM headnode if not already assigned, firewall role is required to modify shorewall configuration:

$ cmsh
[bcm]% device
[bcm->device]% use myheadnode
[bcm->device[myheadnode]]% roles
[bcm->device[myheadnode]->roles]% list

# if firewall role is not listed, assign it
[bcm->device[myheadnode]->roles]% assign firewall
[bcm->device[myheadnode]->roles]% commit

[bcm->device[myheadnode]->roles]% use firewall
[bcm->device[myheadnode]->roles[firewall]]%

Open required ports

The example below opens port 8883 for MQTT server communication. If nvsm-exporter and grafana are also used, please open ports 9123 and 3000 as well.

[bcm->device*[myheadnode*]->roles*[firewall*]]% openports
[bcm->device*[myheadnode*]->roles*[firewall*]->openports]% add ACCEPT net 8883 tcp fw
[bcm->device*[myheadnode*]->roles*[firewall*]->openports[0]]% exit
[bcm->device*[myheadnode*]->roles*[firewall*]]% set shorewall on
[bcm->device*[myheadnode*]->roles*[firewall*]]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Name                             firewall
Revision
Type                             FirewallRole
Add services                     yes
Shorewall                        yes
Open ports                       <0 in submode>
Zones                            <0 in submode>
Interfaces                       <0 in submode>
Policies                         <1 in submode>
[bcm->device*[myheadnode*]->roles*[firewall*]]% commit
[bcm->device[myheadnode]->roles[firewall]->policies]%

NVSM Multinode Provisioning

The provisioning steps are similar with non BCM setup, however some config options in the inventory file need to be adjusted to work with BCM.

Prepare Inventory File

Please refer to docs/nvsm-user-guide/nvsm-multinode.rst for details about preparing inventory file for NVSM multinode provisioning.

When preparing inventory file for BCM managed DGX nodes, please make the following changes:

Use docker host network to bypass BCM container network restriction, this can be done by setting nvsm_aggregator_network to host.
Remove ansible_ssh_pass and ansible_sudo_pass, BCM have set up key based authentication for SSH, allowing the headnode to access DGX nodes without password.
Add /cm/shared/nvsm:/cm/shared/nvsm to nvsm_aggregator_volumes to allow NVSM aggregator container to access the BCM shared storage.
Specify nvsm_dump_directory: /cm/shared/nvsm and nvsm_collect_policy: none, so that nvsm dump health and nvsm stress-test save the result to the shared storage, without needing to transfer the result back.

Here is an example inventory file for BCM managed DGX nodes:

aggregator:
   hosts:
      bcm-headnode.example.com: {}
   vars:
      # Unified multinode image used for aggregator, prometheus, and grafana
      nvsm_multinode_image: "nvcr.io/nvstaging/nvsm/nvsm-multinode:26.03.1"

      # Aggregator
      nvsm_exporter_port: 9123
      nvsm_api_gateway_port: 273
      nvsm_aggregator_container: "nvsm-aggregator"   # Name of nvsm container
      nvsm_aggregator_network: "host"                # Use docker host network
      nvsm_aggregator_volumes: []                    # Extra volumes to mount, e.g. ["/data:/data"]

      # Use BCM shared storage for nvsm data
      nvsm_aggregator_containers:
         - "/cm/shared/nvsm:/cm/shared/nvsm"
      nvsm_dump_directory: "/cm/shared/nvsm"
      nvsm_collect_policy: "none"

      # Dashboard stack - Prometheus & Grafana
      nvsm_enable_dashboard: true
      nvsm_grafana_port: 3000
      nvsm_grafana_admin_user: "nvsm"
      nvsm_grafana_admin_password: "nvsm"

compute:
   hosts:
      dgx01.example.com:
         nvsm_id: 1
         bmc_ip: "192.168.10.1"
      dgx02.example.com:
         nvsm_id: 2
         bmc_ip: "192.168.10.2"
   vars:
      bmc_user: "admin"
      bmc_pass: "admin"

all:
   children:
      compute:
      aggregator:
   vars:
      # Aggregrator variables
      nvsm_mqtt_host: "{{ groups['aggregator'] | first }}"
      nvsm_mqtt_port: 8883

      # SSH user
      ansible_user: "root"

      # Force reprovision
      nvsm_force_reprovision: true

Start Provisioning

To leverage BCM’s SSH key, please add -v /root/.ssh:/root/.ssh in the provision command, the full command will look like below:

$ sudo docker run --rm -e process=pvsn -v /etc/nvsm/aggregator:/etc/nvsm/mn -v /root/.ssh:/root/.ssh nvcr.io/nvstaging/nvsm/nvsm-multinode:26.03.1

NVSM multinode should be provisioned successfully after running the command above, and NVSM aggregator container should be able to communicate with DGX nodes through MQTT protocol. Please refer to docs/nvsm-user-guide/nvsm-multinode.rst for details about how to verify the multinode status and how to use NVSM multinode features.