Manual Addition of NVLink Switch Rack Entries#

This document provides step-by-step instructions for manually adding NVLink Switches to BCM with full Zero Touch Provisioning (ZTP) settings. The process configures a switch to be added into BCM, installs the NVOS software, applies a startup file, and installs the cm-lite-daemon package.

Prerequisites#

For adding NVLink Switches to BCM, ensure you have the following:

MAC address for eth0/COMe0 of the NVLink Switch
IP address for eth0/COMe0 of the NVLink Switch
Management network for eth0/COMe0 of the NVLink Switch
Username and password for SSH access to the NVLink Switch
If eth1/COMe1 is connected instead, obtain the same information for eth1/COMe1

For setting up BCM to perform ZTP for NVLink Switches, you will also need:

NVOS firmware image files
Startup configuration files or templates

Adding NVLink Switch Entries to BCM#

This section covers the basic steps to add NVLink switch entries to BCM’s device database. These steps create the switch device entry and configure basic network connectivity.

Add the switch in BCM, enable cm-lite-daemon, create a new interface using the MAC address and an assigned IP.

cmsh -c "device; add switch <switch_hostname>; set mac <00:00:00:00:00:00>; set hasclientdaemon yes; set kind nvlink; set disablesnmp yes; commit"

Add eth0 interface and set network configuration.

cmsh -c "device; use <switch_hostname>; interfaces; add physical eth0; set ip <switch_ip>; set network <management_network>; commit"

Set management network and verify interface configuration.

cmsh -c "device; use <switch_hostname>; set managementnetwork <management_network>; commit"

Assign the username and password for SSH access.

cmsh -c "device; use <switch_hostname>; accesssettings; set username admin; set password <password>; set restport 443; commit"

ZTP Settings Configuration in BCM (Optional)#

To configure Zero Touch Provisioning (ZTP) to automatically update NVOS firmware and apply startup configurations, follow these steps to configure the ZTP settings in BCM. These settings will be applied to all cloned switch entries.

Enable the API on the switch and create the necessary directory structure.

cmsh -c "device; use <switch_hostname>; ztpsettings; set scripttemplate nvos-ztp.sh; set jsontemplate nvlink-nvos.json; set enableapi yes; set installlitedaemon no; commit; exit; initialize"

Note

The initialize command creates a directory for every NVLink switch entry at /cm/local/apps/cmd/etc/htdocs/switch/<switch_name>/. This directory will be used to store startup configuration files and other switch-specific files.

Copy the startup configuration and image files into the BCM repositories.
# Copy image files to image directory cp <nvos_image_file> /cm/local/apps/cmd/etc/htdocs/switch/image/ # Copy startup configuration to switch-specific directory # (Created after the initialize step) cp <startup_config_file> /cm/local/apps/cmd/etc/htdocs/switch/<switch_name>/
File locations:
- Image directory: /cm/local/apps/cmd/etc/htdocs/switch/image
- Startup configuration directory: /cm/local/apps/cmd/etc/htdocs/switch/<switch_name>
Note

The startup configuration directory is created automatically after running the initialize command in the previous step.

Modify the startup file to include the password field with hashed password from accesssettings.

Edit the startup YAML file to include the following in the password field:

- set:
    system:
      aaa:
        user:
          admin:
            hashed-password: '${CMD_ACCESS_HASHED_PASSWORD}'

Sample complete startup.yaml configuration:

- header:
    model: n5700
    nvue-api-version: nvue_v1
    rev-id: 1.0
    version: 25.02.4220
- set:
    interface:
      eth0-1:
        acl:
          ACL_MGMT_INBOUND_CP_DEFAULT:
            inbound:
              control-plane: {}
          ACL_MGMT_INBOUND_CP_DEFAULT_IPV6:
            inbound:
              control-plane: {}
          ACL_MGMT_INBOUND_DEFAULT:
            inbound: {}
          ACL_MGMT_INBOUND_DEFAULT_IPV6:
            inbound: {}
          ACL_MGMT_OUTBOUND_CP_DEFAULT:
            outbound:
              control-plane: {}
          ACL_MGMT_OUTBOUND_CP_DEFAULT_IPV6:
            outbound:
              control-plane: {}
        type: eth
      lo:
        acl:
          ACL_LOOPBACK_INBOUND_CP_DEFAULT:
            inbound:
              control-plane: {}
          ACL_LOOPBACK_INBOUND_CP_DEFAULT_IPV6:
            inbound:
              control-plane: {}
        type: loopback
    system:
      aaa:
        user:
          admin:
            hashed-password: '${CMD_ACCESS_HASHED_PASSWORD}'

Add additional parameters to the device in BCM to apply the startup file and image updates during the ZTP process.

cmsh -c "device; use <switch_hostname>; set nvconfigurationmode file; set nvconfigurationfile <startup_yaml>; ztpsettings; set jsontemplate nvlink-nvos.json; set fm_config_file <fm_config_file>; set image <NVOS_image_name>; set checkimageonboot yes; exit; commit; initialize"

 cmsh
 device
 use <switch_hostname>

 # Define the startup files matching the name of the file from the folder
 set nvconfigurationmode file
 set nvconfigurationfile <startup_yaml>

 # set the fm_config_file so that an NVLink Switch leader can be elected by cm-lite-daemon
 set fm_config_file gb200_nvl72r1_c2g4.cfg

# Define the ZTP Settings
 ztpsettings
 set jsontemplate nvlink-nvos.json
 set image <NVOS_image_name>
 set checkimageonboot yes

 exit
 commit
 initialize
 quit

Note

The minimum version of NVOS required for the NVLink Switch is 25.02.2134 to support custom scripts that will do the cm-lite-daemon installation.
The Run ZTP on each boot option, when set to Yes, will enable cm-lite-daemon to enable ZTP if it is disabled.

Clone NVLink Switch Entries#

After completing the ZTP settings configuration for your golden switch, clone additional NVLink Switch entries for the rack. The ZTP settings will be inherited by all cloned entries.

For 9 NVLink switches in a rack:
foreach -o <golden_switch_hostname> -n <switch_hostname_02>..<switch_hostname_09> --next-ip ()
Example:
foreach -o a05-18-p1-nvsw-01 -n a05-19-p1-nvsw-02..a05-26-p1-nvsw-09 --next-ip ()
commit
Note

IPs and MACs will have to be updated manually for each cloned entry

The ZTP settings configured in the previous section will be applied to all cloned switches

Each switch will automatically receive the same NVOS firmware and startup configurations during ZTP

Once all the switches are added to BCM, the following commands can be used to verify the configuration.

Verification Commands#

After completing the configuration for all switches:

List all NVLink switches and check their status:
cmsh -c "device; list -t switch |grep -i nvsw"
Verify ZTP completion:
cmsh -c "device; use <switch_hostname>; ssh"
sudo ztp status
Check lite daemon status:
cmsh -c "device; use <switch_hostname>; latesthealthdata"

Troubleshooting#

Common issues and solutions:

ZTP fails: Check network connectivity, verify image and configuration file paths
SSH access fails: Verify accesssettings configuration and network connectivity
Lite daemon installation fails: Check network access to BCM head node, verify package availability
Switch not responding: Verify switch is powered on, check BMC connectivity for remote restart

Log files to check:

/var/log/cmdaemon - BCM daemon logs
/var/log/switch/<switch_name> - Switch-specific ZTP logs