NVIDIA Spectrum Ethernet Switches#
Refer to the official documentation for general guidance on installing and upgrading NVIDIA Spectrum Ethernet switches.
Note
If the Ethernet switches are running Cumulus Linux release 5.9 or earlier (including release 4.x), it’s required to install a new binary image to update to 5.10 or later.
There are two options to update Cumulus Linux based Spectrum Ethernet switches:
Via new binary install (i.e., fresh image install).
Via package upgrade (i.e., use package update tool ‘apt’).
New binary install option can be used in all update scenarios while package update is recommended when the release to be updated to is within the same major release (e.g., from 5.10.x to 5.13.x).
From release 5.10, update via package upgrade always updates to the latest available release in the Cumulus Linux repository. To update to a specific release but not the latest release, use the new binary install option.
Prior to release 5.10, package upgrade only supports the current version plus two.
It’s highly recommended to use only NVUE to configure and manage the Spectrum Ethernet switches as NCLU (Network Command Line Utility) has been deprecated and replaced by NVUE.
Follow the Following Steps if the Switch Is NOT Using MLAG#
The following steps are used to update the Cumulus Linux release from 5.9 or earlier to 5.11 using the new binary (image) install option.
Identify switch hardware model and software release version.
cumulus@sjc2-demeter-tor1-r64-u38:mgmt:~$ nv show system operational applied description -------- ------------------------- ------------------------- ------------------------------ hostname sjc2-demeter-tor1-r64-u38 sjc2-demeter-tor1-r64-u38 Static hostname for the switch build Cumulus Linux 5.3.1 system build version uptime 443 days, 0:09:19 system uptime timezone Etc/UTC system time zone cumulus@sjc2-demeter-tor1-r64-u38:mgmt:~$ nv show platform hardware operational applied description ------------- ---------------------------------- ------- ----------------------------------------- asic-model Spectrum-3 System on Chip (SOC) model asic-vendor Mellanox System On Chip (SOC) vendor base-mac 9C:05:91:C9:F8:00 The base mac address provided by eeprom cpu x86_64 Intel Xeon D D-1527 2.20GHz System CPU Arch disk-size 60022167961 B Hard Disk Size manufacturer Mellanox The platform's manufacturer memory 16411041792 B Hardware RAM model MSN4600 The platform's model identifier onie-version 2021.05-5.3.0008-115200 onie version part-number MSN4600-VS2FC System part number platform-name x86_64-mlnx_msn4600-r0 Hardware platform name port-layout 64 x 200G-QSFP56 System port layout product-name MSN4600 Product Name serial-number MT2348XZ039F System serial number system-mac 9c:05:91:c9:f8:fd The MAC provided by eeprom for system-mac
Save the configuration of the switch after applying any pending configuration changes.
cumulus@sjc2-demeter-tor1-r64-u38:mgmt:~$ nv config apply cumulus@sjc2-demeter-tor1-r64-u38:mgmt:~$ nv config save
Save the configuration file (
/etc/nvue.d/startup.yaml) to a location off the switch using scp or sftp.Save the configuration in commands format to a file and then copy the file to a location off the switch. This file can used as a reference of the NVUE commands used to generate the configuration file (
/etc/nvue.d/startup.yaml).nv config show -o commands > backup.config
Download the Cumulus Linux image to a HTTP or FTP location or copy the image to a location on the switch. In this example, the image is copied to the switch.
Run ONIE install by specifying the location of the image file and applying the switch configuration file after image install with the ‘-t’ option. A reboot is required.
sudo onie-install -a -i cumulus-linux-5.11.1-mlx-amd64.bin -t /etc/nvue.d/startup.yaml && sudo reboot
After the new image is installed, the first time you log in the system will ask you to change the password for the ‘cumulus’ user. The default password for the ‘cumulus’ user is ‘cumulus’.
Verify the system is updated.
cumulus@sjc2-demeter-tor1-r64-u38:mgmt:~$ nv show system operational applied pending --------------- -------------------------------------------- ------------------------- ------------------------- build Cumulus Linux 5.11.1 uptime 18:14:54 hostname sjc2-demeter-tor1-r64-u38 sjc2-demeter-tor1-r64-u38 sjc2-demeter-tor1-r64-u38 product-name Cumulus Linux product-release 5.11.1 platform x86_64-mlnx_msn4600-r0 system-memory 3.14 GB used / 8.59 GB free / 15.27 GB total swap-memory 0 Bytes used / 0 Bytes free / 0 Bytes total health-status OK … [output truncated]
Follow the Following Steps if the Switch Is Using MLAG#
Verify the switch is in the secondary role.
cumulus@sjc2-demeter-tor1-r64-u38:mgmt:~$ nv show mlag operational applied pending -------------- ------------------------- ----------------- ----------------- enable on on on mac-address 44:38:39:be:ef:aa 44:38:39:BE:EF:AA 44:38:39:BE:EF:AA peer-ip fe80::9e05:91ff:fec9:f424 linklocal linklocal priority 32768 32768 32768 init-delay 180 180 180 debug off off off [backup] 10.150.48.116 10.150.48.116 10.150.48.116 peer-priority 32768 backup-active True local-id 9c:05:91:c9:f8:24 peer-id 9c:05:91:c9:f4:24 local-role secondary peer-role primary peer-interface peerlink.4094 peer-alive True backup-reason
Identify switch ports used in MLAG.
cumulus@sjc2-demeter-tor1-r64-u38:mgmt:~$ nv show interface peerlink.4094 operational applied description ------------------------ ----------------- -------- ---------------------------------------------------------------------- base-interface peerlink peerlink The interface under this interface type sub sub The type of interface vlan 4094 4094 VLAN ID … [output truncated] cumulus@sjc2-demeter-tor1-r64-u38:mgmt:~$ nv show interface peerlink operational applied description ------------------------ ----------------- -------- ---------------------------------------------------------------------- type bond peerlink The type of interface bond down-delay 0 0 bond down delay lacp-bypass off off lacp bypass lacp-rate fast fast lacp rate mode lacp bond mode up-delay 0 0 bond up delay [member] swp57 swp57 Set of bond members [member] swp58 swp58 mlag enable off Turn the feature 'on' or 'off'. This feature is disabled by default. … [output truncated]
Shut down the switch ports and peerlink used for MLAG, then apply the changes.
cumulus@sjc2-demeter-tor1-r64-u38:mgmt:~$ nv set interface swp57 link state down cumulus@sjc2-demeter-tor1-r64-u38:mgmt:~$ nv set interface swp58 link state down cumulus@sjc2-demeter-tor1-r64-u38:mgmt:~$ nv set interface peerlink link state down cumulus@sjc2-demeter-tor1-r64-u38:mgmt:~$ nv config apply
Follow the steps 1 to 8 in the upgrade instructions section for switches not using MLAG.
Bring up the switch ports and peerlink used for MLAG, verify the MLAG setup, then apply and save the changes.
cumulus@sjc2-demeter-tor1-r64-u38:mgmt:~$ nv set interface swp57 link state up cumulus@sjc2-demeter-tor1-r64-u38:mgmt:~$ nv set interface swp58 link state up cumulus@sjc2-demeter-tor1-r64-u38:mgmt:~$ nv set interface peerlink link state up cumulus@sjc2-demeter-tor1-r64-u38:mgmt:~$ nv config apply cumulus@sjc2-demeter-tor1-r64-u38:mgmt:~$ nv config save
Make this switch as the primary role in MLAG.
cumulus@sjc2-demeter-tor1-r64-u38:mgmt:~$ nv set mlag priority 2084
Verify the other switch is now in the secondary role.
Repeat the steps above to upgrade the now secondary role MLAG switch.
Remove the priority 2048 and restore the priority back to 32768 on the current primary switch.
cumulus@sjc2-demeter-tor1-r64-u38:mgmt:~$ nv set mlag priority 32768