> This page is for version 11 (default). > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # NVLink Switch Integration The NVIDIA GB200 server platforms have 9 NVLink switch trays per rack. An NVL72 reference rack configuration for an NVIDIA DGX SuperPOD configuration is described at:\ [https://docs.nvidia.com/dgx-superpod/reference-architecture-scalable-infrastructure-gb200/latest/dgx-superpod-components.html#nvlink-switch-tray](https://docs.nvidia.com/dgx-superpod/reference-architecture-scalable-infrastructure-gb200/latest/dgx-superpod-components.html#nvlink-switch-tray) The switch trays connect GPUs in each rack (36 per rack, or 72 per rack) to each other on a 400 Gb/s InfiniBand NDR network. This chapter considers an NVIDIA GB200 server platform that has been configured to work with BCM. ## 8.1 NVLink Switch Listing And Overview * The `list -t switch` command in `device` mode can be used to list the all switches of a configured cluster. * The `list --field kind=nvlink` command in `device` mode can be used to list only the NVLink switches of a configured cluster. NVLink switches are configured as devices on a BMC network, as indicated by the following example: **Example** ```bash [maple->device]% list -t switch Type Hostname (key) MAC IP Network Status -------- --------------------- ------------------ ------------- ---------- ---------- Switch dgx-gb200-m06-nvlsw1 E0:9D:73:05:70:6C 10.140.0.45 ipminet0 [ UP ] Switch dgx-gb200-m07-nvlsw1 B8:E9:24:B9:84:8C 10.140.0.48 ipminet0 [ UP ] Switch dgx-gb200-n01-nvlsw1 E0:9D:73:05:70:5C 10.140.0.51 ipminet0 [ UP ] Switch dgx-gb200-n07-nvlsw1 E0:9D:73:05:70:AC 10.140.0.54 ipminet0 [ UP ] ... ``` A particular NVLink switch can be looked at with: **Example** ```bash [basecm11->device[dgx-gb200-m06-nvlsw1]]% switchoverview Device : dgx-gb200-m06-nvlsw1 State : [ UP ] Model : N5400_LD Serial number : MT1234556789 Part number : 123-4567-89 Port assignment: Port Name Status Uplink Speed RX TX ------ ------- ------ ------ ---------- --------- --------- 1 acp1 UP no 400 Gb/s 64 TiB 64 TiB 2 acp2 UP no 400 Gb/s 64 TiB 64 TiB ... 143 acp143 UP no 400 Gb/s 64 TiB 64 TiB 144 acp144 UP no 400 Gb/s 64 TiB 64 TiB 145 eth0 UP no 1.00 Gb/s 1.22 TiB 6.0 GiB 146 eth1 DOWN no 0 b/s 4.5 KiB 28.7 KiB 147 fnm1 UP no 100 Gb/s 0 B 0 B 148 fnm2 UP no 100 Gb/s 0 B 0 B ``` ## 8.2 NVLink Switch Configuration Each NVLink switch has two redundant ethernet interfaces and a Redfish interface. **Example** ```bash [basecm11->device[a05-p1-nvsw-01]->interfaces]% list Type Network device name IP Network ------------ -------------------- ---------------- ----------- bmc rf0 7.241.3.21 ipminet2 physical eth0 7.241.3.1 ipminet2 physical eth1 7.241.3.11 ipminet2 ``` An NVLink switch runs NVOS, and can be configured to boot using ZTP. The JSON template ZTP file that is provided by BCM is used to check the configured image, and to install `cm-lite-daemon` ([section 2.6.7 of the *Administrator Manual*](/admin/clustermanagement#267-cmdaemon-lite)). If ZTP is configured correctly before the switch first boots, with the default of `Install lite daemon` set to a value of `yes`, then `cm-lite-daemon` installs and the switch is ready for use. **Example** ```bash [basecm11->device[a05-p1-nvsw-01]->ztpsettings]% show Parameter Value ------------------------------------ ------------------------------------------------ Revision Script template nvos-ztp.sh JSON template nvlink-nvos.json Image nvos-amd64-25.02.2141.bin Check image on boot no Run ZTP on each boot yes Install lite daemon yes Enable API yes ``` If an NVLink switch is already booted and is running without `cm-lite-daemon`, or has ZTP disabled, then `cm-lite-daemon` can be installed manually using the `cmsh` utility `litedaemon`. The `litedaemon` utility is a convenient `.deb` package manager for `cm-lite-daemon` for switches for such cases, and uses APT in the backend. The `litedaemon` utility is used with options to download, install, update, and remove the packages: * `download`: This option downloads packages to under `/cm/local/apps/cmd/etc/htdocs/switch/packages/`. If the nodes are not specified, then the downloads are placed on the head node. The download needs to be executed once for all switches. If updates are available in the BCM repositories, then the download should be repeated and the subsequent installation process should also be repeated. **Example** ```bash [basecm11->device]% litedaemon download Download all/cm-config-cm_1trunk_all.deb for 2004: done Download all/cm-lite-daemon_1master-100775-cm-752a72d739_all.deb for 2004: done Download amd64/cm-openssl_3.1.8-100204-cm-a2b95b5141_amd64.deb for 2004: done Download amd64/cm-python312_3.12.11-100035-cm-f4ae26a924_amd64.deb for 2004: done Download amd64/cm-python3_1master-100155-cm-eaf713f03f_amd64.deb for 2004: done ... ``` * `install`: This option installs the downloaded packages on the NVLink switches. It is recommended to do all 9 switches for a rack at the same time; multiple racks can be done at once. **Example** ```bash [basecm11->device]% litedaemon install --rack a05 --intersection --field kind=nvlink success: yes --- stdout --- run: /home/admin/cm-lite-daemon-install.py --head-ips 10.141.255.254 --port 8081 --hostname a05-p1-nvsw-01 --interface eth0 + ping -c 1 10.141.255.254 Using IP 10.141.255.254 without supplied vrf * Download: https://10.141.255.254:8081/switch/packages/2404/files * * Download: https://10.141.255.254:8081/switch/packages/2404/cm-config-cm_11.0_all.deb * * Download: https://10.141.255.254:8081/switch/packages/2404/cm-lite-daemon_1master- 100775-cm-752a72d739_all.deb * * Download: https://10.141.255.254:8081/switch/packages/2404/cm-openssl_3.1.8-100204- cm-a2b95b5141_amd64.deb * * Download: https://10.141.255.254:8081/switch/packages/2404/cm-python312_3.12.11- 100035-cm-f4ae26a924_amd64.deb * * Download: https://10.141.255.254:8081/switch/packages/2404/cm-python3_1master-100155- cm-eaf713f03f_amd64.deb * * Install * * Setup * * Setup from /home/admin * * Certificates * - cluster.pem - bootstrap.pem - bootstrap.key * Register * * Done * ``` * `--update`: This option updates packages on the switches if updated packages have been downloaded: **Example** ```bash [basecm11->device]% litedaemon install --update --field kind=nvlink run: /home/admin/cm-lite-daemon-install.py --head-ips 10.141.255.254 --port 8081 --hostname a05-p1-nvsw-01 --interface eth0 --update + ping -c 1 10.141.255.254 Using IP 10.141.255.254 without supplied vrf * Download: https://10.141.255.254:8081/switch/packages/2404/files * * Download: https://10.141.255.254:8081/switch/packages/2404/cm-config-cm_1trunk _all.deb * * Download: https://10.141.255.254:8081/switch/packages/2404/cm-lite-daemon_1master- 100775-cm-752a72d739_all.deb * * Download: https://10.141.255.254:8081/switch/packages/2404/cm-openssl_3.1.8-100204- cm-a2b95b5141_amd64.deb * * Download: https://10.141.255.254:8081/switch/packages/2404/cm-python312_3.12.11- 100035-cm-f4ae26a924_amd64.deb * * Download: https://10.141.255.254:8081/switch/packages/2404/cm-python3_1master- 100155-cm-eaf713f03f_amd64.deb * * Install * * Restart * ``` * `remove`: All `cm-lite-daemon`-related packages can be removed with: **Example** ```bash [basecm11->device]% litedaemon remove --update --field kind=nvlink * Systemctl stop * * Remove * * Systemctl reload * ``` ## 8.3 NVLink Switch Monitoring NVLink switches are monitored in two ways. * OS data using `cm-lite-daemon` running on the switch * Redfish sampled from the active head node (currently only leak information is reported) All data can be viewed using standard BCM monitoring ```bash [basecm11->device[a05-p1-nvsw-01]]% latestmonitoringdata ... LoadOne 3.21 33s RF_LD_LeakDetection OK 1s Rollup: OK RF_LeakDetector leakage1 OK 1s Detector: OK, Health: OK, State: Enabled RF_LeakDetector leakage2 OK 1s Detector: OK, Health: OK, State: Enabled RF_LeakDetector leakage3 OK 1s Detector: OK, Health: OK, State: Enabled RF_LeakDetector leakage4 OK 1s Detector: OK, Health: OK, State: Enabled RF_LeakDetector leakage5 OK 1s Detector: OK, Health: OK, State: Enabled RF_LeakDetector leakage6 OK 1s Detector: OK, Health: OK, State: Enabled RF_LeakDetector leakage_aggr OK 1s Detector: OK, Health: OK, State: Enabled ``` ## 8.4 NVLink Switch Redfish Events NVLink switches report leaks to all tools that are subscribed to Redfish events. BCM does this for all NVLink switches if the Redfish IP address and BMC settings are configured correctly. The `redfishsubscription` command lists the devices that have a Redfish subscription ([here](/nvidia-mission-control/mcmetricshealthchecksactions#128-redfish-metrics-for-the-dgx-gb200)). ```bash [basecm11->device[a05-p1-nvsw-01]]% redfishsubscription hostname id ip kind port subscribed --------------- ------ ----------- -------- -------- ---------- a05-p1-nvsw-01 1234 7.241.3.21 GB200SW 443 true ``` Testing of the event subscription work can be done from inside `cmsh`. ```bash [basecm11->device[a05-p1-nvsw-01]]% redfishevent --message Test ... ```