Skip to navigation

NVLink Switch Integration

View as Markdown

The NVIDIA GB200 server platforms have 9 NVLink switch trays per rack. An NVL72 reference rack configuration for an NVIDIA DGX SuperPOD configuration is described at:
https://docs.nvidia.com/dgx-superpod/reference-architecture-scalable-infrastructure-gb200/latest/dgx-superpod-components.html#nvlink-switch-tray

The switch trays connect GPUs in each rack (36 per rack, or 72 per rack) to each other on a 400 Gb/s InfiniBand NDR network.

This chapter considers an NVIDIA GB200 server platform that has been configured to work with BCM.

  • The list -t switch command in device mode can be used to list the all switches of a configured cluster.

  • The list --field kind=nvlink command in device mode can be used to list only the NVLink switches of a configured cluster.

NVLink switches are configured as devices on a BMC network, as indicated by the following example:

Example

[maple->device]% list -t switch
Type     Hostname (key)        MAC                 IP            Network    Status
-------- --------------------- ------------------  ------------- ---------- ----------
Switch   dgx-gb200-m06-nvlsw1  E0:9D:73:05:70:6C   10.140.0.45   ipminet0   [   UP   ]
Switch   dgx-gb200-m07-nvlsw1  B8:E9:24:B9:84:8C   10.140.0.48   ipminet0   [   UP   ]
Switch   dgx-gb200-n01-nvlsw1  E0:9D:73:05:70:5C   10.140.0.51   ipminet0   [   UP   ]
Switch   dgx-gb200-n07-nvlsw1  E0:9D:73:05:70:AC   10.140.0.54   ipminet0   [   UP   ]
...

A particular NVLink switch can be looked at with:

Example

[basecm11->device[dgx-gb200-m06-nvlsw1]]% switchoverview
Device         : dgx-gb200-m06-nvlsw1
State          : [   UP   ]
Model          : N5400_LD
Serial number  : MT1234556789
Part number    : 123-4567-89
Port assignment:
Port   Name    Status Uplink Speed      RX        TX
------ ------- ------ ------ ---------- --------- ---------
1      acp1    UP     no     400 Gb/s   64 TiB    64 TiB
2      acp2    UP     no     400 Gb/s   64 TiB    64 TiB
...
143    acp143  UP     no     400 Gb/s   64 TiB    64 TiB
144    acp144  UP     no     400 Gb/s   64 TiB    64 TiB
145    eth0    UP     no     1.00 Gb/s  1.22 TiB  6.0 GiB
146    eth1    DOWN   no     0 b/s      4.5 KiB   28.7 KiB
147    fnm1    UP     no     100 Gb/s   0 B       0 B
148    fnm2    UP     no     100 Gb/s   0 B       0 B

Each NVLink switch has two redundant ethernet interfaces and a Redfish interface.

Example

[basecm11->device[a05-p1-nvsw-01]->interfaces]% list
Type         Network device name  IP               Network
------------ -------------------- ---------------- -----------
bmc          rf0                  7.241.3.21       ipminet2
physical     eth0                 7.241.3.1        ipminet2
physical     eth1                 7.241.3.11       ipminet2

An NVLink switch runs NVOS, and can be configured to boot using ZTP. The JSON template ZTP file that is provided by BCM is used to check the configured image, and to install cm-lite-daemon (section 2.6.7 of the Administrator Manual). If ZTP is configured correctly before the switch first boots, with the default of Install lite daemon set to a value of yes, then cm-lite-daemon installs and the switch is ready for use.

Example

[basecm11->device[a05-p1-nvsw-01]->ztpsettings]% show
Parameter                            Value
------------------------------------ ------------------------------------------------
Revision
Script template                      nvos-ztp.sh
JSON template                        nvlink-nvos.json
Image                                nvos-amd64-25.02.2141.bin
Check image on boot                  no
Run ZTP on each boot                 yes
Install lite daemon                  yes
Enable API                           yes

If an NVLink switch is already booted and is running without cm-lite-daemon, or has ZTP disabled, then cm-lite-daemon can be installed manually using the cmsh utility litedaemon. The litedaemon utility is a convenient .deb package manager for cm-lite-daemon for switches for such cases, and uses APT in the backend. The litedaemon utility is used with options to download, install, update, and remove the packages:

  • download: This option downloads packages to under /cm/local/apps/cmd/etc/htdocs/switch/packages/. If the nodes are not specified, then the downloads are placed on the head node.

    The download needs to be executed once for all switches. If updates are available in the BCM repositories, then the download should be repeated and the subsequent installation process should also be repeated.

    Example

    [basecm11->device]% litedaemon download
    Download all/cm-config-cm_1trunk_all.deb for 2004: done
    Download all/cm-lite-daemon_1master-100775-cm-752a72d739_all.deb for 2004: done
    Download amd64/cm-openssl_3.1.8-100204-cm-a2b95b5141_amd64.deb for 2004: done
    Download amd64/cm-python312_3.12.11-100035-cm-f4ae26a924_amd64.deb for 2004: done
    Download amd64/cm-python3_1master-100155-cm-eaf713f03f_amd64.deb for 2004: done
    ...

  • install: This option installs the downloaded packages on the NVLink switches. It is recommended to do all 9 switches for a rack at the same time; multiple racks can be done at once.

    Example

    [basecm11->device]% litedaemon install --rack a05 --intersection --field kind=nvlink
    success: yes
    --- stdout ---
    run: /home/admin/cm-lite-daemon-install.py --head-ips 10.141.255.254 --port 8081
            --hostname a05-p1-nvsw-01 --interface eth0
    + ping -c 1 10.141.255.254
    Using IP 10.141.255.254 without supplied vrf
    * Download: https://10.141.255.254:8081/switch/packages/2404/files *
    * Download: https://10.141.255.254:8081/switch/packages/2404/cm-config-cm_11.0_all.deb *
    * Download: https://10.141.255.254:8081/switch/packages/2404/cm-lite-daemon_1master-
            100775-cm-752a72d739_all.deb *
    * Download: https://10.141.255.254:8081/switch/packages/2404/cm-openssl_3.1.8-100204-
            cm-a2b95b5141_amd64.deb *
    * Download: https://10.141.255.254:8081/switch/packages/2404/cm-python312_3.12.11-
            100035-cm-f4ae26a924_amd64.deb *
    * Download: https://10.141.255.254:8081/switch/packages/2404/cm-python3_1master-100155-
            cm-eaf713f03f_amd64.deb *
    * Install *
    * Setup *
    * Setup from /home/admin *
    * Certificates *
      - cluster.pem
      - bootstrap.pem
      - bootstrap.key
    * Register *
    * Done *

  • --update: This option updates packages on the switches if updated packages have been downloaded:

    Example

    [basecm11->device]% litedaemon install --update --field kind=nvlink
    run: /home/admin/cm-lite-daemon-install.py --head-ips 10.141.255.254 --port 8081
            --hostname a05-p1-nvsw-01 --interface eth0 --update
    + ping -c 1 10.141.255.254
    Using IP 10.141.255.254 without supplied vrf
    * Download: https://10.141.255.254:8081/switch/packages/2404/files *
    * Download: https://10.141.255.254:8081/switch/packages/2404/cm-config-cm_1trunk
            _all.deb *
    * Download: https://10.141.255.254:8081/switch/packages/2404/cm-lite-daemon_1master-
            100775-cm-752a72d739_all.deb *
    * Download: https://10.141.255.254:8081/switch/packages/2404/cm-openssl_3.1.8-100204-
            cm-a2b95b5141_amd64.deb *
    * Download: https://10.141.255.254:8081/switch/packages/2404/cm-python312_3.12.11-
            100035-cm-f4ae26a924_amd64.deb *
    * Download: https://10.141.255.254:8081/switch/packages/2404/cm-python3_1master-
            100155-cm-eaf713f03f_amd64.deb *
    * Install *
    * Restart *

  • remove: All cm-lite-daemon-related packages can be removed with:

    Example

    [basecm11->device]% litedaemon remove --update --field kind=nvlink
    * Systemctl stop *
    * Remove *
    * Systemctl reload *

NVLink switches are monitored in two ways.

  • OS data using cm-lite-daemon running on the switch

  • Redfish sampled from the active head node (currently only leak information is reported)

All data can be viewed using standard BCM monitoring

[basecm11->device[a05-p1-nvsw-01]]% latestmonitoringdata
...
LoadOne                               3.21   33s
RF_LD_LeakDetection                   OK      1s    Rollup: OK
RF_LeakDetector        leakage1       OK      1s    Detector: OK, Health: OK, State: Enabled
RF_LeakDetector        leakage2       OK      1s    Detector: OK, Health: OK, State: Enabled
RF_LeakDetector        leakage3       OK      1s    Detector: OK, Health: OK, State: Enabled
RF_LeakDetector        leakage4       OK      1s    Detector: OK, Health: OK, State: Enabled
RF_LeakDetector        leakage5       OK      1s    Detector: OK, Health: OK, State: Enabled
RF_LeakDetector        leakage6       OK      1s    Detector: OK, Health: OK, State: Enabled
RF_LeakDetector        leakage_aggr   OK      1s    Detector: OK, Health: OK, State: Enabled

NVLink switches report leaks to all tools that are subscribed to Redfish events. BCM does this for all NVLink switches if the Redfish IP address and BMC settings are configured correctly.

The redfishsubscription command lists the devices that have a Redfish subscription (here).

[basecm11->device[a05-p1-nvsw-01]]% redfishsubscription
hostname        id     ip          kind     port     subscribed
--------------- ------ ----------- -------- -------- ----------
a05-p1-nvsw-01  1234   7.241.3.21  GB200SW  443      true

Testing of the event subscription work can be done from inside cmsh.

[basecm11->device[a05-p1-nvsw-01]]% redfishevent --message Test
...