Switch Configurations#

This documentation is part of NVIDIA DGX BasePOD: Deployment Guide Featuring NVIDIA DGX A100 Systems.

Switch configuration files are captured in this section.

SN4600 #1 (In-band Management Switch)#

  1# Note Make sure to update the IP addresses in the sample config below
  2# eth0 mgmt interface configs
  3nv set interface eth0 ip address 10.227.20.78/26
  4nv set interface eth0 ip gateway 10.227.20.65
  5nv set interface eth0 ip vrf mgmt
  6nv set interface eth0 type eth
  7
  8# Creating SVI interfaces and adding the VLANs to the bridge
  9# Note Make sure to update the IP addresses
 10nv set bridge domain br_default vlan 122
 11nv set bridge domain br_default vlan 121
 12nv set interface vlan121 type svi
 13nv set interface vlan121 ip vrr mac-address 44:38:39:ff:ff:ff
 14nv set interface vlan121 ip vrr address 10.227.52.1/26
 15nv set interface vlan121 ip address 10.227.52.2/26
 16nv set interface vlan122 type svi
 17nv set interface vlan122 ip vrr mac-address 44:38:39:ff:ff:ff
 18nv set interface vlan122 ip vrr address 10.227.48.1/26
 19nv set interface vlan122 ip address 10.227.48.2/26
 20
 21# MLAG configs
 22nv set interface peerlink bond member swp57
 23nv set interface peerlink bond member swp58
 24nv set interface peerlink type peerlink
 25nv set interface peerlink.4094 base-interface peerlink
 26nv set interface peerlink.4094 type sub
 27nv set interface peerlink.4094 vlan 4094
 28nv set mlag backup 10.227.20.79 vrf mgmt
 29nv set mlag enable on
 30nv set mlag mac-address 44:38:39:ff:ff:ff
 31nv set mlag peer-ip linklocal
 32nv set mlag priority 2048
 33
 34### bcm headnode 01
 35nv set interface swp1 bridge domain br_default access 121
 36### bcm headnode 02
 37nv set interface swp2 bridge domain br_default access 121
 38
 39### bond4-20 used to connect to DGX and Kubernetes master nodes
 40nv set interface bond4 bond member swp4
 41nv set interface bond4 bond mlag id 4
 42nv set interface bond4 bridge domain br_default untagged 122
 43nv set interface bond4 bridge domain br_default vlan all
 44nv set interface bond4 bond mlag enable on
 45nv set interface bond4 bond lacp-bypass on
 46nv set interface bond5 bond member swp5
 47nv set interface bond5 bond mlag id 5
 48nv set interface bond5 bridge domain br_default untagged 122
 49nv set interface bond5 bridge domain br_default vlan all
 50nv set interface bond5 bond mlag enable on
 51nv set interface bond5 bond lacp-bypass on
 52nv set interface bond6 bond member swp6
 53nv set interface bond6 bond mlag id 6
 54nv set interface bond6 bridge domain br_default untagged 122
 55nv set interface bond6 bridge domain br_default vlan all
 56nv set interface bond6 bond mlag enable on
 57nv set interface bond6 bond lacp-bypass on
 58nv set interface bond7 bond member swp7
 59nv set interface bond7 bond mlag id 7
 60nv set interface bond7 bridge domain br_default untagged 122
 61nv set interface bond7 bridge domain br_default vlan all
 62nv set interface bond7 bond mlag enable on
 63nv set interface bond7 bond lacp-bypass on
 64nv set interface bond8 bond member swp8
 65nv set interface bond8 bond mlag id 8
 66nv set interface bond8 bridge domain br_default untagged 122
 67nv set interface bond8 bridge domain br_default vlan all
 68nv set interface bond8 bond mlag enable on
 69nv set interface bond8 bond lacp-bypass on
 70nv set interface bond9 bond member swp9
 71nv set interface bond9 bond mlag id 9
 72nv set interface bond9 bridge domain br_default untagged 122
 73nv set interface bond9 bridge domain br_default vlan all
 74nv set interface bond9 bond mlag enable on
 75nv set interface bond9 bond lacp-bypass on
 76nv set interface bond10 bond member swp10
 77nv set interface bond10 bond mlag id 10
 78nv set interface bond10 bridge domain br_default untagged 122
 79nv set interface bond10 bridge domain br_default vlan all
 80nv set interface bond10 bond mlag enable on
 81nv set interface bond10 bond lacp-bypass on
 82nv set interface bond11 bond member swp11
 83nv set interface bond11 bond mlag id 11
 84nv set interface bond11 bridge domain br_default untagged 122
 85nv set interface bond11 bridge domain br_default vlan all
 86nv set interface bond11 bond mlag enable on
 87nv set interface bond11 bond lacp-bypass on
 88nv set interface bond12 bond member swp12
 89nv set interface bond12 bond mlag id 12
 90nv set interface bond12 bridge domain br_default untagged 122
 91nv set interface bond12 bridge domain br_default vlan all
 92nv set interface bond12 bond mlag enable on
 93nv set interface bond12 bond lacp-bypass on
 94nv set interface bond13 bond member swp13
 95nv set interface bond13 bond mlag id 13
 96nv set interface bond13 bridge domain br_default untagged 122
 97nv set interface bond13 bridge domain br_default vlan all
 98nv set interface bond13 bond mlag enable on
 99nv set interface bond13 bond lacp-bypass on
100nv set interface bond14 bond member swp14
101nv set interface bond14 bond mlag id 14
102nv set interface bond14 bridge domain br_default untagged 122
103nv set interface bond14 bridge domain br_default vlan all
104nv set interface bond14 bond mlag enable on
105nv set interface bond14 bond lacp-bypass on
106nv set interface bond15 bond member swp15
107nv set interface bond15 bond mlag id 15
108nv set interface bond15 bridge domain br_default untagged 122
109nv set interface bond15 bridge domain br_default vlan all
110nv set interface bond15 bond mlag enable on
111nv set interface bond15 bond lacp-bypass on
112nv set interface bond16 bond member swp16
113nv set interface bond16 bond mlag id 16
114nv set interface bond16 bridge domain br_default untagged 122
115nv set interface bond16 bridge domain br_default vlan all
116nv set interface bond16 bond mlag enable on
117nv set interface bond16 bond lacp-bypass on
118nv set interface bond17 bond member swp17
119nv set interface bond17 bond mlag id 17
120nv set interface bond17 bridge domain br_default untagged 122
121nv set interface bond17 bridge domain br_default vlan all
122nv set interface bond17 bond mlag enable on
123nv set interface bond17 bond lacp-bypass on
124nv set interface bond18 bond member swp18
125nv set interface bond18 bond mlag id 18
126nv set interface bond18 bridge domain br_default untagged 122
127nv set interface bond18 bridge domain br_default vlan all
128nv set interface bond18 bond mlag enable on
129nv set interface bond18 bond lacp-bypass on
130nv set interface bond19 bond member swp19
131nv set interface bond19 bond mlag id 19
132nv set interface bond19 bridge domain br_default untagged 122
133nv set interface bond19 bridge domain br_default vlan all
134nv set interface bond19 bond mlag enable on
135nv set interface bond19 bond lacp-bypass on
136nv set interface bond20 bond member swp20
137nv set interface bond20 bond mlag id 20
138nv set interface bond20 bridge domain br_default untagged 122
139nv set interface bond20 bridge domain br_default vlan all
140nv set interface bond20 bond mlag enable on
141nv set interface bond20 bond lacp-bypass on
142
143### BGP unnumbered configuration (NOTE: no IPs need to be configured on the BGP interfaces, when using BGP unnumbered)
144nv set router bgp autonomous-system 4200000003
145nv set router bgp enable on
146nv set router bgp router-id 10.227.20.78
147nv set vrf default router bgp address-family ipv4-unicast enable on
148nv set vrf default router bgp address-family ipv4-unicast redistribute connected enable on
149nv set vrf default router bgp enable on
150nv set vrf default router bgp neighbor swp64 remote-as external
151nv set vrf default router bgp neighbor swp64 type unnumbered
152nv set vrf default router bgp neighbor swp63 remote-as external
153nv set vrf default router bgp neighbor swp63 type unnumbered
154nv set vrf default router bgp neighbor swp60 remote-as external
155nv set vrf default router bgp neighbor swp60 type unnumbered
156nv set vrf default router bgp neighbor peerlink.4094 remote-as internal
157nv set vrf default router bgp neighbor peerlink.4094 type unnumbered
158
159### apply and save configuration
160nv config diff
161nv config apply
162nv config save

SN4600 #2 (In-band Management Switch)#

  1# Note Make sure to update the IP addresses in the sample config below
  2# eth0 mgmt interface configs
  3nv set interface eth0 ip address 10.227.20.79/26
  4nv set interface eth0 ip gateway 10.227.20.65
  5nv set interface eth0 ip vrf mgmt
  6nv set interface eth0 type eth
  7
  8# Creating SVI interfaces and adding the VLANs to the bridge
  9# Note Make sure to update the IP addresses
 10nv set bridge domain br_default vlan 122
 11nv set bridge domain br_default vlan 121
 12nv set interface vlan121 type svi
 13nv set interface vlan121 ip vrr mac-address 44:38:39:ff:ff:ff
 14nv set interface vlan121 ip vrr address 10.227.52.1/26
 15nv set interface vlan121 ip address 10.227.52.3/26
 16nv set interface vlan122 type svi
 17nv set interface vlan122 ip vrr mac-address 44:38:39:ff:ff:ff
 18nv set interface vlan122 ip vrr address 10.227.48.1/26
 19nv set interface vlan122 ip address 10.227.48.3/26
 20
 21# MLAG configs
 22nv set interface peerlink bond member swp57
 23nv set interface peerlink bond member swp58
 24nv set interface peerlink type peerlink
 25nv set interface peerlink.4094 base-interface peerlink
 26nv set interface peerlink.4094 type sub
 27nv set interface peerlink.4094 vlan 4094
 28nv set mlag backup 10.227.20.78 vrf mgmt
 29nv set mlag enable on
 30nv set mlag mac-address 44:38:39:ff:ff:ff
 31nv set mlag peer-ip linklocal
 32
 33### bcm headnode 01
 34nv set interface swp1 bridge domain br_default access 122
 35### bcm headnode 02
 36nv set interface swp2 bridge domain br_default access 122
 37
 38### bond4-20 used to connect to DGX and Kubernetes master nodes
 39nv set interface bond4 bond member swp4
 40nv set interface bond4 bond mlag id 4
 41nv set interface bond4 bridge domain br_default untagged 122
 42nv set interface bond4 bridge domain br_default vlan all
 43nv set interface bond4 bond mlag enable on
 44nv set interface bond4 bond lacp-bypass on
 45nv set interface bond5 bond member swp5
 46nv set interface bond5 bond mlag id 5
 47nv set interface bond5 bridge domain br_default untagged 122
 48nv set interface bond5 bridge domain br_default vlan all
 49nv set interface bond5 bond mlag enable on
 50nv set interface bond5 bond lacp-bypass on
 51nv set interface bond6 bond member swp6
 52nv set interface bond6 bond mlag id 6
 53nv set interface bond6 bridge domain br_default untagged 122
 54nv set interface bond6 bridge domain br_default vlan all
 55nv set interface bond6 bond mlag enable on
 56nv set interface bond6 bond lacp-bypass on
 57nv set interface bond7 bond member swp7
 58nv set interface bond7 bond mlag id 7
 59nv set interface bond7 bridge domain br_default untagged 122
 60nv set interface bond7 bridge domain br_default vlan all
 61nv set interface bond7 bond mlag enable on
 62nv set interface bond7 bond lacp-bypass on
 63nv set interface bond8 bond member swp8
 64nv set interface bond8 bond mlag id 8
 65nv set interface bond8 bridge domain br_default untagged 122
 66nv set interface bond8 bridge domain br_default vlan all
 67nv set interface bond8 bond mlag enable on
 68nv set interface bond8 bond lacp-bypass on
 69nv set interface bond9 bond member swp9
 70nv set interface bond9 bond mlag id 9
 71nv set interface bond9 bridge domain br_default untagged 122
 72nv set interface bond9 bridge domain br_default vlan all
 73nv set interface bond9 bond mlag enable on
 74nv set interface bond9 bond lacp-bypass on
 75nv set interface bond10 bond member swp10
 76nv set interface bond10 bond mlag id 10
 77nv set interface bond10 bridge domain br_default untagged 122
 78nv set interface bond10 bridge domain br_default vlan all
 79nv set interface bond10 bond mlag enable on
 80nv set interface bond10 bond lacp-bypass on
 81nv set interface bond11 bond member swp11
 82nv set interface bond11 bond mlag id 11
 83nv set interface bond11 bridge domain br_default untagged 122
 84nv set interface bond11 bridge domain br_default vlan all
 85nv set interface bond11 bond mlag enable on
 86nv set interface bond11 bond lacp-bypass on
 87nv set interface bond12 bond member swp12
 88nv set interface bond12 bond mlag id 12
 89nv set interface bond12 bridge domain br_default untagged 122
 90nv set interface bond12 bridge domain br_default vlan all
 91nv set interface bond12 bond mlag enable on
 92nv set interface bond12 bond lacp-bypass on
 93nv set interface bond13 bond member swp13
 94nv set interface bond13 bond mlag id 13
 95nv set interface bond13 bridge domain br_default untagged 122
 96nv set interface bond13 bridge domain br_default vlan all
 97nv set interface bond13 bond mlag enable on
 98nv set interface bond13 bond lacp-bypass on
 99nv set interface bond14 bond member swp14
100nv set interface bond14 bond mlag id 14
101nv set interface bond14 bridge domain br_default untagged 122
102nv set interface bond14 bridge domain br_default vlan all
103nv set interface bond14 bond mlag enable on
104nv set interface bond14 bond lacp-bypass on
105nv set interface bond15 bond member swp15
106nv set interface bond15 bond mlag id 15
107nv set interface bond15 bridge domain br_default untagged 122
108nv set interface bond15 bridge domain br_default vlan all
109nv set interface bond15 bond mlag enable on
110nv set interface bond15 bond lacp-bypass on
111nv set interface bond16 bond member swp16
112nv set interface bond16 bond mlag id 16
113nv set interface bond16 bridge domain br_default untagged 122
114nv set interface bond16 bridge domain br_default vlan all
115nv set interface bond16 bond mlag enable on
116nv set interface bond16 bond lacp-bypass on
117nv set interface bond17 bond member swp17
118nv set interface bond17 bond mlag id 17
119nv set interface bond17 bridge domain br_default untagged 122
120nv set interface bond17 bridge domain br_default vlan all
121nv set interface bond17 bond mlag enable on
122nv set interface bond17 bond lacp-bypass on
123nv set interface bond18 bond member swp18
124nv set interface bond18 bond mlag id 18
125nv set interface bond18 bridge domain br_default untagged 122
126nv set interface bond18 bridge domain br_default vlan all
127nv set interface bond18 bond mlag enable on
128nv set interface bond18 bond lacp-bypass on
129nv set interface bond19 bond member swp19
130nv set interface bond19 bond mlag id 19
131nv set interface bond19 bridge domain br_default untagged 122
132nv set interface bond19 bridge domain br_default vlan all
133nv set interface bond19 bond mlag enable on
134nv set interface bond19 bond lacp-bypass on
135nv set interface bond20 bond member swp20
136nv set interface bond20 bond mlag id 20
137nv set interface bond20 bridge domain br_default untagged 122
138nv set interface bond20 bridge domain br_default vlan all
139nv set interface bond20 bond mlag enable on
140nv set interface bond20 bond lacp-bypass on
141
142### BGP unnumbered configuration (NOTE: no IPs need to be configured on the BGP interfaces, when using BGP unnumbered)
143nv set router bgp autonomous-system 4200000003
144nv set router bgp enable on
145nv set router bgp router-id 10.227.20.79
146nv set vrf default router bgp address-family ipv4-unicast enable on
147nv set vrf default router bgp address-family ipv4-unicast redistribute connected enable on
148nv set vrf default router bgp enable on
149nv set vrf default router bgp neighbor swp63 remote-as external
150nv set vrf default router bgp neighbor swp63 type unnumbered
151nv set vrf default router bgp neighbor swp64 remote-as external
152nv set vrf default router bgp neighbor swp64 type unnumbered
153nv set vrf default router bgp neighbor swp60 remote-as external
154nv set vrf default router bgp neighbor swp60 type unnumbered
155nv set vrf default router bgp neighbor peerlink.4094 remote-as internal
156nv set vrf default router bgp neighbor peerlink.4094 type unnumbered
157
158### apply and save configuration
159nv config diff
160nv config apply
161nv config save

SN2201 (Out-of-band Management Switch)#

 1#eth0
 2nv set interface eth0 ip address 10.227.20.77/26
 3nv set interface eth0 ip gateway 10.227.20.65
 4nv set interface eth0 ip vrf mgmt
 5nv set interface eth0 type eth
 6
 7# Creating SVI interfaces and adding the VLANs to the bridge
 8# Note Make sure to update the IP addresses
 9nv set interface vlan111 ip address 10.227.20.65/26
10nv set bridge domain br_default vlan 111
11
12### BGP configurations
13nv set router bgp autonomous-system 4200000004
14nv set router bgp enable on
15nv set router bgp router-id 10.227.20.77
16nv set vrf default router bgp address-family ipv4-unicast enable on
17nv set vrf default router bgp address-family ipv4-unicast redistribute connected enable on
18nv set vrf default router bgp enable on
19nv set vrf default router bgp neighbor swp51 remote-as external
20nv set vrf default router bgp neighbor swp51 type unnumbered
21nv set vrf default router bgp neighbor swp52 remote-as external
22nv set vrf default router bgp neighbor swp52 type unnumbered
23
24# interfaces connected to IPMI interfaces of different servers
25nv set interface swp1-40 bridge domain br_default access 111
26
27### apply and save configuration
28nv config diff
29nv config apply
30nv config save

Ethernet Network Configuration Verifications#

Some of the lines in the output have been truncated for readability.

  1### MLAG verifications:
  2root@TOR-01:mgmt:~# net show clag
  3The peer is alive
  4   Our Priority, ID, and Role: 2048 48:b0:2d:cc:b2:bc primary
  5   Peer Priority, ID, and Role: 32768 48:b0:2d:ca:93:7b secondary
  6         Peer Interface and IP: peerlink.4094 fe80::4ab0:2dff:feca:937b (linklocal)
  7                     Backup IP: 10.227.20.79 vrf mgmt (active)
  8                     System MAC: 44:38:39:ff:ff:ff
  9root@TOR-01:mgmt:~#
 10CLAG Interfaces
 11Our Interface   Peer Interface  CLAG Id   Conflicts        Proto-Down Reason
 12-------------   -------------   -------   --------------   -----------------
 13      bond10   -                 10        -                      -
 14      bond11   -                 11        -                      -
 15      bond12   -                 12        -                      -
 16      bond13   bond13            13        -                      -
 17      bond14   -                 14        -                      -
 18      bond15   -                 15        -                      -
 19      bond16   bond16            16        -                      -
 20      bond17   -                 17        -                      -
 21      bond18   -                 18        -                      -
 22      bond19   -                 19        -                      -
 23      bond20   -                 20
 24      bond4   bond4              4         -                      -
 25      bond5   -                  5         -                      -
 26      bond6   bond6              6         -                      -
 27      bond7   -                  7         -                      -
 28      bond8   -                  8         -                      -
 29      bond9   -                  9         -                      -
 30
 31### verifying an access port (below is a access port with VLAN set to 111)
 32
 33network-admin@IPMI-01:mgmt:~$ net show int swp1
 34   Name  MAC                Speed  MTU   Mode
 35--  ----  -----------------  -----  ----  ---------
 36UP  swp1  68:21:5f:4f:14:81  1G     1500  Access/L2
 37
 38Alias
 39-----
 40bcm-bootstrap:eth0
 41
 42All VLANs on L2 Port
 43--------------------
 44111
 45
 46Untagged
 47--------
 48111
 49
 50### verifying a bonded interface in trunk mode (note the native VLAN is set to 122)
 51network-admin@TOR-01:mgmt:~$ net show int bond9
 52   Name    MAC                Speed  MTU   Mode
 53--  ------  -----------------  -----  ----  -------
 54UP  bond9  1c:34:da:29:17:54  100G   9216  802.3ad
 55
 56Bond Details
 57------------------  --------
 58Bond Mode:          802.3ad
 59Load Balancing:     layer3+4
 60Minimum Links:      1
 61LACP Sys Priority:
 62LACP Rate:          1
 63LACP Bypass:        Active
 64
 65All VLANs on L2 Port
 66--------------------
 671,121,122
 68
 69Untagged
 70--------
 71122
 72
 73#### BGP verifications
 74
 75cumulus@TOR-01:mgmt:~$ net show bgp summary
 76show bgp ipv4 unicast summary
 77=============================
 78BGP router identifier 10.227.20.78, local AS number 4200000003 vrf-id 0
 79BGP table version 3
 80RIB entries 5, using 1000 bytes of memory
 81Peers 4, using 91 KiB of memory
 82
 83Neighbor              V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
 84TOR-02(peerlink.4094)  4 4200000003      2705      2706        0    0    0 02:15:00
 85Spine-02(swp64)        4 4200000002      2655      2656        0    0    0 02:12:29
 86Spine-01(swp63)        4 4200000001      2753      2754        0    0    0 02:17:22
 87IPMI-01(swp60)         4 4200000004      2480      2482        0    0    0 02:03:48
 88
 89Total number of neighbors 4
 90
 91
 92cumulus@TOR-02:mgmt:~$ net show bgp summary
 93show bgp ipv4 unicast summary
 94=============================
 95BGP router identifier 10.227.20.79, local AS number 4200000003 vrf-id 0
 96BGP table version 3
 97RIB entries 5, using 1000 bytes of memory
 98Peers 4, using 91 KiB of memory
 99
100Neighbor              V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
101TOR-01(peerlink.4094) 4 4200000003      2692      2692        0    0    0 02:14:20
102Spine-01(swp63)        4 4200000001      2692      2692        0    0    0 02:14:21
103Spine-02(swp64)        4 4200000002      2641      2642        0    0    0 02:11:48
104IPMI-01(swp60)         4 4200000004      2467      2469        0    0    0 02:03:07
105
106Total number of neighbors 4
107
108cumulus@IPMI-01:mgmt:~$ net show bgp summary
109show bgp ipv4 unicast summary
110=============================
111BGP router identifier 10.227.20.77, local AS number 4200000004 vrf-id 0
112BGP table version 3
113RIB entries 5, using 1000 bytes of memory
114Peers 2, using 46 KiB of memory
115
116Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
117TOR-01(swp51)    4 4200000003      2495      2494        0    0    0 02:04:30
118TOR-02(swp52)   4 4200000003      2495      2494        0    0    0 02:04:30
119
120Total number of neighbors 2
On this page