DGX B300 Network Configuration#
Ethernet Management (North-South) Network Configuration#
You can configure the Ethernet Management, or North-South, network manually or automatically, depending on the deployment scale. The following sections describe both approaches.
North-South Ethernet Network - Automated Deployment#
BCM can automate network switch provisioning and fabric configuration for large-scale deployments using bcm-netautogen. Contact your NVIDIA PTAM for NVIS engagement and assistance with large scale deployments.
Manual Network Configuration#
Manual configuration is recommended for smaller deployments such as BasePOD. See the following sections for details.
Network Architecture#
The B300 BasePOD management network uses two SN5600 leaf switches as an MLAG pair for redundant in-band connectivity. Each head node, Kubernetes node, and DGX B300 node is dual-homed to both leaves through host-side bonded interfaces; the leaves provide SVIs and VRR gateways for the Control-Nodes and DGXnet1 VLANs. The dedicated SN2201 OOB switch terminates the IPMI/BMC/OOB VLAN and provides the gateway for the BMC network.
Customer connectivity is provided through BGP from both SN5600 leaves toward the customer gateway, with an internal BGP session across peerlink.4094. The example topology shows one customer gateway; however, two customer gateways linking to both leaf switches are recommended for redundancy.
The out-of-band management plane is terminated on a dedicated SN2201 OOB management switch so BMC/IPMI access remains separate from in-band traffic. The OOB switch provides the VLAN 101 SVI gateway for 10.150.123.0/24 and uplinks separately to the customer OOB network with BGP. BCM head nodes and Kubernetes admin nodes use dedicated interfaces to the OOB/IPMI network. These interfaces support BCM operations such as firmware upgrades and power control on DGX nodes, and allow NMC components running in the Kubernetes administration cluster to access node BMCs. An optional OOB HA switch pair is recommended for deployments that require higher availability.
Figure 1 Physical connectivity topology for the B300 BasePOD network.#
Figure 2 Out-of-band management topology for BMC/IPMI access.#
Note
UFM is not shown in the topology diagrams, but it is connected to both SN5600 switches in the same manner as the other management interfaces for monitoring and management of the XDR fabric.
SN5600 – North-South Management Ethernet Switches#
The SN5600 fabric provides connectivity for in-band management and provisioning of the nodes. The key configuration requirements are:
MLAG between the two SN5600 switches
L3 SVI/VRR for the in-band Control-Nodes and DGXnet1 Ethernet networks.
IPMI/BMC/OOB network termination on the dedicated SN2201 OOB switch.
Each head node, Kubernetes node, and DGX node is dual-homed to the SN5600 switches through a bonded interface.
UFM Ethernet management interfaces are connected in a similar manner to both SN5600 switches for monitoring and management of the XDR fabric.
External connectivity to the customer network, using customer-specified routing arrangements such as BGP in this example.
SN5600-1 Reference Configuration#
# Basic management configuration
nv set system hostname SN5600-LEAF-1
nv set interface eth0 ipv4 dhcp-client set-hostname enabled
nv set interface eth0 ipv4 dhcp-client state enabled
nv set interface eth0 type eth
nv set interface eth0 vrf mgmt
#
# Create SVIs for Internal/Control and DGX networks with VRR as FHRP
nv set bridge domain br_default vlan 100,102-103
nv set interface vlan102 description Internalnet/Control-Nodes
nv set interface vlan102 ipv4 address 10.150.124.2/24
nv set interface vlan102 ipv4 vrr address 10.150.124.1/24
nv set interface vlan102 vlan 102
nv set interface vlan103 description DGXnet1
nv set interface vlan103 ipv4 address 10.150.125.2/24
nv set interface vlan103 ipv4 vrr address 10.150.125.1/24
nv set interface vlan103 vlan 103
nv set interface vlan102-103 ipv4 vrr mac-address 00:1c:73:aa:bb:04
nv set interface vlan102-103 ipv4 vrr state enabled
nv set interface vlan102-103 ipv4 vrr vrr-state up
nv set interface vlan102-103 type svi
nv set router vrr state enabled
# Configure MLAG
# Define inter-chassis peerlink etherchannel/bond
nv set interface peerlink bond member swp63-64
nv set interface peerlink type peerlink
nv set interface peerlink,swp63-64 description MLAG-peerlink-to-SN5600-LEAF-2
#
# Loopback for BGP/MLAG backup routing
nv set interface lo ipv4 address 10.160.254.22/32
nv set interface lo type loopback
#
# Configure Peerlink L3 parameters
nv set interface peerlink.4094 base-interface peerlink
nv set interface peerlink.4094 type sub
nv set interface peerlink.4094 vlan 4094
nv set mlag backup 10.160.254.23
nv set mlag mac-address 44:38:39:FF:0A:00
nv set mlag peer-ip linklocal
# MLAG primary
nv set mlag priority 2048
nv set mlag state enabled
# Example port configuration for head nodes (BCM, Kube)
# BCM Head Nodes
nv set interface bond5 bond member swp5
nv set interface bond5 description BCM-headnode-1
nv set interface bond5 bond mlag id 5
nv set interface bond6 bond member swp6
nv set interface bond6 description BCM-headnode-2
nv set interface bond6 bond mlag id 6
# Head, Kubernetes, and login nodes use bonds 5-14 on VLAN 102
nv set interface bond5-14 bridge domain br_default access 102
# DGX B300 nodes use bonds 15-24 on VLAN 103
nv set interface bond15-24 bridge domain br_default access 103
nv set interface bond5-24 bond mlag state enabled
nv set interface bond5-24 bond lacp-bypass enabled
nv set interface bond5-24 type bond
# Repeat for other management/workloads/compute nodes
#
# Uplink to the customer network.
# Example configuration with BGP unnumbered
nv set router bgp autonomous-system 4200120327
nv set router bgp state enabled
nv set router bgp router-id 10.160.254.22
nv set vrf default router bgp address-family ipv4-unicast state enabled
nv set vrf default router bgp address-family ipv4-unicast redistribute connected state enabled
nv set vrf default router bgp state enabled
# Uplinks via swp61 and swp62
nv set interface swp61 description BGP-uplink-secondary-to-SN5600-Customer-Gw
nv set interface swp62 description BGP-uplink-to-SN5600-Customer-Gw
nv set interface swp61-64 type swp
nv set vrf default router bgp neighbor swp61 remote-as external
nv set vrf default router bgp neighbor swp61 type unnumbered
nv set vrf default router bgp neighbor swp62 remote-as external
nv set vrf default router bgp neighbor swp62 type unnumbered
# Peering to MLAG peer switch
nv set vrf default router bgp neighbor peerlink.4094 remote-as internal
nv set vrf default router bgp neighbor peerlink.4094 type unnumbered
SN5600-2 reference configuration - Same as SN5600-1, with the following changes#
# Basic management configuration
nv set system hostname SN5600-LEAF-2
#
# Create SVIs for Internal/Control and DGX networks with VRR as FHRP
nv set bridge domain br_default vlan 100,102-103
nv set interface vlan102 description Internalnet/Control-Nodes
nv set interface vlan102 ipv4 address 10.150.124.3/24
nv set interface vlan102 ipv4 vrr address 10.150.124.1/24
nv set interface vlan102 vlan 102
nv set interface vlan103 description DGXnet1
nv set interface vlan103 ipv4 address 10.150.125.3/24
nv set interface vlan103 ipv4 vrr address 10.150.125.1/24
nv set interface vlan103 vlan 103
nv set interface vlan102-103 ipv4 vrr mac-address 00:1c:73:aa:bb:04
nv set interface vlan102-103 ipv4 vrr state enabled
nv set interface vlan102-103 ipv4 vrr vrr-state up
nv set interface vlan102-103 type svi
nv set router vrr state enabled
# Configure MLAG
# Define inter-chassis peerlink etherchannel/bond
nv set interface peerlink bond member swp63-64
nv set interface peerlink type peerlink
nv set interface peerlink,swp63-64 description MLAG-peerlink-to-SN5600-LEAF-1
#
# Loopback for BGP/MLAG backup routing
nv set interface lo ipv4 address 10.160.254.23/32
nv set interface lo type loopback
#
# Configure Peerlink L3 parameters
nv set interface peerlink.4094 base-interface peerlink
nv set interface peerlink.4094 type sub
nv set interface peerlink.4094 vlan 4094
nv set mlag backup 10.160.254.22
nv set mlag mac-address 44:38:39:FF:0A:00
nv set mlag peer-ip linklocal
# MLAG secondary
nv set mlag priority 4096
nv set mlag state enabled
# Example port configuration for head nodes (BCM, Kube)
# Same bond ranges and VLAN assignments as SN5600-LEAF-1
nv set interface bond5-14 bridge domain br_default access 102
nv set interface bond15-24 bridge domain br_default access 103
nv set interface bond5-24 bond mlag state enabled
nv set interface bond5-24 bond lacp-bypass enabled
nv set interface bond5-24 type bond
#
# Uplink to the customer network.
# Example configuration with BGP unnumbered
nv set router bgp autonomous-system 4200120327
nv set router bgp state enabled
nv set router bgp router-id 10.160.254.23
nv set vrf default router bgp address-family ipv4-unicast state enabled
nv set vrf default router bgp address-family ipv4-unicast redistribute connected state enabled
nv set vrf default router bgp state enabled
# Uplinks via swp61 and swp62
nv set interface swp61 description BGP-uplink-secondary-to-SN5600-Customer-Gw
nv set interface swp62 description BGP-uplink-to-SN5600-Customer-Gw
nv set interface swp61-64 type swp
nv set vrf default router bgp neighbor swp61 remote-as external
nv set vrf default router bgp neighbor swp61 type unnumbered
nv set vrf default router bgp neighbor swp62 remote-as external
nv set vrf default router bgp neighbor swp62 type unnumbered
# Peering to MLAG peer switch
nv set vrf default router bgp neighbor peerlink.4094 remote-as internal
nv set vrf default router bgp neighbor peerlink.4094 type unnumbered
Verify MLAG status on both SN5600 switches.
nv show mlag
clagctl
Use nv show mlag to verify that MLAG is enabled, the peer is alive, the local and peer roles are correct, and the peer interface is peerlink.4094. Use clagctl to verify CLAG interface state and confirm that the MLAG bonds do not report conflicts or proto-down reasons.
For consistency checks, run the following commands from each SN5600 switch.
nv show mlag consistency-checker global
nv show interface --view=mlag-cc
The global consistency checker verifies peer-wide MLAG parameters such as the system MAC, peer IP, peerlink VLANs, bridge type, STP settings, and peerlink MTU. The interface MLAG consistency view verifies per-interface settings such as CLAG ID, bridge membership, MTU, native VLAN, VLAN membership, and LACP actor or partner MAC values.
If a specific MLAG bond reports a conflict, inspect that bond directly. For example:
nv show interface bond5 bond mlag consistency-checker
Sample MLAG verification output
The output should show the peerlink interface in an operational state and no MLAG consistency conflicts on the configured bonds.
===== SN5600-LEAF-1 =====
SN5600-LEAF-1
--- nv show mlag ---
operational applied
-------------- ------------------------ -----------------
state enabled enabled
mac-address 44:38:39:ff:0a:00 44:38:39:FF:0A:00
peer-ip fe80::4ab0:2dff:fe00:4e6 linklocal
priority 2048 2048
init-delay 180 180
debug disabled disabled
[backup] 10.160.254.23 10.160.254.23
peer-priority 4096
backup-active True
local-id 48:b0:2d:00:02:61
peer-id 48:b0:2d:00:04:e6
local-role primary
peer-role secondary
peer-interface peerlink.4094
peer-alive True
backup-reason
--- clagctl ---
The peer is alive
Our Priority, ID, and Role: 2048 48:b0:2d:00:02:61 primary
Peer Priority, ID, and Role: 4096 48:b0:2d:00:04:e6 secondary
Peer Interface and IP: peerlink.4094 fe80::4ab0:2dff:fe00:4e6 (linklocal)
Backup IP: 10.160.254.23 (active)
System MAC: 44:38:39:ff:0a:00
CLAG Interfaces
Our Interface Peer Interface CLAG Id Conflicts Proto-Down Reason
---------------- ---------------- ------- -------------------- -----------------
bond10 - 10 - -
bond11 - 11 - -
bond12 - 12 - -
bond13 - 13 - -
bond14 - 14 - -
bond15 - 15 - -
bond16 - 16 - -
bond17 - 17 - -
bond18 - 18 - -
bond19 - 19 - -
bond20 - 20 - -
bond21 - 21 - -
bond22 - 22 - -
bond5 - 5 - -
bond6 - 6 - -
bond7 - 7 - -
bond8 - 8 - -
bond9 - 9 - -
--- nv show mlag consistency-checker global ---
Parameter LocalValue PeerValue Conflict Summary
---------------------- ------------------------ ------------------------ -------- -------
anycast-ip - - -
bridge-priority 32768 32768 -
bridge-stp-mode rstp rstp -
bridge-stp-state on on -
bridge-type vlan-aware vlan-aware -
clag-pkg-version 1.6.0-cl5.16.0u6 1.6.0-cl5.16.0u6 -
clag-protocol-version 1.8.0 1.8.0 -
peer-ip fe80::4ab0:2dff:fe00:4e6 fe80::4ab0:2dff:fe00:4e6 -
peerlink-bridge-member Yes Yes -
peerlink-mtu 9216 9216 -
peerlink-native-vlan 1 1 -
peerlink-vlans 1, 100->103 1, 100->103 -
redirect2-enable yes yes -
system-mac 44:38:39:ff:0a:00 44:38:39:ff:0a:00 -
--- nv show interface --view=mlag-cc ---
Interface Parameter LocalValue PeerValue Conflict
--------- ---------------- ----------------- ----------------- --------
bond5 clag-id 5 5 -
lacp-partner-mac 00:00:00:00:00:00 00:00:00:00:00:00 -
lacp-actor-mac 44:38:39:ff:0a:00 44:38:39:ff:0a:00 -
vlan-id 102 102 -
native-vlan 102 102 -
master br_default NOT-SYNCED -
mtu 9216 9216 -
bridge-learning yes yes -
bpdu-guard off off -
restricted-role no no -
bond6 clag-id 6 6 -
lacp-partner-mac 00:00:00:00:00:00 00:00:00:00:00:00 -
lacp-actor-mac 44:38:39:ff:0a:00 44:38:39:ff:0a:00 -
vlan-id 102 102 -
native-vlan 102 102 -
master br_default NOT-SYNCED -
mtu 9216 9216 -
bridge-learning yes yes -
bpdu-guard off off -
restricted-role no no -
--- nv show interface bond5 bond mlag consistency-checker ---
Parameter LocalValue PeerValue Conflict Summary
---------------- ----------------- ----------------- -------- -------
bpdu-guard off off -
bridge-learning yes yes -
clag-id 5 5 -
lacp-actor-mac 44:38:39:ff:0a:00 44:38:39:ff:0a:00 -
lacp-partner-mac 00:00:00:00:00:00 00:00:00:00:00:00 -
master br_default NOT-SYNCED -
mtu 9216 9216 -
native-vlan 102 102 -
restricted-role no no -
vlan-id 102 102 -
===== SN5600-LEAF-2 =====
SN5600-LEAF-2
--- nv show mlag ---
operational applied
-------------- ------------------------ -----------------
state enabled enabled
mac-address 44:38:39:ff:0a:00 44:38:39:FF:0A:00
peer-ip fe80::4ab0:2dff:fe00:261 linklocal
priority 4096 4096
init-delay 180 180
debug disabled disabled
[backup] 10.160.254.22 10.160.254.22
peer-priority 2048
backup-active True
local-id 48:b0:2d:00:04:e6
peer-id 48:b0:2d:00:02:61
local-role secondary
peer-role primary
peer-interface peerlink.4094
peer-alive True
backup-reason
--- clagctl ---
The peer is alive
Our Priority, ID, and Role: 4096 48:b0:2d:00:04:e6 secondary
Peer Priority, ID, and Role: 2048 48:b0:2d:00:02:61 primary
Peer Interface and IP: peerlink.4094 fe80::4ab0:2dff:fe00:261 (linklocal)
Backup IP: 10.160.254.22 (active)
System MAC: 44:38:39:ff:0a:00
CLAG Interfaces
Our Interface Peer Interface CLAG Id Conflicts Proto-Down Reason
---------------- ---------------- ------- -------------------- -----------------
bond10 - 10 - -
bond11 - 11 - -
bond12 - 12 - -
bond13 - 13 - -
bond14 - 14 - -
bond15 - 15 - -
bond16 - 16 - -
bond17 - 17 - -
bond18 - 18 - -
bond19 - 19 - -
bond20 - 20 - -
bond21 - 21 - -
bond22 - 22 - -
bond5 - 5 - -
bond6 - 6 - -
bond7 - 7 - -
bond8 - 8 - -
bond9 - 9 - -
--- nv show mlag consistency-checker global ---
Parameter LocalValue PeerValue Conflict Summary
---------------------- ------------------------ ------------------------ -------- -------
anycast-ip - - -
bridge-priority 32768 32768 -
bridge-stp-mode rstp rstp -
bridge-stp-state on on -
bridge-type vlan-aware vlan-aware -
clag-pkg-version 1.6.0-cl5.16.0u6 1.6.0-cl5.16.0u6 -
clag-protocol-version 1.8.0 1.8.0 -
peer-ip fe80::4ab0:2dff:fe00:261 fe80::4ab0:2dff:fe00:261 -
peerlink-bridge-member Yes Yes -
peerlink-mtu 9216 9216 -
peerlink-native-vlan 1 1 -
peerlink-vlans 1, 100->103 1, 100->103 -
redirect2-enable yes yes -
system-mac 44:38:39:ff:0a:00 44:38:39:ff:0a:00 -
--- nv show interface --view=mlag-cc ---
Interface Parameter LocalValue PeerValue Conflict
--------- ---------------- ----------------- ----------------- --------
bond5 clag-id 5 5 -
lacp-partner-mac 00:00:00:00:00:00 00:00:00:00:00:00 -
lacp-actor-mac 44:38:39:ff:0a:00 44:38:39:ff:0a:00 -
vlan-id 102 102 -
native-vlan 102 102 -
master br_default NOT-SYNCED -
mtu 9216 9216 -
bridge-learning yes yes -
bpdu-guard off off -
restricted-role no no -
bond6 clag-id 6 6 -
lacp-partner-mac 00:00:00:00:00:00 00:00:00:00:00:00 -
lacp-actor-mac 44:38:39:ff:0a:00 44:38:39:ff:0a:00 -
vlan-id 102 102 -
native-vlan 102 102 -
master br_default NOT-SYNCED -
mtu 9216 9216 -
bridge-learning yes yes -
bpdu-guard off off -
restricted-role no no -
--- nv show interface bond5 bond mlag consistency-checker ---
Parameter LocalValue PeerValue Conflict Summary
---------------- ----------------- ----------------- -------- -------
bpdu-guard off off -
bridge-learning yes yes -
clag-id 5 5 -
lacp-actor-mac 44:38:39:ff:0a:00 44:38:39:ff:0a:00 -
lacp-partner-mac 00:00:00:00:00:00 00:00:00:00:00:00 -
master br_default NOT-SYNCED -
mtu 9216 9216 -
native-vlan 102 102 -
restricted-role no no -
vlan-id 102 102 -
BGP/Uplink peering#
Verify BGP peering for the customer uplinks and the internal peerlink peering from each SN5600 switch.
nv show vrf default router bgp neighbor
nv show vrf default router bgp neighbor swp62
sudo vtysh -c "show ip bgp summary"
To inspect the BGP table, run the following command from one SN5600 leaf.
sudo vtysh -c "show ip bgp"
The BGP neighbor table shows the state of each configured peer. In this example, peerlink.4094 and swp62 are established. The swp61 neighbor is configured as a secondary uplink and can remain idle until the corresponding upstream link is connected.
Sample BGP/Uplink peering output
The output should show the expected BGP neighbors in the Established state and the expected prefixes in the BGP table.
===== SN5600-LEAF-1 =====
--- nv show vrf default router bgp neighbor ---
Hostname - Hostname, AS - Remote Autonomous System, Uptime - BGP session up
time, ResetTime - Last connection reset time, Afi-Safi - Address family, PfxSent
- Transmitted prefix counter, PfxRcvd - Recieved prefix counter
Neighbor Hostname AS State Uptime ResetTime MsgRcvd MsgSent Afi-Safi PfxSent PfxRcvd
------------- ------------- ---------- ----------- -------------- --------- ------- ------- ------------ ------- -------
peerlink.4094 SN5600-LEAF-2 4200120327 established 1 day, 2:36:59 2:37:04 31946 31946 ipv4-unicast 5 5
swp61 idle 2:37:04 0 0 ipv4-unicast 0 0
swp62 cumulus 4200000000 established 1 day, 2:36:59 2:37:04 31944 31945 ipv4-unicast 4 1
--- sudo vtysh -c "show ip bgp" (SN5600-LEAF-1 only) ---
BGP table version is 6, local router ID is 10.160.254.22, vrf id 0
Default local pref 100, local AS 4200120327
Status codes: s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath, + multipath nhg,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 0.0.0.0/0 swp62 0 0 4200000000 i
* i peerlink.4094 0 100 0 4200000000 i
*> 10.150.123.0/24 0.0.0.0(SN5600-LEAF-1)
0 32768 ?
* i peerlink.4094 0 100 0 ?
*> 10.150.124.0/24 0.0.0.0(SN5600-LEAF-1)
0 32768 ?
* i peerlink.4094 0 100 0 ?
*> 10.150.125.0/24 0.0.0.0(SN5600-LEAF-1)
0 32768 ?
* i peerlink.4094 0 100 0 ?
*> 10.160.254.22/32 0.0.0.0(SN5600-LEAF-1)
0 32768 ?
*>i10.160.254.23/32 peerlink.4094 0 100 0 ?
Displayed 6 routes and 10 total paths
SN2201 – IPMI Switch for Out-of-Band Management#
All the BMCs are in the same subnet; configure all switch ports connected to the BMCs in the same VLAN. This IPMI network should be accessible from the Management Ethernet to allow the BCM head nodes to control the BMCs. In this example, the IPMI network is terminated on the SN2201 switch and advertised to the customer OOB network with BGP. It is recommended to add redundant uplinks to the customer OOB network.
Example configuration for the SN2201 switch.
nv set system hostname oob-mgmt-switch-leaf-1
#<Basic management configuration>
#
# Set VLAN 101 for BMC ports. Adjust according to the customer specification.
nv set bridge domain br_default vlan 101
#
# Enable the BMC ports as access ports in VLAN 101.
nv set interface swp1-47 bridge domain br_default access 101
nv set interface swp1-48 type swp
#
# Configure the IPMI/BMC/OOB SVI gateway on the OOB switch.
nv set interface vlan101 description IPMI/BMC/OOB
nv set interface vlan101 ipv4 address 10.150.123.1/24
nv set interface vlan101 type svi
nv set interface vlan101 vlan 101
#
# Uplink to customer OOB network with BGP unnumbered.
nv set router bgp autonomous-system 65001
nv set router bgp state enabled
nv set router bgp router-id 10.0.0.1
nv set vrf default router bgp address-family ipv4-unicast state enabled
nv set vrf default router bgp address-family ipv4-unicast redistribute connected state enabled
nv set vrf default router bgp state enabled
nv set vrf default router bgp neighbor swp48 peer-group oob-uplink
nv set vrf default router bgp neighbor swp48 type unnumbered
nv set vrf default router bgp peer-group oob-uplink address-family ipv4-unicast policy inbound route-map DEF_ONLY
nv set vrf default router bgp peer-group oob-uplink address-family ipv4-unicast policy outbound route-map IPMI-RTS-ONLY
nv set vrf default router bgp peer-group oob-uplink remote-as external
Refer to the appendix for complete switch configuration.
After the SN2201 switches are configured, verify that all out-of-band management interfaces are reachable from the network. For example, confirm that you can access the BMC, iLO, or iDRAC interface for each node.
Reference: Cumulus Linux 5.16 Configuration Guide
You can also use NVIDIA Air to simulate and model the network.
Air simulation topology
{
"format": "JSON",
"ztp": "#!/bin/bash\n# Sanitized NVIDIA Air ZTP script.\n# Credential and bootstrap material removed for publication.\nexit 0\n#CUMULUS-AUTOPROVISIONING",
"content": {
"nodes": {
"BCM-headnode-1": {
"cpu": 1,
"memory": 1024,
"storage": 10,
"positioning": {
"x": 107,
"y": 471
},
"nic_model": "virtio",
"cpu_mode": "custom",
"cpu_options": [],
"secureboot": false,
"os": "generic/ubuntu2204",
"labels": {
"group": "SERVER",
"model": "ubuntu-server"
},
"management_interfaces": {
"eth0": {
"ip": "192.168.200.2",
"mac_address": "48:B0:2D:00:07:69"
}
},
"oob": false,
"storage_pci": null,
"pxehost": false,
"boot": "hd",
"features": {
"uefi": false
}
},
"BCM-headnode-2": {
"cpu": 1,
"memory": 1024,
"storage": 10,
"positioning": {
"x": 382,
"y": 471
},
"nic_model": "virtio",
"cpu_mode": "custom",
"cpu_options": [],
"secureboot": false,
"os": "generic/ubuntu2204",
"labels": {
"group": "SERVER",
"model": "ubuntu-server"
},
"management_interfaces": {
"eth0": {
"ip": "192.168.200.3",
"mac_address": "48:B0:2D:00:0A:02"
}
},
"oob": false,
"storage_pci": null,
"pxehost": false,
"boot": "hd",
"features": {
"uefi": false
}
},
"dgx-b300-01": {
"cpu": 1,
"memory": 1024,
"storage": 10,
"positioning": {
"x": 107,
"y": 823
},
"nic_model": "virtio",
"cpu_mode": "custom",
"cpu_options": [],
"secureboot": false,
"os": "generic/ubuntu2204",
"labels": {
"group": "SERVER",
"model": "ubuntu-server"
},
"management_interfaces": {
"eth0": {
"ip": "192.168.200.7",
"mac_address": "48:B0:2D:00:0A:17"
}
},
"oob": false,
"storage_pci": null,
"pxehost": false,
"boot": "hd",
"features": {
"uefi": false
}
},
"dgx-b300-02": {
"cpu": 1,
"memory": 1024,
"storage": 10,
"positioning": {
"x": 382,
"y": 823
},
"nic_model": "virtio",
"cpu_mode": "custom",
"cpu_options": [],
"secureboot": false,
"os": "generic/ubuntu2204",
"labels": {
"group": "SERVER",
"model": "ubuntu-server"
},
"management_interfaces": {
"eth0": {
"ip": "192.168.200.8",
"mac_address": "48:B0:2D:00:0A:1A"
}
},
"oob": false,
"storage_pci": null,
"pxehost": false,
"boot": "hd",
"features": {
"uefi": false
}
},
"dgx-b300-03": {
"cpu": 1,
"memory": 1024,
"storage": 10,
"positioning": {
"x": 657,
"y": 823
},
"nic_model": "virtio",
"cpu_mode": "custom",
"cpu_options": [],
"secureboot": false,
"os": "generic/ubuntu2204",
"labels": {
"group": "SERVER",
"model": "ubuntu-server"
},
"management_interfaces": {
"eth0": {
"ip": "192.168.200.9",
"mac_address": "48:B0:2D:00:0A:1D"
}
},
"oob": false,
"storage_pci": null,
"pxehost": false,
"boot": "hd",
"features": {
"uefi": false
}
},
"dgx-b300-04": {
"cpu": 1,
"memory": 1024,
"storage": 10,
"positioning": {
"x": 932,
"y": 823
},
"nic_model": "virtio",
"cpu_mode": "custom",
"cpu_options": [],
"secureboot": false,
"os": "generic/ubuntu2204",
"labels": {
"group": "SERVER",
"model": "ubuntu-server"
},
"management_interfaces": {
"eth0": {
"ip": "192.168.200.10",
"mac_address": "48:B0:2D:00:0A:20"
}
},
"oob": false,
"storage_pci": null,
"pxehost": false,
"boot": "hd",
"features": {
"uefi": false
}
},
"dgx-b300-05": {
"cpu": 1,
"memory": 1024,
"storage": 10,
"positioning": {
"x": 1207,
"y": 823
},
"nic_model": "virtio",
"cpu_mode": "custom",
"cpu_options": [],
"secureboot": false,
"os": "generic/ubuntu2204",
"labels": {
"group": "SERVER",
"model": "ubuntu-server"
},
"management_interfaces": {
"eth0": {
"ip": "192.168.200.11",
"mac_address": "48:B0:2D:00:0A:23"
}
},
"oob": false,
"storage_pci": null,
"pxehost": false,
"boot": "hd",
"features": {
"uefi": false
}
},
"dgx-b300-06": {
"cpu": 1,
"memory": 1024,
"storage": 10,
"positioning": {
"x": 107,
"y": 999
},
"nic_model": "virtio",
"cpu_mode": "custom",
"cpu_options": [],
"secureboot": false,
"os": "generic/ubuntu2204",
"labels": {
"group": "SERVER",
"model": "ubuntu-server"
},
"management_interfaces": {
"eth0": {
"ip": "192.168.200.12",
"mac_address": "48:B0:2D:00:0A:26"
}
},
"oob": false,
"storage_pci": null,
"pxehost": false,
"boot": "hd",
"features": {
"uefi": false
}
},
"dgx-b300-07": {
"cpu": 1,
"memory": 1024,
"storage": 10,
"positioning": {
"x": 382,
"y": 999
},
"nic_model": "virtio",
"cpu_mode": "custom",
"cpu_options": [],
"secureboot": false,
"os": "generic/ubuntu2204",
"labels": {
"group": "SERVER",
"model": "ubuntu-server"
},
"management_interfaces": {
"eth0": {
"ip": "192.168.200.13",
"mac_address": "48:B0:2D:00:0A:29"
}
},
"oob": false,
"storage_pci": null,
"pxehost": false,
"boot": "hd",
"features": {
"uefi": false
}
},
"dgx-b300-08": {
"cpu": 1,
"memory": 1024,
"storage": 10,
"positioning": {
"x": 657,
"y": 999
},
"nic_model": "virtio",
"cpu_mode": "custom",
"cpu_options": [],
"secureboot": false,
"os": "generic/ubuntu2204",
"labels": {
"group": "SERVER",
"model": "ubuntu-server"
},
"management_interfaces": {
"eth0": {
"ip": "192.168.200.14",
"mac_address": "48:B0:2D:00:0A:2C"
}
},
"oob": false,
"storage_pci": null,
"pxehost": false,
"boot": "hd",
"features": {
"uefi": false
}
},
"dgx-b300-09": {
"cpu": 1,
"memory": 1024,
"storage": 10,
"positioning": {
"x": 932,
"y": 999
},
"nic_model": "virtio",
"cpu_mode": "custom",
"cpu_options": [],
"secureboot": false,
"os": "generic/ubuntu2204",
"labels": {
"group": "SERVER",
"model": "ubuntu-server"
},
"management_interfaces": {
"eth0": {
"ip": "192.168.200.15",
"mac_address": "48:B0:2D:00:0A:2F"
}
},
"oob": false,
"storage_pci": null,
"pxehost": false,
"boot": "hd",
"features": {
"uefi": false
}
},
"dgx-b300-10": {
"cpu": 1,
"memory": 1024,
"storage": 10,
"positioning": {
"x": 1207,
"y": 999
},
"nic_model": "virtio",
"cpu_mode": "custom",
"cpu_options": [],
"secureboot": false,
"os": "generic/ubuntu2204",
"labels": {
"group": "SERVER",
"model": "ubuntu-server"
},
"management_interfaces": {
"eth0": {
"ip": "192.168.200.16",
"mac_address": "48:B0:2D:00:0A:32"
}
},
"oob": false,
"storage_pci": null,
"pxehost": false,
"boot": "hd",
"features": {
"uefi": false
}
},
"k8s-system-admin-01": {
"cpu": 1,
"memory": 1024,
"storage": 10,
"positioning": {
"x": 657,
"y": 471
},
"nic_model": "virtio",
"cpu_mode": "custom",
"cpu_options": [],
"secureboot": false,
"os": "generic/ubuntu2204",
"labels": {
"group": "SERVER",
"model": "ubuntu-server"
},
"management_interfaces": {
"eth0": {
"ip": "192.168.200.17",
"mac_address": "48:B0:2D:00:0A:05"
}
},
"oob": false,
"storage_pci": null,
"pxehost": false,
"boot": "hd",
"features": {
"uefi": false
}
},
"k8s-system-admin-02": {
"cpu": 1,
"memory": 1024,
"storage": 10,
"positioning": {
"x": 932,
"y": 471
},
"nic_model": "virtio",
"cpu_mode": "custom",
"cpu_options": [],
"secureboot": false,
"os": "generic/ubuntu2204",
"labels": {
"group": "SERVER",
"model": "ubuntu-server"
},
"management_interfaces": {
"eth0": {
"ip": "192.168.200.18",
"mac_address": "48:B0:2D:00:0A:08"
}
},
"oob": false,
"storage_pci": null,
"pxehost": false,
"boot": "hd",
"features": {
"uefi": false
}
},
"k8s-system-admin-03": {
"cpu": 1,
"memory": 1024,
"storage": 10,
"positioning": {
"x": 1207,
"y": 471
},
"nic_model": "virtio",
"cpu_mode": "custom",
"cpu_options": [],
"secureboot": false,
"os": "generic/ubuntu2204",
"labels": {
"group": "SERVER",
"model": "ubuntu-server"
},
"management_interfaces": {
"eth0": {
"ip": "192.168.200.19",
"mac_address": "48:B0:2D:00:0A:0B"
}
},
"oob": false,
"storage_pci": null,
"pxehost": false,
"boot": "hd",
"features": {
"uefi": false
}
},
"k8s-system-user-01": {
"cpu": 1,
"memory": 1024,
"storage": 10,
"positioning": {
"x": 1375,
"y": 550
},
"nic_model": "virtio",
"cpu_mode": "custom",
"cpu_options": [],
"secureboot": false,
"os": "generic/ubuntu2204",
"labels": {
"group": "SERVER",
"model": "ubuntu-server"
},
"management_interfaces": {
"eth0": {
"ip": "192.168.200.20",
"mac_address": "48:B0:2D:00:0A:0E"
}
},
"oob": false,
"storage_pci": null,
"pxehost": false,
"boot": "hd",
"features": {
"uefi": false
}
},
"k8s-system-user-02": {
"cpu": 1,
"memory": 1024,
"storage": 10,
"positioning": {
"x": 1650,
"y": 550
},
"nic_model": "virtio",
"cpu_mode": "custom",
"cpu_options": [],
"secureboot": false,
"os": "generic/ubuntu2204",
"labels": {
"group": "SERVER",
"model": "ubuntu-server"
},
"management_interfaces": {
"eth0": {
"ip": "192.168.200.21",
"mac_address": "48:B0:2D:00:0A:11"
}
},
"oob": false,
"storage_pci": null,
"pxehost": false,
"boot": "hd",
"features": {
"uefi": false
}
},
"k8s-system-user-03": {
"cpu": 1,
"memory": 1024,
"storage": 10,
"positioning": {
"x": 1925,
"y": 550
},
"nic_model": "virtio",
"cpu_mode": "custom",
"cpu_options": [],
"secureboot": false,
"os": "generic/ubuntu2204",
"labels": {
"group": "SERVER",
"model": "ubuntu-server"
},
"management_interfaces": {
"eth0": {
"ip": "192.168.200.22",
"mac_address": "48:B0:2D:00:0A:14"
}
},
"oob": false,
"storage_pci": null,
"pxehost": false,
"boot": "hd",
"features": {
"uefi": false
}
},
"slogin-01": {
"cpu": 1,
"memory": 1024,
"storage": 10,
"positioning": {
"x": 2200,
"y": 550
},
"nic_model": "virtio",
"cpu_mode": "custom",
"cpu_options": [],
"secureboot": false,
"os": "generic/ubuntu2204",
"labels": {
"group": "SERVER",
"model": "ubuntu-server"
},
"management_interfaces": {
"eth0": {
"ip": "192.168.200.23",
"mac_address": "48:B0:2D:00:0A:35"
}
},
"oob": false,
"storage_pci": null,
"pxehost": false,
"boot": "hd",
"features": {
"uefi": false
}
},
"slogin-02": {
"cpu": 1,
"memory": 1024,
"storage": 10,
"positioning": {
"x": 2475,
"y": 550
},
"nic_model": "virtio",
"cpu_mode": "custom",
"cpu_options": [],
"secureboot": false,
"os": "generic/ubuntu2204",
"labels": {
"group": "SERVER",
"model": "ubuntu-server"
},
"management_interfaces": {
"eth0": {
"ip": "192.168.200.24",
"mac_address": "48:B0:2D:00:0A:38"
}
},
"oob": false,
"storage_pci": null,
"pxehost": false,
"boot": "hd",
"features": {
"uefi": false
}
},
"SN5600-Customer-Gw": {
"cpu": 2,
"memory": 4096,
"storage": 20,
"positioning": {
"x": 139,
"y": 30
},
"nic_model": "virtio",
"cpu_mode": "custom",
"cpu_options": [],
"secureboot": false,
"os": "cumulus-vx-5.16.1",
"labels": {
"group": "SWITCH",
"model": "SN5600"
},
"management_interfaces": {
"eth0": {
"ip": "192.168.200.4",
"mac_address": "48:B0:2D:00:09:EE"
}
},
"oob": false,
"storage_pci": null,
"pxehost": false,
"boot": "hd",
"features": {
"uefi": false
}
},
"SN5600-LEAF-1": {
"cpu": 2,
"memory": 4096,
"storage": 20,
"positioning": {
"x": 275,
"y": 275
},
"nic_model": "virtio",
"cpu_mode": "custom",
"cpu_options": [],
"secureboot": false,
"os": "cumulus-vx-5.16.1",
"labels": {
"group": "SWITCH",
"model": "SN5600"
},
"management_interfaces": {
"eth0": {
"ip": "192.168.200.5",
"mac_address": "48:B0:2D:00:02:8C"
}
},
"oob": false,
"storage_pci": null,
"pxehost": false,
"boot": "hd",
"features": {
"uefi": false
}
},
"SN5600-LEAF-2": {
"cpu": 2,
"memory": 4096,
"storage": 20,
"positioning": {
"x": 550,
"y": 275
},
"nic_model": "virtio",
"cpu_mode": "custom",
"cpu_options": [],
"secureboot": false,
"os": "cumulus-vx-5.16.1",
"labels": {
"group": "SWITCH",
"model": "SN5600"
},
"management_interfaces": {
"eth0": {
"ip": "192.168.200.6",
"mac_address": "48:B0:2D:00:05:11"
}
},
"oob": false,
"storage_pci": null,
"pxehost": false,
"boot": "hd",
"features": {
"uefi": false
}
}
},
"links": [
[
{
"interface": "eth1",
"node": "BCM-headnode-1",
"mac": "48:B0:2D:00:07:6B"
},
{
"interface": "swp5",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:78"
}
],
[
{
"interface": "eth2",
"node": "BCM-headnode-1",
"mac": "48:B0:2D:00:07:6A"
},
{
"interface": "swp5",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:FD"
}
],
[
{
"interface": "eth1",
"node": "BCM-headnode-2",
"mac": "48:B0:2D:00:0A:03"
},
{
"interface": "swp6",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:60"
}
],
[
{
"interface": "eth2",
"node": "BCM-headnode-2",
"mac": "48:B0:2D:00:0A:04"
},
{
"interface": "swp6",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:E5"
}
],
[
{
"interface": "eth1",
"node": "dgx-b300-01",
"mac": "48:B0:2D:00:0A:18"
},
{
"interface": "swp15",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:96"
}
],
[
{
"interface": "eth2",
"node": "dgx-b300-01",
"mac": "48:B0:2D:00:0A:19"
},
{
"interface": "swp15",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:1B"
}
],
[
{
"interface": "eth1",
"node": "dgx-b300-02",
"mac": "48:B0:2D:00:0A:1B"
},
{
"interface": "swp16",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:79"
}
],
[
{
"interface": "eth2",
"node": "dgx-b300-02",
"mac": "48:B0:2D:00:0A:1C"
},
{
"interface": "swp16",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:FE"
}
],
[
{
"interface": "eth1",
"node": "dgx-b300-03",
"mac": "48:B0:2D:00:0A:1E"
},
{
"interface": "swp17",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:7A"
}
],
[
{
"interface": "eth2",
"node": "dgx-b300-03",
"mac": "48:B0:2D:00:0A:1F"
},
{
"interface": "swp17",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:FF"
}
],
[
{
"interface": "eth1",
"node": "dgx-b300-04",
"mac": "48:B0:2D:00:0A:21"
},
{
"interface": "swp18",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:8A"
}
],
[
{
"interface": "eth2",
"node": "dgx-b300-04",
"mac": "48:B0:2D:00:0A:22"
},
{
"interface": "swp18",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:0F"
}
],
[
{
"interface": "eth1",
"node": "dgx-b300-05",
"mac": "48:B0:2D:00:0A:24"
},
{
"interface": "swp19",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:6B"
}
],
[
{
"interface": "eth2",
"node": "dgx-b300-05",
"mac": "48:B0:2D:00:0A:25"
},
{
"interface": "swp19",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:F0"
}
],
[
{
"interface": "eth1",
"node": "dgx-b300-06",
"mac": "48:B0:2D:00:0A:27"
},
{
"interface": "swp20",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:8E"
}
],
[
{
"interface": "eth2",
"node": "dgx-b300-06",
"mac": "48:B0:2D:00:0A:28"
},
{
"interface": "swp20",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:13"
}
],
[
{
"interface": "eth1",
"node": "dgx-b300-07",
"mac": "48:B0:2D:00:0A:2A"
},
{
"interface": "swp21",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:69"
}
],
[
{
"interface": "eth2",
"node": "dgx-b300-07",
"mac": "48:B0:2D:00:0A:2B"
},
{
"interface": "swp21",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:EE"
}
],
[
{
"interface": "eth1",
"node": "dgx-b300-08",
"mac": "48:B0:2D:00:0A:2D"
},
{
"interface": "swp22",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:90"
}
],
[
{
"interface": "eth2",
"node": "dgx-b300-08",
"mac": "48:B0:2D:00:0A:2E"
},
{
"interface": "swp22",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:15"
}
],
[
{
"interface": "eth1",
"node": "dgx-b300-09",
"mac": "48:B0:2D:00:0A:30"
},
{
"interface": "swp23",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:88"
}
],
[
{
"interface": "eth2",
"node": "dgx-b300-09",
"mac": "48:B0:2D:00:0A:31"
},
{
"interface": "swp23",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:0D"
}
],
[
{
"interface": "eth1",
"node": "dgx-b300-10",
"mac": "48:B0:2D:00:0A:33"
},
{
"interface": "swp24",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:73"
}
],
[
{
"interface": "eth2",
"node": "dgx-b300-10",
"mac": "48:B0:2D:00:0A:34"
},
{
"interface": "swp24",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:F8"
}
],
[
{
"interface": "eth1",
"node": "k8s-system-admin-01",
"mac": "48:B0:2D:00:0A:06"
},
{
"interface": "swp7",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:67"
}
],
[
{
"interface": "eth2",
"node": "k8s-system-admin-01",
"mac": "48:B0:2D:00:0A:07"
},
{
"interface": "swp7",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:EC"
}
],
[
{
"interface": "eth1",
"node": "k8s-system-admin-02",
"mac": "48:B0:2D:00:0A:09"
},
{
"interface": "swp8",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:7C"
}
],
[
{
"interface": "eth2",
"node": "k8s-system-admin-02",
"mac": "48:B0:2D:00:0A:0A"
},
{
"interface": "swp8",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:01"
}
],
[
{
"interface": "eth1",
"node": "k8s-system-admin-03",
"mac": "48:B0:2D:00:0A:0C"
},
{
"interface": "swp9",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:70"
}
],
[
{
"interface": "eth2",
"node": "k8s-system-admin-03",
"mac": "48:B0:2D:00:0A:0D"
},
{
"interface": "swp9",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:F5"
}
],
[
{
"interface": "eth1",
"node": "k8s-system-user-01",
"mac": "48:B0:2D:00:0A:0F"
},
{
"interface": "swp10",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:89"
}
],
[
{
"interface": "eth2",
"node": "k8s-system-user-01",
"mac": "48:B0:2D:00:0A:10"
},
{
"interface": "swp10",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:0E"
}
],
[
{
"interface": "eth1",
"node": "k8s-system-user-02",
"mac": "48:B0:2D:00:0A:12"
},
{
"interface": "swp11",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:65"
}
],
[
{
"interface": "eth2",
"node": "k8s-system-user-02",
"mac": "48:B0:2D:00:0A:13"
},
{
"interface": "swp11",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:EA"
}
],
[
{
"interface": "eth1",
"node": "k8s-system-user-03",
"mac": "48:B0:2D:00:0A:15"
},
{
"interface": "swp12",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:66"
}
],
[
{
"interface": "eth2",
"node": "k8s-system-user-03",
"mac": "48:B0:2D:00:0A:16"
},
{
"interface": "swp12",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:EB"
}
],
[
{
"interface": "eth1",
"node": "slogin-01",
"mac": "48:B0:2D:00:0A:36"
},
{
"interface": "swp13",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:8B"
}
],
[
{
"interface": "eth2",
"node": "slogin-01",
"mac": "48:B0:2D:00:0A:37"
},
{
"interface": "swp13",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:10"
}
],
[
{
"interface": "eth1",
"node": "slogin-02",
"mac": "48:B0:2D:00:0A:39"
},
{
"interface": "swp14",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:68"
}
],
[
{
"interface": "eth2",
"node": "slogin-02",
"mac": "48:B0:2D:00:0A:3A"
},
{
"interface": "swp14",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:ED"
}
],
[
{
"interface": "swp1",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:FF"
},
{
"interface": "swp62",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:9C"
}
],
[
{
"interface": "swp2",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:E4"
},
{
"interface": "swp62",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:21"
}
],
[
{
"interface": "swp63",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:61"
},
{
"interface": "swp63",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:E6"
}
],
[
{
"interface": "swp64",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:86"
},
{
"interface": "swp64",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:0B"
}
],
[
{
"interface": "swp10",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:EB"
},
"unconnected"
],
[
{
"interface": "swp11",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:C7"
},
"unconnected"
],
[
{
"interface": "swp12",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:C8"
},
"unconnected"
],
[
{
"interface": "swp13",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:ED"
},
"unconnected"
],
[
{
"interface": "swp14",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:CA"
},
"unconnected"
],
[
{
"interface": "swp15",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:F8"
},
"unconnected"
],
[
{
"interface": "swp16",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:DB"
},
"unconnected"
],
[
{
"interface": "swp17",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:DC"
},
"unconnected"
],
[
{
"interface": "swp18",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:EC"
},
"unconnected"
],
[
{
"interface": "swp19",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:CD"
},
"unconnected"
],
[
{
"interface": "swp20",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:F0"
},
"unconnected"
],
[
{
"interface": "swp21",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:CB"
},
"unconnected"
],
[
{
"interface": "swp22",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:F2"
},
"unconnected"
],
[
{
"interface": "swp23",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:EA"
},
"unconnected"
],
[
{
"interface": "swp24",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:D5"
},
"unconnected"
],
[
{
"interface": "swp25",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:D0"
},
"unconnected"
],
[
{
"interface": "swp26",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:F9"
},
"unconnected"
],
[
{
"interface": "swp27",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:D6"
},
"unconnected"
],
[
{
"interface": "swp28",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:E5"
},
"unconnected"
],
[
{
"interface": "swp29",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:F4"
},
"unconnected"
],
[
{
"interface": "swp3",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:D1"
},
"unconnected"
],
[
{
"interface": "swp30",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:D7"
},
"unconnected"
],
[
{
"interface": "swp31",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:FC"
},
"unconnected"
],
[
{
"interface": "swp32",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:D4"
},
"unconnected"
],
[
{
"interface": "swp33",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:EF"
},
"unconnected"
],
[
{
"interface": "swp34",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:0A:01"
},
"unconnected"
],
[
{
"interface": "swp35",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:D3"
},
"unconnected"
],
[
{
"interface": "swp36",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:CF"
},
"unconnected"
],
[
{
"interface": "swp37",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:FB"
},
"unconnected"
],
[
{
"interface": "swp38",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:C4"
},
"unconnected"
],
[
{
"interface": "swp39",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:FD"
},
"unconnected"
],
[
{
"interface": "swp4",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:F3"
},
"unconnected"
],
[
{
"interface": "swp40",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:E1"
},
"unconnected"
],
[
{
"interface": "swp41",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:0A:00"
},
"unconnected"
],
[
{
"interface": "swp42",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:CE"
},
"unconnected"
],
[
{
"interface": "swp43",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:E6"
},
"unconnected"
],
[
{
"interface": "swp44",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:F1"
},
"unconnected"
],
[
{
"interface": "swp45",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:E3"
},
"unconnected"
],
[
{
"interface": "swp46",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:F5"
},
"unconnected"
],
[
{
"interface": "swp47",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:C1"
},
"unconnected"
],
[
{
"interface": "swp48",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:E0"
},
"unconnected"
],
[
{
"interface": "swp49",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:C5"
},
"unconnected"
],
[
{
"interface": "swp5",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:DA"
},
"unconnected"
],
[
{
"interface": "swp50",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:DD"
},
"unconnected"
],
[
{
"interface": "swp51",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:DF"
},
"unconnected"
],
[
{
"interface": "swp52",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:FA"
},
"unconnected"
],
[
{
"interface": "swp53",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:CC"
},
"unconnected"
],
[
{
"interface": "swp54",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:D8"
},
"unconnected"
],
[
{
"interface": "swp55",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:F7"
},
"unconnected"
],
[
{
"interface": "swp56",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:F6"
},
"unconnected"
],
[
{
"interface": "swp57",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:E9"
},
"unconnected"
],
[
{
"interface": "swp58",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:C6"
},
"unconnected"
],
[
{
"interface": "swp59",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:D9"
},
"unconnected"
],
[
{
"interface": "swp6",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:C2"
},
"unconnected"
],
[
{
"interface": "swp60",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:E7"
},
"unconnected"
],
[
{
"interface": "swp61",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:E2"
},
"unconnected"
],
[
{
"interface": "swp62",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:FE"
},
"unconnected"
],
[
{
"interface": "swp63",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:C3"
},
"unconnected"
],
[
{
"interface": "swp64",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:E8"
},
"unconnected"
],
[
{
"interface": "swp7",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:C9"
},
"unconnected"
],
[
{
"interface": "swp8",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:DE"
},
"unconnected"
],
[
{
"interface": "swp9",
"node": "SN5600-Customer-Gw",
"mac": "48:B0:2D:00:09:D2"
},
"unconnected"
],
[
{
"interface": "swp1",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:9D"
},
"unconnected"
],
[
{
"interface": "swp2",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:82"
},
"unconnected"
],
[
{
"interface": "swp25",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:6E"
},
"unconnected"
],
[
{
"interface": "swp26",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:97"
},
"unconnected"
],
[
{
"interface": "swp27",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:74"
},
"unconnected"
],
[
{
"interface": "swp28",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:83"
},
"unconnected"
],
[
{
"interface": "swp29",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:92"
},
"unconnected"
],
[
{
"interface": "swp3",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:6F"
},
"unconnected"
],
[
{
"interface": "swp30",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:75"
},
"unconnected"
],
[
{
"interface": "swp31",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:9A"
},
"unconnected"
],
[
{
"interface": "swp32",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:72"
},
"unconnected"
],
[
{
"interface": "swp33",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:8D"
},
"unconnected"
],
[
{
"interface": "swp34",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:9F"
},
"unconnected"
],
[
{
"interface": "swp35",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:71"
},
"unconnected"
],
[
{
"interface": "swp36",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:6D"
},
"unconnected"
],
[
{
"interface": "swp37",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:99"
},
"unconnected"
],
[
{
"interface": "swp38",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:62"
},
"unconnected"
],
[
{
"interface": "swp39",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:9B"
},
"unconnected"
],
[
{
"interface": "swp4",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:91"
},
"unconnected"
],
[
{
"interface": "swp40",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:7F"
},
"unconnected"
],
[
{
"interface": "swp41",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:9E"
},
"unconnected"
],
[
{
"interface": "swp42",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:6C"
},
"unconnected"
],
[
{
"interface": "swp43",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:84"
},
"unconnected"
],
[
{
"interface": "swp44",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:8F"
},
"unconnected"
],
[
{
"interface": "swp45",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:81"
},
"unconnected"
],
[
{
"interface": "swp46",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:93"
},
"unconnected"
],
[
{
"interface": "swp47",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:5F"
},
"unconnected"
],
[
{
"interface": "swp48",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:7E"
},
"unconnected"
],
[
{
"interface": "swp49",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:63"
},
"unconnected"
],
[
{
"interface": "swp50",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:7B"
},
"unconnected"
],
[
{
"interface": "swp51",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:7D"
},
"unconnected"
],
[
{
"interface": "swp52",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:98"
},
"unconnected"
],
[
{
"interface": "swp53",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:6A"
},
"unconnected"
],
[
{
"interface": "swp54",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:76"
},
"unconnected"
],
[
{
"interface": "swp55",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:95"
},
"unconnected"
],
[
{
"interface": "swp56",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:94"
},
"unconnected"
],
[
{
"interface": "swp57",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:87"
},
"unconnected"
],
[
{
"interface": "swp58",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:64"
},
"unconnected"
],
[
{
"interface": "swp59",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:77"
},
"unconnected"
],
[
{
"interface": "swp60",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:85"
},
"unconnected"
],
[
{
"interface": "swp61",
"node": "SN5600-LEAF-1",
"mac": "48:B0:2D:00:02:80"
},
"unconnected"
],
[
{
"interface": "swp1",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:22"
},
"unconnected"
],
[
{
"interface": "swp2",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:07"
},
"unconnected"
],
[
{
"interface": "swp25",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:F3"
},
"unconnected"
],
[
{
"interface": "swp26",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:1C"
},
"unconnected"
],
[
{
"interface": "swp27",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:F9"
},
"unconnected"
],
[
{
"interface": "swp28",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:08"
},
"unconnected"
],
[
{
"interface": "swp29",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:17"
},
"unconnected"
],
[
{
"interface": "swp3",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:F4"
},
"unconnected"
],
[
{
"interface": "swp30",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:FA"
},
"unconnected"
],
[
{
"interface": "swp31",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:1F"
},
"unconnected"
],
[
{
"interface": "swp32",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:F7"
},
"unconnected"
],
[
{
"interface": "swp33",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:12"
},
"unconnected"
],
[
{
"interface": "swp34",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:24"
},
"unconnected"
],
[
{
"interface": "swp35",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:F6"
},
"unconnected"
],
[
{
"interface": "swp36",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:F2"
},
"unconnected"
],
[
{
"interface": "swp37",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:1E"
},
"unconnected"
],
[
{
"interface": "swp38",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:E7"
},
"unconnected"
],
[
{
"interface": "swp39",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:20"
},
"unconnected"
],
[
{
"interface": "swp4",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:16"
},
"unconnected"
],
[
{
"interface": "swp40",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:04"
},
"unconnected"
],
[
{
"interface": "swp41",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:23"
},
"unconnected"
],
[
{
"interface": "swp42",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:F1"
},
"unconnected"
],
[
{
"interface": "swp43",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:09"
},
"unconnected"
],
[
{
"interface": "swp44",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:14"
},
"unconnected"
],
[
{
"interface": "swp45",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:06"
},
"unconnected"
],
[
{
"interface": "swp46",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:18"
},
"unconnected"
],
[
{
"interface": "swp47",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:E4"
},
"unconnected"
],
[
{
"interface": "swp48",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:03"
},
"unconnected"
],
[
{
"interface": "swp49",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:E8"
},
"unconnected"
],
[
{
"interface": "swp50",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:00"
},
"unconnected"
],
[
{
"interface": "swp51",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:02"
},
"unconnected"
],
[
{
"interface": "swp52",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:1D"
},
"unconnected"
],
[
{
"interface": "swp53",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:EF"
},
"unconnected"
],
[
{
"interface": "swp54",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:FB"
},
"unconnected"
],
[
{
"interface": "swp55",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:1A"
},
"unconnected"
],
[
{
"interface": "swp56",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:19"
},
"unconnected"
],
[
{
"interface": "swp57",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:0C"
},
"unconnected"
],
[
{
"interface": "swp58",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:E9"
},
"unconnected"
],
[
{
"interface": "swp59",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:04:FC"
},
"unconnected"
],
[
{
"interface": "swp60",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:0A"
},
"unconnected"
],
[
{
"interface": "swp61",
"node": "SN5600-LEAF-2",
"mac": "48:B0:2D:00:05:05"
},
"unconnected"
]
],
"oob": true
},
"name": "DGXB300-Network"
}
InfiniBand Compute Fabric#
The InfiniBand compute fabric for B300 deployments is built on NVIDIA XDR InfiniBand switches. NVIDIA UFM is used to manage the InfiniBand XDR fabric. UFM Ethernet management interfaces are connected to both SN5600 switches in the same manner as other management nodes. These interfaces are used to access the UFM UI for monitoring and management of the XDR fabric.
UFM connects to the XDR switches for monitoring and management of the XDR fabric through the dedicated FNM (Fabric Network Management) ports on the XDR switches. FNM ports are dedicated in-band management ports on the InfiniBand XDR switches.
XDR InfiniBand Compute Fabric Configuration#
InfiniBand XDR switch configuration and bring-up can be manual or automated. For large-scale deployments, contact your NVIDIA PTAM for NVIS engagement and assistance with automated XDR switch deployment.
For small-scale deployments, manual configuration is recommended. The following sections describe the manual configuration steps for XDR switches.
XDR Switch Bring-Up and Configuration#
Follow the XDR Switch Bring-Up and Configuration Guide to configure the XDR switches.
Verify that the InfiniBand fabric interfaces to the DGX nodes and FNM ports for UFM, where applicable, are configured and enabled.
Click to expand XDR interface verification output
admin@kant-xdr-leaf-2:~$ nv show int
Interface State Speed MTU Type Description IB Speed IB Subnet Logical State Physical State Summary
--------- ----- ----- ----- -------- ----------- -------- ------------------ ------------- -------------- ---------------------------
eth0 up 1G 1500 eth ip.address: 10.150.124.41/24
eth1 down 1500 eth
fnm1 up 400G 4096 fnm ndr infiniband-default Active LinkUp
sw1p1 up 800G 4096 ib xdr infiniband-default Active LinkUp
sw1p2 up 800G 4096 ib xdr infiniband-default Active LinkUp
sw2p1 up 800G 4096 ib xdr infiniband-default Active LinkUp
sw2p2 up 800G 4096 ib xdr infiniband-default Active LinkUp
sw3p1 up 800G 4096 ib xdr infiniband-default Active LinkUp
sw3p2 up 800G 4096 ib xdr infiniband-default Active LinkUp
# FNM Port status after UFM is up.
admin@kant-xdr-leaf-2:~$ nv show int fnm1
operational applied pending
----------------------------------- ------------------ ------- -------
link
auto-negotiate on on on
duplex full full
speed 400G 1G 1G
counters
in-bytes 786.49 MB
in-pkts 2864264
in-drops 0
in-errors 0
out-bytes 786.28 MB
out-pkts 2862773
out-drops 0
out-errors 0
in-symbol-errors 0
out-wait 0
link-error-recovery 0
link-downed 0
port-rcv-remote-physical-errors 0
port-rcv-switch-relay-errors 2
port-rcv-constraint-errors 0
local-link-integrity-errors 0
qp1-drops 0
buffer-overrun-errors 0
rcv-icrc-errors 0
tx-parity-errors 0
unicast-in-pkts 2862773
unicast-out-pkts 2862773
multicast-in-pkts 1491
multicast-out-pkts 0
[diagnostics]
mtu 4096 4096 4096
op-vls VL0-VL1 VL0-VL7 VL0-VL7
lanes 4X 4X 4X
ib-speed ndr auto auto
state up up up
supported-ib-speed sdr,hdr,ndr,xdr
max-supported-mtu 4096
physical-state LinkUp
logical-state Active
supported-lanes 4X
vl-capabilities VL0-VL7
ib-subnet infiniband-default
round-trip-latency 1900 ns
type fnm fnm fnm
admin@kant-xdr-leaf-2:~$
UFM Bring-Up and Configuration#
To bring up the UFM appliance, follow the UFM appliance configuration steps to access the appliance through the BMC, in-band port, or serial console.
Configure the management Ethernet interfaces by editing the netplan file. The following sample netplan configuration is provided for reference. Because UFM is dual-homed to both SN5600 switches, a bond configuration provides redundancy. The bond interface uses a static IP address, and the default gateway is set to the SN5600 leaf switches.
Click to expand UFM netplan sample configuration
root@UFM:~# cat /etc/netplan/01-netcfg.yaml
network:
version: 2
renderer: networkd
ethernets:
enp139s0f0np0:
optional: true
enp139s0f1np1:
optional: true
bonds:
bond0:
interfaces:
- enp139s0f0np0
- enp139s0f1np1
addresses:
- 10.150.124.50/24
parameters:
mode: 802.3ad
lacp-rate: fast
transmit-hash-policy: layer3+4
mii-monitor-interval: 100
routes:
- to: default
via: 10.150.124.1
optional: true
Apply the configuration by running netplan apply, and verify connectivity to the SN5600 switches with ping.
Next, install and configure the license by following the UFM Enterprise getting started instructions.
Verify the license installation with the following commands:
Click to expand UFM license verification output
root@UFM:~# ufmcli
NVIDIA UFM Enterprise Appliance Management
UFM > enable
localhost # show ufm license
|------------------------------------------------------------------------------------------------------------------------------------------|
| Customer ID | SN | swName | Type | MAC Address | Exp. Date |Limit| Functionality | Status |
|------------------------------------------------------------------------------------------------------------------------------------------|
|NVES Generic |xxxxxxxxxxxxx|UFM Enterprise |Subscription |c8:4b:d6:f1:5d:f0|2027-01-29 |10 |1 Years |Valid |
|------------------------------------------------------------------------------------------------------------------------------------------|
UFM #
Verify that the UFM server is running from the Linux shell. Exit ufmcli before running the command.
Click to expand UFM service status output
root@localhost:~# systemctl status ufm-enterprise.service
● ufm-enterprise.service - ufm-enterprise
Loaded: loaded (/etc/systemd/system/ufm-enterprise.service; enabled; preset: enabled)
Active: active (running) since Fri 2026-02-13 06:16:50 UTC; 4 months 22 days ago
Process: 11210 ExecStartPre=/usr/bin/docker rm -f ufm (code=exited, status=0/SUCCESS)
Process: 11219 ExecStartPre=/bin/bash -c hostname > /opt/ufm/files/indicators/master_host_name (code=exited, status=0/SUCCESS)
Process: 11222 ExecStartPre=/bin/sh -c ufm_timezone=$(timedatectl | grep "Time zone" | awk "{print \$3}"); echo $ufm_timezone > /tmp/ufm_timezone (code=exited, status=0/SUCCESS)
Main PID: 11231 (sh)
Tasks: 36 (limit: 76501)
Memory: 12.0M (peak: 17.2M)
CPU: 45min 47.887s
CGroup: /system.slice/ufm-enterprise.service
├─11231 /bin/sh -c "ufm_timezone=\$(cat /tmp/ufm_timezone); /usr/bin/docker run --rm -t --name=ufm --network=host --shm-size=256m --volume /opt/ufm/files:/opt/ufm/shared_config_fil>
└─11233 /usr/bin/docker run --rm -t --name=ufm --network=host --shm-size=256m --volume /opt/ufm/files:/opt/ufm/shared_config_files --volume /dev/log:/dev/log --volume /sys/fs/cgroup:/sys/fs/cgroup:ro >
Feb 13 06:16:58 localhost sh[11233]: [Fri Feb 13 06:16:58.471553 2026] [proxy:warn] [pid 1487] AH01146: Ignoring parameter 'timeout=300' for worker 'http://127.0.0.1/ufmRestInternal' because of worker sharing
Feb 13 06:16:58 localhost sh[11233]: [Fri Feb 13 06:16:58.471598 2026] [proxy:warn] [pid 1487] AH01146: Ignoring parameter 'retry=1' for worker 'http://127.0.0.1/ufmRestInternal' because of worker sharing
Feb 13 06:16:58 localhost sh[11233]: [Fri Feb 13 06:16:58.471602 2026] [proxy:warn] [pid 1487] AH01146: Ignoring parameter 'Keepalive=On' for worker 'http://127.0.0.1/ufmRestInternal' because of worker sharing
Feb 13 06:16:58 localhost sh[11233]: [Fri Feb 13 06:16:58.471604 2026] [proxy:warn] [pid 1487] AH01146: Ignoring parameter 'timeout=300' for worker 'http://127.0.0.1/ufmRestInternal' because of worker sharing
Feb 13 06:16:58 localhost sh[11233]: httpd not running, trying to start
Feb 13 06:16:58 localhost sh[11233]: Restarting apache2: [ OK ]
Feb 13 06:17:00 localhost sh[11233]: Starting AuthenticationServer: [ OK ]
Feb 13 06:17:00 localhost sh[11233]: Starting UFM main module: [ OK ]
Feb 13 06:17:03 localhost sh[11233]: Starting UnhealthyPorts: [ OK ]
Feb 13 06:17:03 localhost sh[11233]: Starting Telemetry Sampling: [ OK ]
Perform a service sanity check with the following command:
Click to expand UFM sanity check output
root@UFM:~# ufm_enterprise_sanity.sh
Checking Service...
Done
Checking Images...
Done
Checking Containers...
Done
Checking ufm REST server...
Done
Sanity tests completed successfully!
root@localhost:~#
Verify all the services are running:
Click to expand UFM status output
root@UFM:~# ufmcli
NVIDIA UFM Enterprise Appliance Management
UFM > enable
UFM # show ufm status
===========================================================================================
UFM Main Processes
===========================================================================================
ModelMain Process is : [ Running ]
Authentication Service Process is : [ Running ]
Opensm Process is : [ Running ]
Unhealthy Ports Process is : [ Running ]
UFM Health Process is : [ Running ]
UFM Primary Telemetry Process is : [ Running ]
UFM Secondary Telemetry Process is : [ Running ]
UFM Running
Ensure that the InfiniBand in-band fabric interface is configured and up on the UFM appliance.
Click to expand UFM fabric interface configuration output
root@UFM:/etc# grep -w "fabric_interface" /opt/ufm/files/conf/gv.cfg
fabric_interface = ib0
Verify that the InfiniBand fabric interface link to the FNM port on the XDR switch is up.
Click to expand UFM InfiniBand link status output
root@UFM:~# ibstat
CA 'mlx5_0'
CA type: MT4131
Number of ports: 1
Firmware version: 40.45.1020
Hardware version: 0
Node GUID: 0x549b2403005ac0fe
System image GUID: 0x549b2403005ac0fe
Port 1:
State: Active
Physical state: LinkUp
Rate: 400
Base lid: 1
LMC: 0
SM lid: 1
Capability mask: 0xa751ec48
Port GUID: 0x549b2403005ac0fe
Link layer: InfiniBand
CA 'mlx5_3'
CA type: MT4131
Number of ports: 1
Firmware version: 40.45.1020
Hardware version: 0
Node GUID: 0x549b2403005aa63e
System image GUID: 0x549b2403005aa63e
Port 1:
State: Active
Physical state: LinkUp
Rate: 400
Base lid: 2
LMC: 0
SM lid: 1
Capability mask: 0xa751ec48
Port GUID: 0x549b2403005aa63e
Link layer: InfiniBand
Log in to the web UI using the default credentials and IP address configured on the UFM appliance.
Default UFM web UI credentials are as follows:
Username:
adminPassword:
123456
For detailed UFM configuration steps, refer to UFM Guides.
Note
HA mode is recommended when UFM high availability is required. Refer to the UFM HA Pair configuration guide for installing and configuring a UFM HA pair.
InfiniBand/Ethernet Storage Fabric Specific Configurations#
A DGX SuperPOD/BasePOD typically also includes dedicated storage, but the configuration is outside the scope of this document. Contact the vendor of the storage solution being used for instructions on configuring the high-performance storage portions of a SuperPOD/DGX BasePOD.
Appendix A - Switch Configuration#
SN5600-1/TOR1 Configuration#
Click to expand SN5600-1/TOR1 configuration
# Bridge domain VLANs
nv set bridge domain br_default vlan 100,102-103
# Host-facing MLAG bonds and access VLAN assignments
nv set interface bond5 bond member swp5
nv set interface bond5 bond mlag id 5
nv set interface bond5 description BCM-headnode-1
nv set interface bond5-14 bridge domain br_default access 102
nv set interface bond5-24 bond lacp-bypass enabled
nv set interface bond5-24 bond mlag state enabled
nv set interface bond5-24 type bond
nv set interface bond6 bond member swp6
nv set interface bond6 bond mlag id 6
nv set interface bond6 description BCM-headnode-2
nv set interface bond7 bond member swp7
nv set interface bond7 bond mlag id 7
nv set interface bond7 description k8s-system-admin-01
nv set interface bond8 bond member swp8
nv set interface bond8 bond mlag id 8
nv set interface bond8 description k8s-system-admin-02
nv set interface bond9 bond member swp9
nv set interface bond9 bond mlag id 9
nv set interface bond9 description k8s-system-admin-03
nv set interface bond10 bond member swp10
nv set interface bond10 bond mlag id 10
nv set interface bond10 description k8s-system-user-01
nv set interface bond11 bond member swp11
nv set interface bond11 bond mlag id 11
nv set interface bond11 description k8s-system-user-02
nv set interface bond12 bond member swp12
nv set interface bond12 bond mlag id 12
nv set interface bond12 description k8s-system-user-03
nv set interface bond13 bond member swp13
nv set interface bond13 bond mlag id 13
nv set interface bond13 description slogin-01
nv set interface bond14 bond member swp14
nv set interface bond14 bond mlag id 14
nv set interface bond14 description slogin-02
nv set interface bond15 bond member swp15
nv set interface bond15 bond mlag id 15
nv set interface bond15 description dgx-b300-01
nv set interface bond15-24 bridge domain br_default access 103
nv set interface bond16 bond member swp16
nv set interface bond16 bond mlag id 16
nv set interface bond16 description dgx-b300-02
nv set interface bond17 bond member swp17
nv set interface bond17 bond mlag id 17
nv set interface bond17 description dgx-b300-03
nv set interface bond18 bond member swp18
nv set interface bond18 bond mlag id 18
nv set interface bond18 description dgx-b300-04
nv set interface bond19 bond member swp19
nv set interface bond19 bond mlag id 19
nv set interface bond19 description dgx-b300-05
nv set interface bond20 bond member swp20
nv set interface bond20 bond mlag id 20
nv set interface bond20 description dgx-b300-06
nv set interface bond21 bond member swp21
nv set interface bond21 bond mlag id 21
nv set interface bond21 description dgx-b300-07
nv set interface bond22 bond member swp22
nv set interface bond22 bond mlag id 22
nv set interface bond22 description dgx-b300-08
nv set interface bond23 bond member swp23
nv set interface bond23 bond mlag id 23
nv set interface bond23 description dgx-b300-09
nv set interface bond24 bond member swp24
nv set interface bond24 bond mlag id 24
nv set interface bond24 description dgx-b300-10
# Management interface
nv set interface eth0 ipv4 dhcp-client set-hostname enabled
nv set interface eth0 ipv4 dhcp-client state enabled
nv set interface eth0 type eth
nv set interface eth0 vrf mgmt
# Loopback interface for BGP and MLAG backup routing
nv set interface lo ipv4 address 10.160.254.22/32
nv set interface lo type loopback
# MLAG peerlink
nv set interface peerlink bond member swp63-64
nv set interface peerlink type peerlink
nv set interface peerlink,swp63-64 description MLAG-peerlink-to-SN5600-LEAF-2
nv set interface peerlink.4094 base-interface peerlink
nv set interface peerlink.4094 type sub
nv set interface peerlink.4094 vlan 4094
# Customer BGP uplinks
nv set interface swp61 description BGP-uplink-secondary-to-SN5600-Customer-Gw
nv set interface swp61-64 type swp
nv set interface swp62 description BGP-uplink-to-SN5600-Customer-Gw
# SVI and VRR gateway configuration for in-band networks
nv set interface vlan102-103 ipv4 vrr mac-address 00:1c:73:aa:bb:04
nv set interface vlan102-103 ipv4 vrr state enabled
nv set interface vlan102-103 ipv4 vrr vrr-state up
nv set interface vlan102-103 type svi
nv set interface vlan102 description Internalnet/Control-Nodes
nv set interface vlan102 ipv4 address 10.150.124.2/24
nv set interface vlan102 ipv4 vrr address 10.150.124.1/24
nv set interface vlan102 vlan 102
nv set interface vlan103 description DGXnet1
nv set interface vlan103 ipv4 address 10.150.125.2/24
nv set interface vlan103 ipv4 vrr address 10.150.125.1/24
nv set interface vlan103 vlan 103
# MLAG global settings
nv set mlag backup 10.160.254.23
nv set mlag mac-address 44:38:39:FF:0A:00
nv set mlag peer-ip linklocal
nv set mlag priority 2048
nv set mlag state enabled
# BGP global settings
nv set router bgp autonomous-system 4200120327
nv set router bgp router-id 10.160.254.22
nv set router bgp state enabled
# BGP route policy for default-only inbound and local-routes-only outbound advertisements
nv set router policy prefix-list DEF_ONLY rule 10 action permit
nv set router policy prefix-list DEF_ONLY rule 10 match 0.0.0.0/0
nv set router policy prefix-list DEF_ONLY type ipv4
nv set router policy prefix-list LOCAL_ROUTES rule 10 action permit
nv set router policy prefix-list LOCAL_ROUTES rule 10 match 10.150.124.0/24
nv set router policy prefix-list LOCAL_ROUTES rule 20 action permit
nv set router policy prefix-list LOCAL_ROUTES rule 20 match 10.150.125.0/24
nv set router policy prefix-list LOCAL_ROUTES rule 30 action permit
nv set router policy prefix-list LOCAL_ROUTES rule 30 match 10.160.254.22/32
nv set router policy prefix-list LOCAL_ROUTES type ipv4
nv set router policy prefix-list NO_DEF rule 10 action permit
nv set router policy prefix-list NO_DEF rule 10 match 0.0.0.0/0
nv set router policy prefix-list NO_DEF type ipv4
nv set router policy route-map DEF_ONLY rule 5 action permit
nv set router policy route-map DEF_ONLY rule 5 match ip-prefix-list DEF_ONLY
nv set router policy route-map DEF_ONLY rule 5 match type ipv4
nv set router policy route-map DEF_ONLY rule 100 action deny
nv set router policy route-map LOCAL-RTS-ONLY rule 5 action deny
nv set router policy route-map LOCAL-RTS-ONLY rule 5 match ip-prefix-list NO_DEF
nv set router policy route-map LOCAL-RTS-ONLY rule 5 match type ipv4
nv set router policy route-map LOCAL-RTS-ONLY rule 10 action permit
nv set router policy route-map LOCAL-RTS-ONLY rule 10 match ip-prefix-list LOCAL_ROUTES
nv set router policy route-map LOCAL-RTS-ONLY rule 10 match type ipv4
nv set router policy route-map LOCAL-RTS-ONLY rule 100 action deny
# VRR and DHCP relay services
nv set router vrr state enabled
nv set service dhcp-relay default downstream-interface vlan102-103 server-group-name dhcp-servers
nv set service dhcp-relay default server-group dhcp-servers server 10.150.124.11
nv set service dhcp-relay default server-group dhcp-servers server 10.150.124.12
nv set service dhcp-relay default server-group dhcp-servers upstream-interface vlan102
# System users, roles, services, and management settings
nv set system aaa class nvapply action allow
nv set system aaa class nvapply command-path / permission all
nv set system aaa class nvshow action allow
nv set system aaa class nvshow command-path / permission ro
nv set system aaa class sudo action allow
nv set system aaa class sudo command-path / permission all
nv set system aaa role nvue-admin class nvapply
nv set system aaa role nvue-monitor class nvshow
nv set system aaa role system-admin class nvapply
nv set system aaa role system-admin class sudo
nv set system aaa user cumulus full-name cumulus,,,
nv set system aaa user cumulus hashed-password '*'
nv set system aaa user cumulus role system-admin
nv set system api state enabled
nv set system config auto-save state enabled
nv set system control-plane acl acl-default-dos inbound
nv set system control-plane acl acl-default-whitelist inbound
nv set system date-time timezone America/Los_Angeles
nv set system dns server 1.1.1.1
nv set system dns server 8.8.8.8
nv set system docker state enabled
nv set system docker vrf mgmt
nv set system hostname SN5600-LEAF-1
nv set system ntp server 0.cumulusnetworks.pool.ntp.org
nv set system ntp server 1.cumulusnetworks.pool.ntp.org
nv set system ntp server 2.cumulusnetworks.pool.ntp.org
nv set system ntp server 3.cumulusnetworks.pool.ntp.org
nv set system ntp server time.nist.gov
nv set system ntp state enabled
nv set system ntp vrf mgmt
nv set system ssh-server permit-root-login enabled
nv set system ssh-server state enabled
nv set system ssh-server vrf mgmt
nv set system wjh channel forwarding trigger l2
nv set system wjh channel forwarding trigger l3
nv set system wjh channel forwarding trigger tunnel
nv set system wjh state enabled
# VRF BGP address family and neighbor settings
nv set vrf default router bgp address-family ipv4-unicast redistribute connected state enabled
nv set vrf default router bgp address-family ipv4-unicast state enabled
nv set vrf default router bgp neighbor peerlink.4094 remote-as internal
nv set vrf default router bgp neighbor peerlink.4094 timers connection-retry 10
nv set vrf default router bgp neighbor peerlink.4094 timers hold 10
nv set vrf default router bgp neighbor peerlink.4094 timers keepalive 3
nv set vrf default router bgp neighbor peerlink.4094 timers route-advertisement auto
nv set vrf default router bgp neighbor peerlink.4094 type unnumbered
nv set vrf default router bgp neighbor swp61 address-family ipv4-unicast policy inbound route-map DEF_ONLY
nv set vrf default router bgp neighbor swp61 address-family ipv4-unicast policy outbound route-map LOCAL-RTS-ONLY
nv set vrf default router bgp neighbor swp61 remote-as external
nv set vrf default router bgp neighbor swp61 timers connection-retry 10
nv set vrf default router bgp neighbor swp61 timers hold 10
nv set vrf default router bgp neighbor swp61 timers keepalive 3
nv set vrf default router bgp neighbor swp61 timers route-advertisement auto
nv set vrf default router bgp neighbor swp61 type unnumbered
nv set vrf default router bgp neighbor swp62 address-family ipv4-unicast policy inbound route-map DEF_ONLY
nv set vrf default router bgp neighbor swp62 address-family ipv4-unicast policy outbound route-map LOCAL-RTS-ONLY
nv set vrf default router bgp neighbor swp62 remote-as external
nv set vrf default router bgp neighbor swp62 timers connection-retry 10
nv set vrf default router bgp neighbor swp62 timers hold 10
nv set vrf default router bgp neighbor swp62 timers keepalive 3
nv set vrf default router bgp neighbor swp62 timers route-advertisement auto
nv set vrf default router bgp neighbor swp62 type unnumbered
nv set vrf default router bgp state enabled
SN5600-2/TOR2 Configuration#
Click to expand SN5600-2/TOR2 configuration
# Bridge domain VLANs
nv set bridge domain br_default vlan 100,102-103
# Host-facing MLAG bonds and access VLAN assignments
nv set interface bond5 bond member swp5
nv set interface bond5 bond mlag id 5
nv set interface bond5 description BCM-headnode-1
nv set interface bond5-14 bridge domain br_default access 102
nv set interface bond5-24 bond lacp-bypass enabled
nv set interface bond5-24 bond mlag state enabled
nv set interface bond5-24 type bond
nv set interface bond6 bond member swp6
nv set interface bond6 bond mlag id 6
nv set interface bond6 description BCM-headnode-2
nv set interface bond7 bond member swp7
nv set interface bond7 bond mlag id 7
nv set interface bond7 description k8s-system-admin-01
nv set interface bond8 bond member swp8
nv set interface bond8 bond mlag id 8
nv set interface bond8 description k8s-system-admin-02
nv set interface bond9 bond member swp9
nv set interface bond9 bond mlag id 9
nv set interface bond9 description k8s-system-admin-03
nv set interface bond10 bond member swp10
nv set interface bond10 bond mlag id 10
nv set interface bond10 description k8s-system-user-01
nv set interface bond11 bond member swp11
nv set interface bond11 bond mlag id 11
nv set interface bond11 description k8s-system-user-02
nv set interface bond12 bond member swp12
nv set interface bond12 bond mlag id 12
nv set interface bond12 description k8s-system-user-03
nv set interface bond13 bond member swp13
nv set interface bond13 bond mlag id 13
nv set interface bond13 description slogin-01
nv set interface bond14 bond member swp14
nv set interface bond14 bond mlag id 14
nv set interface bond14 description slogin-02
nv set interface bond15 bond member swp15
nv set interface bond15 bond mlag id 15
nv set interface bond15 description dgx-b300-01
nv set interface bond15-24 bridge domain br_default access 103
nv set interface bond16 bond member swp16
nv set interface bond16 bond mlag id 16
nv set interface bond16 description dgx-b300-02
nv set interface bond17 bond member swp17
nv set interface bond17 bond mlag id 17
nv set interface bond17 description dgx-b300-03
nv set interface bond18 bond member swp18
nv set interface bond18 bond mlag id 18
nv set interface bond18 description dgx-b300-04
nv set interface bond19 bond member swp19
nv set interface bond19 bond mlag id 19
nv set interface bond19 description dgx-b300-05
nv set interface bond20 bond member swp20
nv set interface bond20 bond mlag id 20
nv set interface bond20 description dgx-b300-06
nv set interface bond21 bond member swp21
nv set interface bond21 bond mlag id 21
nv set interface bond21 description dgx-b300-07
nv set interface bond22 bond member swp22
nv set interface bond22 bond mlag id 22
nv set interface bond22 description dgx-b300-08
nv set interface bond23 bond member swp23
nv set interface bond23 bond mlag id 23
nv set interface bond23 description dgx-b300-09
nv set interface bond24 bond member swp24
nv set interface bond24 bond mlag id 24
nv set interface bond24 description dgx-b300-10
# Management interface
nv set interface eth0 ipv4 dhcp-client set-hostname enabled
nv set interface eth0 ipv4 dhcp-client state enabled
nv set interface eth0 type eth
nv set interface eth0 vrf mgmt
# Loopback interface for BGP and MLAG backup routing
nv set interface lo ipv4 address 10.160.254.23/32
nv set interface lo type loopback
# MLAG peerlink
nv set interface peerlink bond member swp63-64
nv set interface peerlink type peerlink
nv set interface peerlink,swp63-64 description MLAG-peerlink-to-SN5600-LEAF-1
nv set interface peerlink.4094 base-interface peerlink
nv set interface peerlink.4094 type sub
nv set interface peerlink.4094 vlan 4094
# Customer BGP uplinks
nv set interface swp61 description BGP-uplink-secondary-to-SN5600-Customer-Gw
nv set interface swp61-64 type swp
nv set interface swp62 description BGP-uplink-to-SN5600-Customer-Gw
# SVI and VRR gateway configuration for in-band networks
nv set interface vlan102-103 ipv4 vrr mac-address 00:1c:73:aa:bb:04
nv set interface vlan102-103 ipv4 vrr state enabled
nv set interface vlan102-103 ipv4 vrr vrr-state up
nv set interface vlan102-103 type svi
nv set interface vlan102 description Internalnet/Control-Nodes
nv set interface vlan102 ipv4 address 10.150.124.3/24
nv set interface vlan102 ipv4 vrr address 10.150.124.1/24
nv set interface vlan102 vlan 102
nv set interface vlan103 description DGXnet1
nv set interface vlan103 ipv4 address 10.150.125.3/24
nv set interface vlan103 ipv4 vrr address 10.150.125.1/24
nv set interface vlan103 vlan 103
# MLAG global settings
nv set mlag backup 10.160.254.22
nv set mlag mac-address 44:38:39:FF:0A:00
nv set mlag peer-ip linklocal
nv set mlag priority 4096
nv set mlag state enabled
# BGP global settings
nv set router bgp autonomous-system 4200120327
nv set router bgp router-id 10.160.254.23
nv set router bgp state enabled
# BGP route policy for default-only inbound and local-routes-only outbound advertisements
nv set router policy prefix-list DEF_ONLY rule 10 action permit
nv set router policy prefix-list DEF_ONLY rule 10 match 0.0.0.0/0
nv set router policy prefix-list DEF_ONLY type ipv4
nv set router policy prefix-list LOCAL_ROUTES rule 10 action permit
nv set router policy prefix-list LOCAL_ROUTES rule 10 match 10.150.124.0/24
nv set router policy prefix-list LOCAL_ROUTES rule 20 action permit
nv set router policy prefix-list LOCAL_ROUTES rule 20 match 10.150.125.0/24
nv set router policy prefix-list LOCAL_ROUTES rule 30 action permit
nv set router policy prefix-list LOCAL_ROUTES rule 30 match 10.160.254.23/32
nv set router policy prefix-list LOCAL_ROUTES type ipv4
nv set router policy prefix-list NO_DEF rule 10 action permit
nv set router policy prefix-list NO_DEF rule 10 match 0.0.0.0/0
nv set router policy prefix-list NO_DEF type ipv4
nv set router policy route-map DEF_ONLY rule 5 action permit
nv set router policy route-map DEF_ONLY rule 5 match ip-prefix-list DEF_ONLY
nv set router policy route-map DEF_ONLY rule 5 match type ipv4
nv set router policy route-map DEF_ONLY rule 100 action deny
nv set router policy route-map LOCAL-RTS-ONLY rule 5 action deny
nv set router policy route-map LOCAL-RTS-ONLY rule 5 match ip-prefix-list NO_DEF
nv set router policy route-map LOCAL-RTS-ONLY rule 5 match type ipv4
nv set router policy route-map LOCAL-RTS-ONLY rule 10 action permit
nv set router policy route-map LOCAL-RTS-ONLY rule 10 match ip-prefix-list LOCAL_ROUTES
nv set router policy route-map LOCAL-RTS-ONLY rule 10 match type ipv4
nv set router policy route-map LOCAL-RTS-ONLY rule 100 action deny
# VRR and DHCP relay services
nv set router vrr state enabled
nv set service dhcp-relay default downstream-interface vlan102-103 server-group-name dhcp-servers
nv set service dhcp-relay default server-group dhcp-servers server 10.150.124.11
nv set service dhcp-relay default server-group dhcp-servers server 10.150.124.12
nv set service dhcp-relay default server-group dhcp-servers upstream-interface vlan102
# System users, roles, services, and management settings
nv set system aaa class nvapply action allow
nv set system aaa class nvapply command-path / permission all
nv set system aaa class nvshow action allow
nv set system aaa class nvshow command-path / permission ro
nv set system aaa class sudo action allow
nv set system aaa class sudo command-path / permission all
nv set system aaa role nvue-admin class nvapply
nv set system aaa role nvue-monitor class nvshow
nv set system aaa role system-admin class nvapply
nv set system aaa role system-admin class sudo
nv set system aaa user cumulus full-name cumulus,,,
nv set system aaa user cumulus hashed-password '*'
nv set system aaa user cumulus role system-admin
nv set system api state enabled
nv set system config auto-save state enabled
nv set system control-plane acl acl-default-dos inbound
nv set system control-plane acl acl-default-whitelist inbound
nv set system date-time timezone America/Los_Angeles
nv set system dns server 1.1.1.1
nv set system dns server 8.8.8.8
nv set system docker state enabled
nv set system docker vrf mgmt
nv set system hostname SN5600-LEAF-2
nv set system ntp server 0.cumulusnetworks.pool.ntp.org
nv set system ntp server 1.cumulusnetworks.pool.ntp.org
nv set system ntp server 2.cumulusnetworks.pool.ntp.org
nv set system ntp server 3.cumulusnetworks.pool.ntp.org
nv set system ntp server time.nist.gov
nv set system ntp state enabled
nv set system ntp vrf mgmt
nv set system ssh-server permit-root-login enabled
nv set system ssh-server state enabled
nv set system ssh-server vrf mgmt
nv set system wjh channel forwarding trigger l2
nv set system wjh channel forwarding trigger l3
nv set system wjh channel forwarding trigger tunnel
nv set system wjh state enabled
# VRF BGP address family and neighbor settings
nv set vrf default router bgp address-family ipv4-unicast redistribute connected state enabled
nv set vrf default router bgp address-family ipv4-unicast state enabled
nv set vrf default router bgp neighbor peerlink.4094 remote-as internal
nv set vrf default router bgp neighbor peerlink.4094 timers connection-retry 10
nv set vrf default router bgp neighbor peerlink.4094 timers hold 10
nv set vrf default router bgp neighbor peerlink.4094 timers keepalive 3
nv set vrf default router bgp neighbor peerlink.4094 timers route-advertisement auto
nv set vrf default router bgp neighbor peerlink.4094 type unnumbered
nv set vrf default router bgp neighbor swp61 address-family ipv4-unicast policy inbound route-map DEF_ONLY
nv set vrf default router bgp neighbor swp61 address-family ipv4-unicast policy outbound route-map LOCAL-RTS-ONLY
nv set vrf default router bgp neighbor swp61 remote-as external
nv set vrf default router bgp neighbor swp61 timers connection-retry 10
nv set vrf default router bgp neighbor swp61 timers hold 10
nv set vrf default router bgp neighbor swp61 timers keepalive 3
nv set vrf default router bgp neighbor swp61 timers route-advertisement auto
nv set vrf default router bgp neighbor swp61 type unnumbered
nv set vrf default router bgp neighbor swp62 address-family ipv4-unicast policy inbound route-map DEF_ONLY
nv set vrf default router bgp neighbor swp62 address-family ipv4-unicast policy outbound route-map LOCAL-RTS-ONLY
nv set vrf default router bgp neighbor swp62 remote-as external
nv set vrf default router bgp neighbor swp62 timers connection-retry 10
nv set vrf default router bgp neighbor swp62 timers hold 10
nv set vrf default router bgp neighbor swp62 timers keepalive 3
nv set vrf default router bgp neighbor swp62 timers route-advertisement auto
nv set vrf default router bgp neighbor swp62 type unnumbered
nv set vrf default router bgp state enabled
SN2201 Switch Configuration-IPMI#
Click to expand SN2201 IPMI switch configuration
# Bridge domain and IPMI/BMC/OOB access VLAN
nv set bridge domain br_default vlan 101
# Management interface
nv set interface eth0 ipv4 dhcp-client set-hostname enabled
nv set interface eth0 ipv4 dhcp-client state enabled
nv set interface eth0 type eth
nv set interface eth0 vrf mgmt
# BMC access ports and IPMI/BMC/OOB SVI gateway
nv set interface swp1-47 bridge domain br_default access 101
nv set interface swp1-48 type swp
nv set interface vlan101 description IPMI/BMC/OOB
nv set interface vlan101 ipv4 address 10.150.123.1/24
nv set interface vlan101 type svi
nv set interface vlan101 vlan 101
# BGP global settings
nv set router bgp autonomous-system 65001
nv set router bgp state enabled
nv set router bgp router-id 10.0.0.1
# BGP route policy for default-only inbound and IPMI-route-only outbound advertisements
nv set router policy prefix-list DEF_ONLY rule 10 action permit
nv set router policy prefix-list DEF_ONLY rule 10 match 0.0.0.0/0
nv set router policy prefix-list DEF_ONLY type ipv4
nv set router policy prefix-list IPMI_ROUTES rule 10 action permit
nv set router policy prefix-list IPMI_ROUTES rule 10 match 10.150.123.0/24
nv set router policy prefix-list IPMI_ROUTES type ipv4
nv set router policy prefix-list NO_DEF rule 10 action permit
nv set router policy prefix-list NO_DEF rule 10 match 0.0.0.0/0
nv set router policy prefix-list NO_DEF type ipv4
nv set router policy route-map DEF_ONLY rule 5 action permit
nv set router policy route-map DEF_ONLY rule 5 match ip-prefix-list DEF_ONLY
nv set router policy route-map DEF_ONLY rule 5 match type ipv4
nv set router policy route-map DEF_ONLY rule 100 action deny
nv set router policy route-map IPMI-RTS-ONLY rule 5 action deny
nv set router policy route-map IPMI-RTS-ONLY rule 5 match ip-prefix-list NO_DEF
nv set router policy route-map IPMI-RTS-ONLY rule 5 match type ipv4
nv set router policy route-map IPMI-RTS-ONLY rule 10 action permit
nv set router policy route-map IPMI-RTS-ONLY rule 10 match ip-prefix-list IPMI_ROUTES
nv set router policy route-map IPMI-RTS-ONLY rule 10 match type ipv4
nv set router policy route-map IPMI-RTS-ONLY rule 100 action deny
# DNS settings
nv set service dns default search simulation
nv set service dns default server 1.1.1.1
nv set service dns default server 8.8.8.8
nv set service dns default server 192.168.200.1
# NTP settings
nv set service ntp mgmt server 0.cumulusnetworks.pool.ntp.org
nv set service ntp mgmt server 1.cumulusnetworks.pool.ntp.org
nv set service ntp mgmt server 2.cumulusnetworks.pool.ntp.org
nv set service ntp mgmt server 3.cumulusnetworks.pool.ntp.org
nv set service ntp mgmt server time.nist.gov
# System users, roles, services, and management settings
nv set system aaa class nvapply action allow
nv set system aaa class nvapply command-path / permission all
nv set system aaa class nvshow action allow
nv set system aaa class nvshow command-path / permission ro
nv set system aaa class sudo action allow
nv set system aaa class sudo command-path / permission all
nv set system aaa role nvue-admin class nvapply
nv set system aaa role nvue-monitor class nvshow
nv set system aaa role system-admin class nvapply
nv set system aaa role system-admin class sudo
nv set system aaa user cumulus full-name cumulus,,,
nv set system aaa user cumulus hashed-password '*'
nv set system aaa user cumulus role system-admin
nv set system api state enabled
nv set system config auto-save state enabled
nv set system control-plane acl acl-default-dos inbound
nv set system control-plane acl acl-default-whitelist inbound
nv set system hostname oob-mgmt-switch-leaf-1
nv set system reboot mode cold
nv set system ssh-server state enabled
nv set system timezone America/Los_Angeles
nv set system wjh channel forwarding trigger l2
nv set system wjh channel forwarding trigger l3
nv set system wjh channel forwarding trigger tunnel
nv set system wjh state enabled
# VRF BGP neighbors for OOB uplinks
nv set vrf default router bgp address-family ipv4-unicast state enabled
nv set vrf default router bgp address-family ipv4-unicast redistribute connected state enabled
nv set vrf default router bgp state enabled
nv set vrf default router bgp neighbor swp48 peer-group oob-uplink
nv set vrf default router bgp neighbor swp48 type unnumbered
nv set vrf default router bgp peer-group oob-uplink address-family ipv4-unicast policy inbound route-map DEF_ONLY
nv set vrf default router bgp peer-group oob-uplink address-family ipv4-unicast policy outbound route-map IPMI-RTS-ONLY
nv set vrf default router bgp peer-group oob-uplink remote-as external
QM3400-1 Full Configuration#
Click to expand QM3400-1 full configuration
nv set interface eth0 ip address 10.150.134.41/25
nv set interface eth0 ip gateway 10.150.134.1
nv set interface eth0-1 acl ACL_MGMT_INBOUND_CP_DEFAULT inbound control-plane
nv set interface eth0-1 acl ACL_MGMT_INBOUND_CP_DEFAULT_IPV6 inbound control-plane
nv set interface eth0-1 acl ACL_MGMT_INBOUND_DEFAULT inbound
nv set interface eth0-1 acl ACL_MGMT_INBOUND_DEFAULT_IPV6 inbound
nv set interface eth0-1 acl ACL_MGMT_OUTBOUND_CP_DEFAULT outbound control-plane
nv set interface eth0-1 acl ACL_MGMT_OUTBOUND_CP_DEFAULT_IPV6 outbound control-plane
nv set interface eth0-1 type eth
nv set interface lo acl ACL_LOOPBACK_INBOUND_CP_DEFAULT inbound control-plane
nv set interface lo acl ACL_LOOPBACK_INBOUND_CP_DEFAULT_IPV6 inbound control-plane
nv set interface lo type loopback
nv set system aaa user admin password '*'
nv set system dns server 103.247.36.36
nv set system dns server 103.247.37.37
nv set system hostname QM3400-1
nv set system ntp server time.google.com
nv set system ntp server time.nist.gov
QM3400-2 Full Configuration#
Click to expand QM3400-2 full configuration
nv set interface eth0 ip address 10.150.134.42/25
nv set interface eth0 ip gateway 10.150.134.1
nv set interface eth0-1 acl ACL_MGMT_INBOUND_CP_DEFAULT inbound control-plane
nv set interface eth0-1 acl ACL_MGMT_INBOUND_CP_DEFAULT_IPV6 inbound control-plane
nv set interface eth0-1 acl ACL_MGMT_INBOUND_DEFAULT inbound
nv set interface eth0-1 acl ACL_MGMT_INBOUND_DEFAULT_IPV6 inbound
nv set interface eth0-1 acl ACL_MGMT_OUTBOUND_CP_DEFAULT outbound control-plane
nv set interface eth0-1 acl ACL_MGMT_OUTBOUND_CP_DEFAULT_IPV6 outbound control-plane
nv set interface eth0-1 type eth
nv set interface lo acl ACL_LOOPBACK_INBOUND_CP_DEFAULT inbound control-plane
nv set interface lo acl ACL_LOOPBACK_INBOUND_CP_DEFAULT_IPV6 inbound control-plane
nv set interface lo type loopback
nv set system aaa user admin password '*'
nv set system dns server 103.247.36.36
nv set system dns server 103.247.37.37
nv set system hostname QM3400-2
nv set system ntp server time.google.com
nv set system ntp server time.nist.gov