Skip to navigation

NVLink Technology

View as Markdown

This chapter introduces NVLink technology as used in NVIDIA GB200 and GB300 server platforms with BCM. NVLink is NVIDIA’s high-speed interconnect technology that enables direct GPU-to-GPU communication with significantly higher bandwidth and lower latency than traditional PCIe connections.

Section 7.1 provides an overview of NVLink technology and its role in GB200/GB300 systems.

Section 7.2 describes the NVLink architecture in GB200 and GB300 platforms.

Section 7.3 covers BCM commands for querying NVLink status and information.

NVLink is NVIDIA’s proprietary high-bandwidth, energy-efficient interconnect that enables fast communication between GPUs and between GPUs and CPUs. In GB200 and GB300 systems, NVLink provides the backbone for GPU-to-GPU communication within and across compute trays.

The GB200 and GB300 platforms use fifth-generation NVLink (NVLink 5), which provides:

  • 900 GB/s bidirectional bandwidth per link

  • Support for up to 576 GPUs in a single NVLink domain

  • NVLink Switch architecture for scaling beyond direct GPU connections

  • Integration with NVIDIA Grace CPUs for unified memory architecture

Compared to PCIe Gen5, NVLink 5 provides approximately 14x higher bandwidth. This makes NVLink essential for getting the best out of:

  • Large language model (LLM) training requiring frequent gradient synchronization

  • Multi-GPU inference workloads

  • High-performance computing applications with significant inter-GPU communication

  • Memory pooling across multiple GPUs

The NVIDIA GB200 and GB300 platforms implement a sophisticated NVLink topology that connects GPUs within compute trays and across the entire rack through NVLink switches.

7.2.1 Compute Tray Configuration

Each GB200/GB300 compute tray contains:

  • 2 NVIDIA Grace CPUs

  • 4 NVIDIA Blackwell GPUs (GB200/GB300)

  • NVLink connections between all GPUs within the tray

  • NVLink connections to external NVLink switch trays

The GPUs within a compute tray are fully connected via NVLink, allowing any GPU to communicate directly with any other GPU in the same tray at full NVLink bandwidth.

GB200 NVL72 rack configurations include 9 NVLink switch trays per rack. Each switch tray contains:

  • 2 NVSwitch chips per tray

  • 72 NVLink ports per switch (36 top, 36 bottom per NVSwitch chip)

  • 144 NVLink ports per switch tray total

The NVLink switches enable all-to-all GPU communication across the entire rack, creating a unified GPU fabric where any GPU can communicate with any other GPU at high bandwidth.

7.2.3 NVL72 Topology

The NVL72 reference configuration provides:

  • 72 GPUs per rack (18 compute trays × 4 GPUs)

  • Full bisection bandwidth between all GPUs

  • 130 TB/s total NVLink bandwidth per rack

  • Single NVLink domain spanning all 72 GPUs

This topology is optimized for large-scale AI training workloads, where model parallelism requires frequent communication between GPUs.

NVLink status and information can be queried using commands available in cmsh device mode.

The list command in device mode can filter for NVLink switches using the --field option. This displays all NVLink switches configured in the cluster:

Example

[basecm11->device]% list --field kind=nvlink
Type    Hostname           MAC                IP           Network   Status
------- ------------------ ------------------ ------------ --------- ----------
Switch  a07-p01-nvsw-01    E0:9D:73:05:70:6C  10.140.0.45  ipminet0  [   UP   ]
Switch  a07-p01-nvsw-02    B8:E9:24:B9:84:8C  10.140.0.46  ipminet0  [   UP   ]
Switch  a07-p01-nvsw-03    E0:9D:73:05:71:2A  10.140.0.47  ipminet0  [   UP   ]
...

The list command has many filtering options. Details can be seen by running the help command (help list).

NVLink switch status can be monitored through the standard status command of device mode. The switches report their operational state, link status, and any errors through the BCM monitoring system:

Example

[basecm11->device]% use a07-p01-nvsw-01
[basecm11->device[a07-p01-nvsw-01]]% status
a07-p01-nvsw-01 .............. [   UP   ]

The nvfabricinfo command is run in device mode, and displays information about NVLink fabric domains. It shows the domain identifier, the active switch, and all switches that are part of each NVLink domain:

Example

[basecm11->device]% nvfabricinfo
Domain  Active           Switches
------- ---------------- ---------------------------------
A07     a07-p01-nvsw-01  a07-p01-nvsw-01..a07-p01-nvsw-09

The output shows:

  • Domain: The NVLink domain identifier

  • Active: The currently active/primary switch for the domain

  • Switches: All NVLink switches that are part of the domain

The --raw option displays detailed JSON output showing the status of each switch in the fabric, including NMX controller and telemetry application status:

Example

[basecm11->device]% nvfabricinfo --raw
===== A07 =====
{
  "58519ce2-f695-4a4f-976b-99074aa13f65": {
    "apps": {
      "nmx-controller": {
        "addition-info": "CONTROL_PLANE_STATE_CONFIGURED",
        "app-id": "nmx-c-nvos",
        "app-ver": "4.21.30",
        "capabilities": "sm, gfm, fib, gw-api",
        "status": "ok"
      },
      "nmx-telemetry": {
        "app-id": "nmx-telemetry",
        "app-ver": "3.5.1",
        "capabilities": "nvl, gnmi, syslog, bmc",
        "status": "ok"
      }
    },
    "hostname": "a07-p01-nvsw-01",
    "leader": true,
    "success": true
  },
  "1f79262d-437e-4da4-8197-1522723e0861": {
    "apps": {},
    "hostname": "a07-p01-nvsw-02",
    "leader": false,
    "message": "nmx-controller not started",
    "success": true
  },
  ...
}

The raw output for each switch shows:

  • hostname: The switch hostname

  • leader: Whether this switch is the active leader for the domain

  • apps: Running NMX applications (nmx-controller, nmx-telemetry) with version and status

  • success: Whether the switch query was successful

  • message: Status message (for example, if NMX controller is not started)

The --start option can be used to switch the active/leader switch for a domain. This is useful for maintenance or failover scenarios:

Example

[basecm11->device]% nvfabricinfo --start a07-p01-nvsw-02

This command initiates a leader transition, making a07-p01-nvsw-02 the new active switch for the NVLink domain. The NMX controller starts on the new leader and stops on the previous leader.

The help text (help nvfabricinfo) has further details on the nvfabricinfo command, including options for filtering and formatting.

The nvplatforminfo command is run in device mode, and displays detailed NVLink platform information for each GPU in the cluster. It shows the physical location and identification of GPUs within the NVLink fabric, including slot, tray, and module assignments:

Example

[basecm11->device]% nvplatforminfo
Node                GPU  IB GUID             Slot  Tray  Host ID  Peer type  Module ID
------------------- ---- ------------------- ----- ----- -------- ---------- ---------
a07-p01-dgx-02-c01  0    0x02e56c0fe280b0f4  1     0     1        1          2
a07-p01-dgx-02-c01  1    0x7b948c1631919719  1     0     1        1          1
a07-p01-dgx-02-c01  2    0x7415054f46b585bb  1     0     1        1          4
a07-p01-dgx-02-c01  3    0x52ef972251ac3248  1     0     1        1          3
a07-p01-dgx-02-c02  0    0x16d5a2b400bf3441  2     1     1        1          2
a07-p01-dgx-02-c02  1    0x64ba6860fde1f9d4  2     1     1        1          1
a07-p01-dgx-02-c02  2    0x68e21af78702759a  2     1     1        1          4
a07-p01-dgx-02-c02  3    0x89114b471859c432  2     1     1        1          3
...

The preceding output shows:

  • GPU: The GPU index within the node (0-3 for GB200/GB300 compute trays)

  • IB GUID: The InfiniBand Global Unique Identifier for the GPU

  • Slot: The physical slot position in the rack

  • Tray: The compute tray number

  • Host ID: The host identifier within the NVLink domain

  • Peer type: The peer connection type

  • Module ID: The module identifier within the tray

The help text (help nvplatforminfo) has further details on the nvplatforminfo command, including options for filtering by node, category, rack, and so on:

The nvdomaininfo command is run within device mode, and displays the NVLink domain membership status for each GPU. It shows whether GPUs have successfully joined the NVLink cluster, and shows their assigned cluster UUID and clique ID:

Example

[basecm11->device]% nvdomaininfo
Node                GPU  Status       Cluster UUID                          Clique ID
------------------- ---- ------------ ------------------------------------- ---------
a07-p01-dgx-02-c01  0    Success      084aea8f-abb0-4209-a016-7ab97d55609c  32766
a07-p01-dgx-02-c01  1    Success      084aea8f-abb0-4209-a016-7ab97d55609c  32766
a07-p01-dgx-02-c01  2    Success      084aea8f-abb0-4209-a016-7ab97d55609c  32766
a07-p01-dgx-02-c01  3    Success      084aea8f-abb0-4209-a016-7ab97d55609c  32766
...
a07-p01-dgx-02-c16  0    Success      3185472a-9c82-4f72-98a7-dc1fddfd188b  32766
a07-p01-dgx-02-c16  1    Success      3185472a-9c82-4f72-98a7-dc1fddfd188b  32766
a07-p01-dgx-02-c16  2    In progress  00000000-0000-0000-0000-000000000000  0
a07-p01-dgx-02-c16  3    In progress  00000000-0000-0000-0000-000000000000  0
...

The output shows:

  • Status: The domain join status (Success, In progress, or error states)

  • Cluster UUID: The unique identifier of the NVLink cluster that the GPU belongs to

  • Clique ID: The clique identifier within the NVLink domain

GPUs with an In progress status are still in the process of joining the NVLink domain. They show a zeroed UUID until the join completes. Different cluster UUIDs indicate GPUs belonging to separate NVLink domains.

The help text (help nvdomaininfo) has further details on the nvdomaininfo command, including options for filtering by node, category, rack, and so on.

The nvsdnpartitioninfo command displays information about NVLink SDN (Software Defined Networking) partitions. It shows how GPUs are grouped into partitions across the NVLink fabric, including the associated switches and health status. The command is run in device mode:

Example

[basecm11->device]% nvsdnpartitioninfo
Rack  Switches          Clique  Name                Health   Node GPUs
----- ----------------- ------- ------------------- -------- ----------------------
A07   nvsw-01..nvsw-09  32766   Default Partition   healthy  dgx-02-c[01-17]:[0-3]

The output shows:

  • Rack: The rack identifier containing the partition

  • Switches: The NVLink switches that are part of the partition

  • Clique: The clique identifier for the partition

  • Name: The partition name (e.g., Default Partition)

  • Health: The health status of the partition (healthy, degraded, etc.)

  • Node GPUs: The nodes and GPU indices included in the partition

The help text (help nvsdnpartitioninfo) has further details on the nvsdnpartitioninfo command, including options for filtering by node, category, rack, and so on.

The nvlinkinfo command displays the status of individual NVLink links on each GPU. Each GPU in a GB200/GB300 system has 18 NVLink links connecting it to other GPUs and NVLink switches. The command is run in device mode (output truncated):

Example

[basecm11->device]% nvlinkinfo
Node                Type  Entity  Link  Status
------------------- ----- ------- ----- ------
a07-p01-dgx-02-c01  GPU   0       0     Up
a07-p01-dgx-02-c01  GPU   0       1     Up
a07-p01-dgx-02-c01  GPU   0       2     Up
...
a07-p01-dgx-02-c01  GPU   0       17    Up
a07-p01-dgx-02-c01  GPU   1       0     Up
...
a07-p01-dgx-02-c01  GPU   3       17    Up
a07-p01-dgx-02-c02  GPU   0       0     Up
...
a07-p01-dgx-02-c18  GPU   3       17    Up

The output shows:

  • Type: The device type (GPU)

  • Entity: The GPU index within the node (0-3)

  • Link: The NVLink link number (0-17 for 18 links per GPU)

  • Status: The link status (Up or Down)

Each compute tray has 4 GPUs with 18 NVLink links each, resulting in 72 link entries per node.

The help text (help nvlinkinfo) has further details on the nvlinkinfo command, including options for filtering by node, category, rack, and so on.

The NVIDIA Mission Control features allow NVLink measurables to be monitored (section 12.3).

For comprehensive NVLink monitoring, the NMX (NVIDIA Management Extensions) telemetry system provides detailed metrics on:

  • NVLink port utilization and bandwidth

  • error counts and link health

  • temperature and power consumption

  • traffic patterns and congestion

NMX telemetry integration with BCM is covered further in the BCM NMX documentation (section 2.1).