Skip to navigation

NVLink SDN Partitioning

View as Markdown

NVLink SDN (Software Defined Networking) partitioning allows administrators to split a rack’s NVLink domain into smaller GPU partitions. This is useful for multi-tenancy, workload isolation, or maintenance scenarios where not all GPUs in a rack should belong to a single NVLink fabric domain.

NVLink SDN definitions are managed via the nvlinksdn mode in cmsh, and applied to racks using the applynvsdnpartitions command in rack mode:

cmsh
|
|-- nvlinksdn -- partitions
|
|-- rack ------- applynvsdnpartitions
...

Section 9.1 provides an overview of NVLink SDN definitions and the pre-defined partition templates.

Section 9.2 describes how to view NVLink SDN definitions and their properties.

Section 9.3 covers the partitions submode for viewing and managing individual partitions.

Section 9.4 describes the partition properties in detail.

Section 9.5 explains how to create custom NVLink SDN definitions.

Section 9.6 covers applying NVLink SDN partition configurations to racks.

Section 9.7 describes the use of SDN partition configurations to specify Slurm topology blocks.

BCM ships with a set of pre-defined NVLink SDN partition templates. These templates cover common partitioning schemes for an NVL72 rack with 18 compute trays (72 GPUs). The Default Partition assigns all GPUs to a single partition, while the other templates split the 18 nodes into various groupings such as 2x9 Nodes (two partitions of 9 nodes each) or 3x6 Nodes (three partitions of 6 nodes each).

The available NVLink SDN definitions can be listed using the list command in nvlinksdn mode:

Example

[basecm11->nvlinksdn]% list
Name (key)         Partitions                 Description
------------------ -------------------------- -----------
1x2+1x16 Nodes     1, 2
1x2+1x4+2x6 Nodes  1, 2, 3, 4
1x6+1x12 Nodes     1, 2
1x8+1x10 Nodes     1, 2
2x3+1x12 Nodes     1, 2, 3
2x4+1x10 Nodes     1, 2, 3
2x9 Nodes          1, 2
3x4+1x6 Nodes      1, 2, 3, 4
3x6 Nodes          1, 2, 3
4x3+1x6 Nodes      1, 2, 3, 4, 5
6x3 Nodes          1, 2, 3, 4, 5, 6
9x2 Nodes          1, 2, 3, 4, 5, 6, 7, 8, 9
Default Partition  32766

The Partitions column shows the partition indexes defined within each NVLink SDN definition. The Default Partition uses index 32766, which is the reserved index for a partition of kind All that includes all GPUs.

The properties of an NVLink SDN definition can be viewed with the show command after selecting it with use:

Example

[basecm11->nvlinksdn]% use "Default Partition"
[basecm11->nvlinksdn[Default Partition]]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Name                             Default Partition
Revision
Description
Partitions                       32766

Or, without dropping into the object, tab-completion with show from nvlinksdn mode also works:

Example

[basecm11->nvlinksdn]% show <TAB><TAB>
1x2+1x16 Nodes     1x8+1x10 Nodes     2x9 Nodes          4x3+1x6 Nodes      Default Partition
1x2+1x4+2x6 Nodes  2x3+1x12 Nodes     3x4+1x6 Nodes      6x3 Nodes
1x6+1x12 Nodes     2x4+1x10 Nodes     3x6 Nodes          9x2 Nodes
[basecm11->nvlinksdn]% show Default Partition
Parameter                        Value
-------------------------------- ------------------------------------------------
Name                             Default Partition
Revision
Description
Partitions                       32766

A multi-partition definition shows the partition indices as a comma-separated list:

Example

[basecm11->nvlinksdn]% use "2x9 Nodes"
[basecm11->nvlinksdn[2x9 Nodes]]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Name                             2x9 Nodes
Revision
Description
Partitions                       1, 2

9.3 Partitions Submode

Each NVLink SDN definition contains one or more partitions. The partitions are managed through the partitions submode, which is entered using the partitions command on a selected NVLink SDN definition:

Example

[basecm11->nvlinksdn[Default Partition]]% partitions
[basecm11->nvlinksdn[Default Partition]->partitions]% list
Name               Index  Kind        Resiliency mode  Mcast limit  Info
------------------ ------ ----------- ---------------- ------------ ----------------
Default Partition  32766  All         Full bandwidth   1024         all

The Default Partition has a single partition with kind All, which means that all GPUs in the rack are included. The Info column shows a summary based on the partition kind.

A multi-partition definition such as 2x9 Nodes has two partitions, each covering 9 nodes:

Example

[basecm11->nvlinksdn[2x9 Nodes]->partitions]% list
Name         Index  Kind        Resiliency mode  Mcast limit  Info
------------ ------ ----------- ---------------- ------------ ----------------
partition-1  1      Node count  Full bandwidth   512          nodes: 9
partition-2  2      Node count  Full bandwidth   512          nodes: 9

A more complex definition such as 1x2+1x4+2x6 Nodes has four partitions, with multicast limits allocated in proportion to the number of nodes:

Example

[basecm11->nvlinksdn[1x2+1x4+2x6 Nodes]->partitions]% list
Name         Index  Kind        Resiliency mode  Mcast limit  Info
------------ ------ ----------- ---------------- ------------ ----------------
partition-1  1      Node count  Full bandwidth   113              nodes: 2
partition-2  2      Node count  Full bandwidth   228              nodes: 4
partition-3  3      Node count  Full bandwidth   341              nodes: 6
partition-4  4      Node count  Full bandwidth   342              nodes: 6

The full details of a partition can be viewed by selecting it with use and running show:

Example

[basecm11->nvlinksdn[2x9 Nodes]->partitions]% use partition-1
[basecm11->nvlinksdn[2x9 Nodes]->partitions[partition-1]]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Info                             nodes: 9
Name                             partition-1
Revision
Index                            1
Kind                             Node count
Count                            9
Mcast limit                      512
Resiliency mode                  Full bandwidth

or more directly, without dropping into the object, by running show with tab-completion from partitions mode:

Example

[basecm11->nvlinksdn[2x9 Nodes]->partitions]% show partition-<TAB><TAB>
partition-1  partition-2
[basecm11->nvlinksdn[2x9 Nodes]->partitions]% show partition-1 <TAB><TAB>
Parameter                        Value
-------------------------------- ------------------------------------------------
Info                             nodes: 9
Name                             partition-1
Revision
Index                            1
Kind                             Node count
Count                            9
Mcast limit                      512
Resiliency mode                  Full bandwidth

9.4 Partition Properties

Each partition has the following properties:

  • Kind: Determines how GPUs are assigned to the partition. The available kinds are:

    • All: Includes all GPUs in the partition. Only one partition with this kind may exist in an NVLink SDN definition, and it cannot coexist with other partitions.

    • Node count: Specifies the partition by number of compute nodes.

    • GPU count: Specifies the partition by number of GPUs.

    • Locations: Specifies the partition by explicit GPU locations in the format .<slot-id>.<host-id>.<gpu-id>.

    • GUID: Specifies the partition by GPU GUIDs. The GUIDs can be retrieved using the nvplatforminfo command in device mode (section 7.3.4).

  • Index: The partition index, ranging from 1 to 32766. Index 32766 is reserved value, and is for partitions with kind All.

  • Count: The number of nodes or GPUs in the partition. This property is only displayed when the kind is Node count or GPU count.

  • Locations: A list of GPU locations. This property is only displayed when the kind is Locations.

  • GUIDs: A list of GPU GUIDs. This property is only displayed when the kind is GUID.

  • Mcast limit: The multicast limit allocated to the partition. The sum of multicast limits across all partitions in an NVLink SDN definition must not exceed 1024.

  • Resiliency mode: Determines how the partition handles NVLink trunk link failures. The available modes are:

    • Full bandwidth: On trunk link failure, attempts automatic recovery. If no spare trunk links are available, excludes GPUs from the fabric to ensure remaining GPUs maintain full, non-degraded bandwidth.

    • Adaptive bandwidth: If a trunk link fails and no spares are available, keeps all GPUs active but allows them to operate at a lower, degraded bandwidth.

    • User action required: On a link failure without spares, enters an unhealthy state and halts automatic rerouting, requiring manual intervention to recover.

The help set command in the partitions submode displays a useful help text with a summary of the preceding information.

Example

[basecm11->nvlinksdn[2x9 Nodes]->partitions]% help set
...
Parameters:
          name ................ Name
          index ............... Index
          kind ................ Partition kind
                                      - ALL: Include all GPUs in the partition
                                      - NODE_COUNT: Specify partition by number of nodes
                                      - GPU_COUNT: Specify partition by number of GPUs
...

Custom NVLink SDN definitions can be created using the add command in nvlinksdn mode. The following example creates a definition with two partitions of 9 nodes each, named part-a and part-b:

Example

[basecm11->nvlinksdn]% add "my-custom"
[basecm11->nvlinksdn*[my-custom*]]% partitions
[basecm11->nvlinksdn*[my-custom*]->partitions]% add part-a 1 9
[basecm11->nvlinksdn*[my-custom*]->partitions*[part-a*]]% exit
[basecm11->nvlinksdn*[my-custom*]->partitions]% add part-b 2 9
[basecm11->nvlinksdn*[my-custom*]->partitions*[part-b*]]% list
Name (key)   Index  Kind        Info
------------ ------ ----------- ----------------
part-a       1      Node count  nodes: 9
part-b       2      Node count  nodes: 9
[basecm11->nvlinksdn*[my-custom*]->partitions*[part-b*]]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Info                             nodes: 9
Name                             part-b
Revision
Index                            2
Kind                             Node count
Count                            9
Mcast limit                      0
Resiliency mode                  Full bandwidth
[basecm11->nvlinksdn*[my-custom*]->partitions*[part-b*]]% exit; exit
[basecm11->nvlinksdn*[my-custom*]]% commit

The add command in the partitions submode supports these syntax formats:

  • add <partition name>: Creates a partition with an automatically assigned index. The value of the parameter Kind is inherited from existing partitions, or it defaults to the value Node count.

  • add <partition name> <index>: Creates a partition with the specified value of <index>.

  • add <partition name> <index> <node-count>: Creates a partition with the specified value of <index>. The specified value of sets the value of the parameter Count, which is the number of nodes in the partition.

  • add <partition name> <index> gpus <gpu-count>: Creates a partition with the specified value of <index>. The use of the gpu flag means that

    • the value of the parameter Kind is set to GPU count

    • the specified value of sets the value of the parameter Count, which is the number of gpus in the partition.

9.5.1 Changing Partition Kind

A newly added partition defaults to a value of Node count for the parameter Kind. The value for Kind can be changed with the set command, and the displayed properties adapt accordingly. For example, switching to the GUID kind causes the Count property to be replaced by GUIDs, and the value displayed for the Info parameter changes accordingly:

Example

[basecm11->nvlinksdn*[test*]]% partitions
[basecm11->nvlinksdn*[test*]->partitions]% add partition-1
[basecm11->nvlinksdn*[test*]->partitions*[partition-1*]]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Info                             nodes: 1
Name                             partition-1
Revision
Index                            1
Kind                             Node count
Count                            1
Mcast limit                      0
Resiliency mode                  Full bandwidth
[basecm11->nvlinksdn*[test*]->partitions*[partition-1*]]% set kind <TAB><TAB>
All   GPU count   GUID   Locations    Node count            <---- values explained in section~9.4
[basecm11->nvlinksdn*[test*]->partitions*[partition-1*]]% set kind GUID
[basecm11->nvlinksdn*[test*]->partitions*[partition-1*]]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Info                             guids:
Name                             partition-1
Revision
Index                            1
Kind                             GUID
GUIDs
Mcast limit                      0
Resiliency mode                  Full bandwidth

The nvplatforminfo command (section 7.3.4) in device mode retrieves the IB GUID for each GPU. These are the GUID values that can be set for the GUIDs parameter when the Kind parameter is set to GUID.

9.5.2 Validation Rules

When committing an NVLink SDN definition, the following validation rules are enforced:

  • Partition names must be unique within the definition.

  • Partition index values must be unique within the definition.

  • A partition with kind set to All cannot coexist with other partitions.

  • All partitions must have the same kind. Mixed kinds of partitions in an SDN succeed, but produce a warning.

  • For Node count partitions: the sum of node counts must equal 18.

  • For GPU count partitions: the sum of GPU counts must equal 36, 72, or 144.

  • For Locations partitions: a location must not appear in multiple partitions.

  • The sum of multicast limits across all partitions must not exceed 1024.

Once an NVLink SDN definition has been configured, it can be applied to racks using the applynvsdnpartitions command in rack mode. This command sends the partition configuration to the NVLink switches in the specified racks.

Example

[basecm11->rack]% help applynvsdnpartitions
Name:
          applynvsdnpartitions - Apply NVLink SDN partition configuration to racks
Usage:
          applynvsdnpartitions [OPTIONS] <nvlink_sdn_name>
Options:
          --location <location>
               Comma separated list of locations to filter on
          --building <building>
               Comma separated list of buildings to filter on
          --room <room>
               Comma separated list of rooms to filter on
          --row <row>
               Comma separated list of rows to filter on
Examples:
 applynvsdnpartitions "Default Partition"          Apply default partition to current or all
                                                   racks
 applynvsdnpartitions --building HQ "2x9 Nodes"    Apply partitions to all racks in HQ
 applynvsdnpartitions rack01 rack02 "3x6 Nodes"    Apply partitions to the specified racks

The arguments and options to applynvsdnpartitions have these features and behaviors:

  • Arguments:

    • The last argument is always the name of the NVLink SDN definition.

    • Any preceding arguments are rack names.

    • If no racks are specified and a rack is currently selected (with use), then the partition is applied to that rack.

    • If no rack is selected either, then the partition is applied to all racks.

  • Options: Racks can be filtered by physical location attributes using the --location, --building, --room, and --row options.

Each GPU operation takes about 1 second. With 72 GPUs per rack requiring both removal and re-addition, the command typically takes about 1 to 3 minutes per rack to complete.

The command output shows the result for each rack. The operation first removes all GPUs from their current partitions, then creates the new partitions, and then adds GPUs to them.

Example

[basecm11->rack]% applynvsdnpartitions a01 "2x9 Nodes"
*** a01 ***
+++ stdout +++
 - run: /usr/bin/nv action restore sdn partition 32766 uuid 13910308286599238321 no-reroute
Action executing ...
GPU uuid '13910308286599238321' removed from partition 32766
Routing table update deferred (no-reroute)
Action succeeded
 - run: /usr/bin/nv action restore sdn partition 32766 uuid 13874279489580274353 no-reroute
Action executing ...
GPU uuid '13874279489580274353' removed from partition 32766
Routing table update deferred (no-reroute)
Action succeeded
...  (remaining GPUs removed from partition 32766)
 - run: /usr/bin/nv action sdn add partition 1 name partition-1 resiliency-mode full-
        bandwidth mcast-limit 512
Action executing ...
Partition 1 is successfully created
Action succeeded
 - run: /usr/bin/nv action update sdn partition 1 uuid 1781225835824106024 no-reroute
Action executing ...
GPU uuid '1781225835824106024' added to partition 1
Routing table update deferred (no-reroute)
Action succeeded
...  (remaining GPUs added to partition 1)
 - run: /usr/bin/nv action update sdn partition 1 reroute
Action executing ...
Recalculating routing table for partition 1...
Routing table updated successfully
Action succeeded
 - run: /usr/bin/nv action sdn add partition 2 name partition-2 resiliency-mode full-
        bandwidth mcast-limit 512
Action executing ...
Partition 2 is successfully created
Action succeeded
 - run: /usr/bin/nv action update sdn partition 2 uuid 12697951332570188328 no-reroute
Action executing ...
GPU uuid '12697951332570188328' added to partition 2
Routing table update deferred (no-reroute)
Action succeeded
...  (remaining GPUs added to partition 2)
 - run: /usr/bin/nv action update sdn partition 2 reroute
Action executing ...
Recalculating routing table for partition 2...
Routing table updated successfully
Action succeeded

The current partition state of the cluster can be verified after applying using the nvsdnpartitioninfo command in device mode (section 7.3.6). The following example shows the result after applying 3x6 Nodes to rack a01 and 2x4+1x10 Nodes to rack a02:

Example

[basecm11]% device nvsdnpartitioninfo
Rack  Switches                  Clique  Name         Health   Node GPUs
----- ------------------------- ------- ------------ -------- -------------------
a01   a01-nvsw-01..a01-nvsw-09  1       partition-1  healthy  a01-c[01-06]:[0-3]
a01   a01-nvsw-01..a01-nvsw-09  2       partition-2  healthy  a01-c[07-12]:[0-3]
a01   a01-nvsw-01..a01-nvsw-09  3       partition-3  healthy  a01-c[13-18]:[0-3]
a02   a02-nvsw-01..a02-nvsw-09  1       partition-1  healthy  a02-c[01-04]:[0-3]
a02   a02-nvsw-01..a02-nvsw-09  2       partition-2  healthy  a02-c[05-08]:[0-3]
a02   a02-nvsw-01..a02-nvsw-09  3       partition-3  healthy  a02-c[09-18]:[0-3]

The output shows that each rack can have a different NVLink SDN definition applied. Rack a01 has three equal partitions of 6 nodes each, while rack a02 has two partitions of 4 nodes and one partition of 10 nodes.

9.7 Slurm Topology Integration

Topology blocks (here) in the Slurm workload manager can use NVLink SDN partitions. This can improve performance by ensuring that Slurm schedules jobs on nodes that belong to the same NVLink partition, so that GPU-to-GPU communication is kept within the high-bandwidth NVLink fabric.

To enable using NVLink SDN partitions as topology blocks, the Block entity is set to SDN in the Slurm topology block settings:

Example

[basecm11->wlm[slurm]]% topologysettings
[basecm11->wlm[slurm]->topologysettings]% blocksettings
[basecm11->wlm[slurm]->topologysettings->blocksettings]% show
Parameter                        Value
-------------------------------- ------------------------------------------------
Block sizes
Revision
Block entity                     Rack
Allowed racks
Allowed nodegroups
[basecm11->wlm[slurm]->topologysettings->blocksettings]% set blockentity SDN
[basecm11->wlm*[slurm*]->topologysettings*->blocksettings*]% commit

Setting Block entity to SDN, means that Slurm topology blocks are then derived from the NVLink SDN partitions that are currently applied on each rack. The generated topology can be viewed with the topology print command.

The following example shows the topology after applying 3x6 Nodes to rack a01, while rack a02 still has the Default Partition (all 18 nodes in a single block):

Example

[basecm11->wlm[slurm]]% topology print
# Topology configuration:
BlockName=a01-partition-1 Nodes=a01-c[01-06]
BlockName=a01-partition-2 Nodes=a01-c[07-12]
BlockName=a01-partition-3 Nodes=a01-c[13-18]
BlockName=a02-default_partition Nodes=a02-c[01-18]

The output shows that rack a01 is split into three blocks of 6 nodes matching the applied NVLink SDN partitions, while rack a02 has a single block covering all 18 nodes.