> This page is for version 11 (default). > For other versions, use one of these documentation indexes: > - 11 (default): https://docs.nvidia.com/base-command-manager/product-manuals/11/llms.txt > - 10: https://docs.nvidia.com/base-command-manager/product-manuals/10/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.nvidia.com/base-command-manager/product-manuals/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.nvidia.com/base-command-manager/product-manuals/_mcp/server. # NVLink SDN Partitioning NVLink SDN (Software Defined Networking) partitioning allows administrators to split a rack’s NVLink domain into smaller GPU partitions. This is useful for multi-tenancy, workload isolation, or maintenance scenarios where not all GPUs in a rack should belong to a single NVLink fabric domain. NVLink SDN definitions are managed via the `nvlinksdn` mode in `cmsh`, and applied to racks using the `applynvsdnpartitions` command in `rack` mode: ```bash cmsh | |-- nvlinksdn -- partitions | |-- rack ------- applynvsdnpartitions ... ``` Section [9.1](#91-nvlink-sdn-overview) provides an overview of NVLink SDN definitions and the pre-defined partition templates. Section [9.2](#92-viewing-nvlink-sdn-definitions) describes how to view NVLink SDN definitions and their properties. Section [9.3](#93-partitions-submode) covers the partitions submode for viewing and managing individual partitions. Section [9.4](#94-partition-properties) describes the partition properties in detail. Section [9.5](#95-creating-custom-nvlink-sdn-definitions) explains how to create custom NVLink SDN definitions. Section [9.6](#96-applying-nvlink-sdn-partitions) covers applying NVLink SDN partition configurations to racks. Section [9.7](#97-slurm-topology-integration) describes the use of SDN partition configurations to specify Slurm topology blocks. ## 9.1 NVLink SDN Overview BCM ships with a set of pre-defined NVLink SDN partition templates. These templates cover common partitioning schemes for an NVL72 rack with 18 compute trays (72 GPUs). The `Default Partition` assigns all GPUs to a single partition, while the other templates split the 18 nodes into various groupings such as `2x9 Nodes` (two partitions of 9 nodes each) or `3x6 Nodes` (three partitions of 6 nodes each). The available NVLink SDN definitions can be listed using the `list` command in `nvlinksdn` mode: **Example** ```bash [basecm11->nvlinksdn]% list Name (key) Partitions Description ------------------ -------------------------- ----------- 1x2+1x16 Nodes 1, 2 1x2+1x4+2x6 Nodes 1, 2, 3, 4 1x6+1x12 Nodes 1, 2 1x8+1x10 Nodes 1, 2 2x3+1x12 Nodes 1, 2, 3 2x4+1x10 Nodes 1, 2, 3 2x9 Nodes 1, 2 3x4+1x6 Nodes 1, 2, 3, 4 3x6 Nodes 1, 2, 3 4x3+1x6 Nodes 1, 2, 3, 4, 5 6x3 Nodes 1, 2, 3, 4, 5, 6 9x2 Nodes 1, 2, 3, 4, 5, 6, 7, 8, 9 Default Partition 32766 ``` The `Partitions` column shows the partition indexes defined within each NVLink SDN definition. The `Default Partition` uses index 32766, which is the reserved index for a partition of kind `All` that includes all GPUs. ## 9.2 Viewing NVLink SDN Definitions The properties of an NVLink SDN definition can be viewed with the `show` command after selecting it with `use`: **Example** ```bash [basecm11->nvlinksdn]% use "Default Partition" [basecm11->nvlinksdn[Default Partition]]% show Parameter Value -------------------------------- ------------------------------------------------ Name Default Partition Revision Description Partitions 32766 ``` Or, without dropping into the object, tab-completion with `show` from `nvlinksdn` mode also works: **Example** ```bash [basecm11->nvlinksdn]% show 1x2+1x16 Nodes 1x8+1x10 Nodes 2x9 Nodes 4x3+1x6 Nodes Default Partition 1x2+1x4+2x6 Nodes 2x3+1x12 Nodes 3x4+1x6 Nodes 6x3 Nodes 1x6+1x12 Nodes 2x4+1x10 Nodes 3x6 Nodes 9x2 Nodes [basecm11->nvlinksdn]% show Default Partition Parameter Value -------------------------------- ------------------------------------------------ Name Default Partition Revision Description Partitions 32766 ``` A multi-partition definition shows the partition indices as a comma-separated list: **Example** ```bash [basecm11->nvlinksdn]% use "2x9 Nodes" [basecm11->nvlinksdn[2x9 Nodes]]% show Parameter Value -------------------------------- ------------------------------------------------ Name 2x9 Nodes Revision Description Partitions 1, 2 ``` ## 9.3 Partitions Submode Each NVLink SDN definition contains one or more partitions. The partitions are managed through the `partitions` submode, which is entered using the `partitions` command on a selected NVLink SDN definition: **Example** ```bash [basecm11->nvlinksdn[Default Partition]]% partitions [basecm11->nvlinksdn[Default Partition]->partitions]% list Name Index Kind Resiliency mode Mcast limit Info ------------------ ------ ----------- ---------------- ------------ ---------------- Default Partition 32766 All Full bandwidth 1024 all ``` The `Default Partition` has a single partition with kind `All`, which means that all GPUs in the rack are included. The `Info` column shows a summary based on the partition kind. A multi-partition definition such as `2x9 Nodes` has two partitions, each covering 9 nodes: **Example** ```bash [basecm11->nvlinksdn[2x9 Nodes]->partitions]% list Name Index Kind Resiliency mode Mcast limit Info ------------ ------ ----------- ---------------- ------------ ---------------- partition-1 1 Node count Full bandwidth 512 nodes: 9 partition-2 2 Node count Full bandwidth 512 nodes: 9 ``` A more complex definition such as `1x2+1x4+2x6 Nodes` has four partitions, with multicast limits allocated in proportion to the number of nodes: **Example** ```bash [basecm11->nvlinksdn[1x2+1x4+2x6 Nodes]->partitions]% list Name Index Kind Resiliency mode Mcast limit Info ------------ ------ ----------- ---------------- ------------ ---------------- partition-1 1 Node count Full bandwidth 113 nodes: 2 partition-2 2 Node count Full bandwidth 228 nodes: 4 partition-3 3 Node count Full bandwidth 341 nodes: 6 partition-4 4 Node count Full bandwidth 342 nodes: 6 ``` The full details of a partition can be viewed by selecting it with `use` and running `show`: **Example** ```bash [basecm11->nvlinksdn[2x9 Nodes]->partitions]% use partition-1 [basecm11->nvlinksdn[2x9 Nodes]->partitions[partition-1]]% show Parameter Value -------------------------------- ------------------------------------------------ Info nodes: 9 Name partition-1 Revision Index 1 Kind Node count Count 9 Mcast limit 512 Resiliency mode Full bandwidth ``` or more directly, without dropping into the object, by running `show` with tab-completion from `partitions` mode: **Example** ```bash [basecm11->nvlinksdn[2x9 Nodes]->partitions]% show partition- partition-1 partition-2 [basecm11->nvlinksdn[2x9 Nodes]->partitions]% show partition-1 Parameter Value -------------------------------- ------------------------------------------------ Info nodes: 9 Name partition-1 Revision Index 1 Kind Node count Count 9 Mcast limit 512 Resiliency mode Full bandwidth ``` ## 9.4 Partition Properties Each partition has the following properties: * `Kind`: Determines how GPUs are assigned to the partition. The available kinds are: * `All`: Includes all GPUs in the partition. Only one partition with this kind may exist in an NVLink SDN definition, and it cannot coexist with other partitions. * `Node count`: Specifies the partition by number of compute nodes. * `GPU count`: Specifies the partition by number of GPUs. * `Locations`: Specifies the partition by explicit GPU locations in the format `...`. * `GUID`: Specifies the partition by GPU GUIDs. The GUIDs can be retrieved using the `nvplatforminfo` command in `device` mode (section [7.3.4](/nvidia-mission-control/nvlinkinfo#734-nvlink-platform-information-nvplatforminfo)). * `Index`: The partition index, ranging from 1 to 32766. Index 32766 is reserved value, and is for partitions with kind `All`. * `Count`: The number of nodes or GPUs in the partition. This property is only displayed when the kind is `Node count` or `GPU count`. * `Locations`: A list of GPU locations. This property is only displayed when the kind is `Locations`. * `GUIDs`: A list of GPU GUIDs. This property is only displayed when the kind is `GUID`. * `Mcast limit`: The multicast limit allocated to the partition. The sum of multicast limits across all partitions in an NVLink SDN definition must not exceed 1024. * `Resiliency mode`: Determines how the partition handles NVLink trunk link failures. The available modes are: * `Full bandwidth`: On trunk link failure, attempts automatic recovery. If no spare trunk links are available, excludes GPUs from the fabric to ensure remaining GPUs maintain full, non-degraded bandwidth. * `Adaptive bandwidth`: If a trunk link fails and no spares are available, keeps all GPUs active but allows them to operate at a lower, degraded bandwidth. * `User action required`: On a link failure without spares, enters an unhealthy state and halts automatic rerouting, requiring manual intervention to recover. The `help set` command in the `partitions` submode displays a useful help text with a summary of the preceding information. **Example** ```bash [basecm11->nvlinksdn[2x9 Nodes]->partitions]% help set ... Parameters: name ................ Name index ............... Index kind ................ Partition kind - ALL: Include all GPUs in the partition - NODE_COUNT: Specify partition by number of nodes - GPU_COUNT: Specify partition by number of GPUs ... ``` ## 9.5 Creating Custom NVLink SDN Definitions Custom NVLink SDN definitions can be created using the `add` command in `nvlinksdn` mode. The following example creates a definition with two partitions of 9 nodes each, named `part-a` and `part-b`: **Example** ```bash [basecm11->nvlinksdn]% add "my-custom" [basecm11->nvlinksdn*[my-custom*]]% partitions [basecm11->nvlinksdn*[my-custom*]->partitions]% add part-a 1 9 [basecm11->nvlinksdn*[my-custom*]->partitions*[part-a*]]% exit [basecm11->nvlinksdn*[my-custom*]->partitions]% add part-b 2 9 [basecm11->nvlinksdn*[my-custom*]->partitions*[part-b*]]% list Name (key) Index Kind Info ------------ ------ ----------- ---------------- part-a 1 Node count nodes: 9 part-b 2 Node count nodes: 9 [basecm11->nvlinksdn*[my-custom*]->partitions*[part-b*]]% show Parameter Value -------------------------------- ------------------------------------------------ Info nodes: 9 Name part-b Revision Index 2 Kind Node count Count 9 Mcast limit 0 Resiliency mode Full bandwidth [basecm11->nvlinksdn*[my-custom*]->partitions*[part-b*]]% exit; exit [basecm11->nvlinksdn*[my-custom*]]% commit ``` The `add` command in the partitions submode supports these syntax formats: * `add `: Creates a partition with an automatically assigned index. The value of the parameter `Kind` is inherited from existing partitions, or it defaults to the value `Node count`. * `add `: Creates a partition with the specified value of ``. * `add `: Creates a partition with the specified value of ``. The specified value of sets the value of the parameter `Count`, which is the number of nodes in the partition. * `add gpus `: Creates a partition with the specified value of ``. The use of the `gpu` flag means that * the value of the parameter `Kind` is set to `GPU count` * the specified value of sets the value of the parameter `Count`, which is the number of gpus in the partition. ### 9.5.1 Changing Partition Kind A newly added partition defaults to a value of `Node count` for the parameter `Kind`. The value for `Kind` can be changed with the `set` command, and the displayed properties adapt accordingly. For example, switching to the `GUID` kind causes the `Count` property to be replaced by `GUIDs`, and the value displayed for the `Info` parameter changes accordingly: **Example** ```bash [basecm11->nvlinksdn*[test*]]% partitions [basecm11->nvlinksdn*[test*]->partitions]% add partition-1 [basecm11->nvlinksdn*[test*]->partitions*[partition-1*]]% show Parameter Value -------------------------------- ------------------------------------------------ Info nodes: 1 Name partition-1 Revision Index 1 Kind Node count Count 1 Mcast limit 0 Resiliency mode Full bandwidth [basecm11->nvlinksdn*[test*]->partitions*[partition-1*]]% set kind All GPU count GUID Locations Node count <---- values explained in section~9.4 [basecm11->nvlinksdn*[test*]->partitions*[partition-1*]]% set kind GUID [basecm11->nvlinksdn*[test*]->partitions*[partition-1*]]% show Parameter Value -------------------------------- ------------------------------------------------ Info guids: Name partition-1 Revision Index 1 Kind GUID GUIDs Mcast limit 0 Resiliency mode Full bandwidth ``` The `nvplatforminfo` command (section [7.3.4](/nvidia-mission-control/nvlinkinfo#734-nvlink-platform-information-nvplatforminfo)) in `device` mode retrieves the IB GUID for each GPU. These are the GUID values that can be set for the `GUIDs` parameter when the `Kind` parameter is set to `GUID`. ### 9.5.2 Validation Rules When committing an NVLink SDN definition, the following validation rules are enforced: * Partition names must be unique within the definition. * Partition index values must be unique within the definition. * A partition with kind set to `All` cannot coexist with other partitions. * All partitions must have the same kind. Mixed kinds of partitions in an SDN succeed, but produce a warning. * For `Node count` partitions: the sum of node counts must equal 18. * For `GPU count` partitions: the sum of GPU counts must equal 36, 72, or 144. * For `Locations` partitions: a location must not appear in multiple partitions. * The sum of multicast limits across all partitions must not exceed 1024. ## 9.6 Applying NVLink SDN Partitions Once an NVLink SDN definition has been configured, it can be applied to racks using the `applynvsdnpartitions` command in `rack` mode. This command sends the partition configuration to the NVLink switches in the specified racks. **Example** ```bash [basecm11->rack]% help applynvsdnpartitions Name: applynvsdnpartitions - Apply NVLink SDN partition configuration to racks Usage: applynvsdnpartitions [OPTIONS] Options: --location Comma separated list of locations to filter on --building Comma separated list of buildings to filter on --room Comma separated list of rooms to filter on --row Comma separated list of rows to filter on Examples: applynvsdnpartitions "Default Partition" Apply default partition to current or all racks applynvsdnpartitions --building HQ "2x9 Nodes" Apply partitions to all racks in HQ applynvsdnpartitions rack01 rack02 "3x6 Nodes" Apply partitions to the specified racks ``` The arguments and options to `applynvsdnpartitions` have these features and behaviors: * Arguments: * The last argument is always the name of the NVLink SDN definition. * Any preceding arguments are rack names. * If no racks are specified and a rack is currently selected (with `use`), then the partition is applied to that rack. * If no rack is selected either, then the partition is applied to all racks. * Options: Racks can be filtered by physical location attributes using the `--location`, `--building`, `--room`, and `--row` options. Each GPU operation takes about 1 second. With 72 GPUs per rack requiring both removal and re-addition, the command typically takes about 1 to 3 minutes per rack to complete. The command output shows the result for each rack. The operation first removes all GPUs from their current partitions, then creates the new partitions, and then adds GPUs to them. **Example** ```bash [basecm11->rack]% applynvsdnpartitions a01 "2x9 Nodes" *** a01 *** +++ stdout +++ - run: /usr/bin/nv action restore sdn partition 32766 uuid 13910308286599238321 no-reroute Action executing ... GPU uuid '13910308286599238321' removed from partition 32766 Routing table update deferred (no-reroute) Action succeeded - run: /usr/bin/nv action restore sdn partition 32766 uuid 13874279489580274353 no-reroute Action executing ... GPU uuid '13874279489580274353' removed from partition 32766 Routing table update deferred (no-reroute) Action succeeded ... (remaining GPUs removed from partition 32766) - run: /usr/bin/nv action sdn add partition 1 name partition-1 resiliency-mode full- bandwidth mcast-limit 512 Action executing ... Partition 1 is successfully created Action succeeded - run: /usr/bin/nv action update sdn partition 1 uuid 1781225835824106024 no-reroute Action executing ... GPU uuid '1781225835824106024' added to partition 1 Routing table update deferred (no-reroute) Action succeeded ... (remaining GPUs added to partition 1) - run: /usr/bin/nv action update sdn partition 1 reroute Action executing ... Recalculating routing table for partition 1... Routing table updated successfully Action succeeded - run: /usr/bin/nv action sdn add partition 2 name partition-2 resiliency-mode full- bandwidth mcast-limit 512 Action executing ... Partition 2 is successfully created Action succeeded - run: /usr/bin/nv action update sdn partition 2 uuid 12697951332570188328 no-reroute Action executing ... GPU uuid '12697951332570188328' added to partition 2 Routing table update deferred (no-reroute) Action succeeded ... (remaining GPUs added to partition 2) - run: /usr/bin/nv action update sdn partition 2 reroute Action executing ... Recalculating routing table for partition 2... Routing table updated successfully Action succeeded ``` The current partition state of the cluster can be verified after applying using the `nvsdnpartitioninfo` command in `device` mode (section [7.3.6](/nvidia-mission-control/nvlinkinfo#736-nvlink-sdn-partition-information-nvsdnpartitioninfo)). The following example shows the result after applying `3x6 Nodes` to rack `a01` and `2x4+1x10 Nodes` to rack `a02`: **Example** ```bash [basecm11]% device nvsdnpartitioninfo Rack Switches Clique Name Health Node GPUs ----- ------------------------- ------- ------------ -------- ------------------- a01 a01-nvsw-01..a01-nvsw-09 1 partition-1 healthy a01-c[01-06]:[0-3] a01 a01-nvsw-01..a01-nvsw-09 2 partition-2 healthy a01-c[07-12]:[0-3] a01 a01-nvsw-01..a01-nvsw-09 3 partition-3 healthy a01-c[13-18]:[0-3] a02 a02-nvsw-01..a02-nvsw-09 1 partition-1 healthy a02-c[01-04]:[0-3] a02 a02-nvsw-01..a02-nvsw-09 2 partition-2 healthy a02-c[05-08]:[0-3] a02 a02-nvsw-01..a02-nvsw-09 3 partition-3 healthy a02-c[09-18]:[0-3] ``` The output shows that each rack can have a different NVLink SDN definition applied. Rack `a01` has three equal partitions of 6 nodes each, while rack `a02` has two partitions of 4 nodes and one partition of 10 nodes. ## 9.7 Slurm Topology Integration Topology blocks ([here](/admin/workloadmanagement#slurm-topology-configuration)) in the Slurm workload manager can use NVLink SDN partitions. This can improve performance by ensuring that Slurm schedules jobs on nodes that belong to the same NVLink partition, so that GPU-to-GPU communication is kept within the high-bandwidth NVLink fabric. To enable using NVLink SDN partitions as topology blocks, the `Block entity` is set to `SDN` in the Slurm topology block settings: **Example** ```bash [basecm11->wlm[slurm]]% topologysettings [basecm11->wlm[slurm]->topologysettings]% blocksettings [basecm11->wlm[slurm]->topologysettings->blocksettings]% show Parameter Value -------------------------------- ------------------------------------------------ Block sizes Revision Block entity Rack Allowed racks Allowed nodegroups [basecm11->wlm[slurm]->topologysettings->blocksettings]% set blockentity SDN [basecm11->wlm*[slurm*]->topologysettings*->blocksettings*]% commit ``` Setting `Block entity` to `SDN`, means that Slurm topology blocks are then derived from the NVLink SDN partitions that are currently applied on each rack. The generated topology can be viewed with the `topology print` command. The following example shows the topology after applying `3x6 Nodes` to rack `a01`, while rack `a02` still has the `Default Partition` (all 18 nodes in a single block): **Example** ```bash [basecm11->wlm[slurm]]% topology print # Topology configuration: BlockName=a01-partition-1 Nodes=a01-c[01-06] BlockName=a01-partition-2 Nodes=a01-c[07-12] BlockName=a01-partition-3 Nodes=a01-c[13-18] BlockName=a02-default_partition Nodes=a02-c[01-18] ``` The output shows that rack `a01` is split into three blocks of 6 nodes matching the applied NVLink SDN partitions, while rack `a02` has a single block covering all 18 nodes.