gRPC API

Hello(ClientHello) returns (ServerHello)

The first RPC call client that must be send after gRPC connection establishment

Request:

Response:

ServerHello.serverHeader.returnCode:

• NMX_ST_SUCCESS - Client hello was successful

• NMX_ST_BADPARAM - Missing or invalid gatewayId (e.g. empty string)

• NMX_ST_VERSION_MISMATCH - Major version of client and server protos do not match

Server closes the gRPC connection if:

• Client ProtoMsgMajorVersion is not the same as server

• Client attempts to call any RPC before successful completion of Hello() RPC

Subscribe(SubscribeRequest) returns (stream ServerNotification)

The client calls this RPC to subscribe for asynchronous push notifications. SubscribeRequest has the gatewayID, and the notifyOnSelfChange that should be set to false.

Asynchronous push notifications:

ServerNotification.serverHeader.returnCode:

• NMX_ST_SUCCESS - Client subscription was successful

• NMX_ST_BADPARAM - Missing or invalid gatewayId

GetStaticConfig(GetStaticConfigRequest) returns (StaticConfigResponse)

The client calls this RPC to read the current static configuration. The client may request to receive either full files, or parameters from the files

Request:

Supported values for configFileName:

• sm_config

• fm_config

• rdm_config

• chassis_mapping

Response:

StaticConfigResponse.serverHeader.returnCode:

• NMX_ST_SUCCESS - Client request for static configuration was successful

• NMX_ST_BADPARAM - Missing or invalid input parameter (e.g. empty strings)

Example of fm_config file content:

            

            
#    Description: Determine whether a default partition needs to be created
#    Possible Values:
#       0             - No partitions are created during GFM initialization. GFM disables routing until an API request 
#                     to create a partition is successful.
#       1(default)    - Creates a default partition during GFM initialization. GFM creates the partition to include
#                     all GPUs in the topology and enables routing so that all GPUs can communicate to each other.
MNNVL_ENABLE_DEFAULT_PARTITION=1
 
#    Description: Determine resiliency mode for default partition and when it is unspecified on a user partition
#    Possible Values:
#       1             - resiliency mode RESILIENCY_MODE_FULL_BANDWIDTH
#       2(default)    - resiliency mode RESILIENCY_MODE_ADAPTIVE_BANDWIDTH
#       3             - resiliency mode RESILIENCY_MODE_USER_ACTION_REQ
MNNVL_DEFAULT_RESILIENCY_MODE=2
 
#    Description: Set type of default partition (specified by location or gpuuid)
#    Possible Values:
#       1             - Creates a default partition using locations of GPUs
#       2(default)    - Creates a default partition using GPU UIDs
MNNVL_DEFAULT_PARTITION_TYPE=2
 
MNNVL_TOPOLOGY=gb200_nvl36r1_c2g4_topology
        

SetStaticConfig(SetStaticConfigRequest) returns (ReturnCode)

The client calls this RPC to update the static configuration. Client may request to update either full files, or parameters in the files.

Request:

Supported values for configFileName:

• sm_config

• fm_config

• rdm_config

• chassis_mapping

All config files are compliant to Linux INI format (https://en.wikipedia.org/wiki/INI_file), with the following restrictions:

• format “key = value”

• No support for sections

• No support for hierarchy

• Case insensitive

• Support for comments

• No support for duplicate names

• Support for Quoted values

• Support for Line continuation

• Support for Escape characters

The "fm config" file must include the MNNVL_TOPOLOGY parameter with one of the values:

• gb200_nvl36r1_c2g4_topology - 36 GPUs, Single chassis, 2 CPUs, 4 GPUs per compute tray

• gb200_nvl36r1_c2g2_topology - 36 GPUs, Single chassis, 2 CPUs, 2 GPUs per compute tray

• gb200_nvl72r1_c2g4_topology - 72 GPUs, Single chassis, 2 CPUs, 4 GPUs per compute tray

• gb200_nvl72r2_c2g4_topology - 72 GPUs, Two chassis, 2 CPUs, 4 GPUs per compute tray

• gb200_nvl72r2_c2g2_topology - 72 GPUs, Two chassis, 2 CPUs, 2 GPUs per compute tray

The chassis_mapping includes the mapping between chassis-id and chassis-serial-number, see example for the file format:

chassisId1 ABC123ABC123A
chassisId2 XYZ987XYZ987X

ReturnCode:

• NMX_ST_SUCCESS - Client request for static configuration was successful

• NMX_ST_BADPARAM - Missing or invalid input parameter

GetDomainProperties(GetDomainPropertiesRequest) returns (DomainProperties)

Domain Properties provide an overview of the current topology that NMX-C manages. They provide the maximum number of expected resources(including but not limited to gpus, switches, compute nodes, switch nodes, partitions, nvlinks) in the domain.

The client calls this RPC to get static properties of the domain. The GetDomainPropertiesRequest has the context and gatewayID.

Response:

DomainProperties.serverHeader.returnCode:

• NMX_ST_SUCCESS - successfully populated the NVLink Domain Properties

• NMX_ST_BADPARAM - Invalid input parameter

• NMX_ST_GENERIC_ERROR - Call to GFM API has failed

GetDomainStateInfo(GetDomainStateInfoRequest) returns (DomainStateInfo)

The client calls this RPC to get the dynamic properties of the domain. The GetDomainStateInfoRequest has the context and gatewayID.

Response:

DomainStateInfo.serverHeader.returnCode:

• NMX_ST_SUCCESS - successfully populated the NVLink Domain Properties

• NMX_ST_BADPARAM - Invalid input parameter

• NMX_ST_GENERIC_ERROR - Call to GFM API has failed

The following are the values of controlPlaneState:

            

            
NMX_CONTROL_PLANE_STATE_UNDEFINED  = 0;    //!< Control plane state is undefined
NMX_CONTROL_PLANE_STATE_OFFLINE    = 1;    //!< Control plane state is offline
NMX_CONTROL_PLANE_STATE_STANDBY    = 2;    //!< Control plane state is standby, NvLink Domain is not operational
NMX_CONTROL_PLANE_STATE_CONFIGURED = 3;    //!< Control plane state is configured, NvLink Domain is operational
NMX_CONTROL_PLANE_STATE_TIMEOUT    = 4;    //!< Control plane state is timeout
NMX_CONTROL_PLANE_STATE_ERROR      = 5;    //!< Control plane state is error, user provided an invalid configuration 
NMX_CONTROL_PLANE_STATE_UNCONFIGURED = 6;  //!< Control plane state is unconfigured, pending user provided configuration
        

Control Plane states and associated configuration status description strings are explained below

• NMX_CONTROL_PLANE_STATE_UNCONFIGURED - Pending required FM configuration.

  • CONFIG_PENDING_UUID - Pending NVLink Domain UUID. FM waits indefinitely until set.

  • CONFIG_PENDING_TOPOLOGY - Pending MNNVL_TOPOLOGY config. FM waits indefinitely until set.

  • CONFIG_PENDING_CHASSIS_ID_MAPPING - Pending chassis Id mapping. FM waits for GFM_WAIT_TIMEOUT_SEC until set. If not set, FM allocates the mapping during the initial resource discovery.

  • CONFIG_RECEIVED - FM received all the required configuration.

• NMX_CONTROL_PLANE_STATE_ERROR - Error validating FM configuration. Restart NMX-C after the configuration error is fixed.

  • CONFIG_ERROR_INCORRECT_TOPOLOGY_FILE - Encountered error while processing the topology file specified in MNNVL_TOPOLOGY

  • CONFIG_ERROR_CHASSIS_ID_MAPPING_COUNT - Detected mismatch between number of entries in the chassis Id mapping specified and expected number of chassis read from the topology file specified in MNNVL_TOPOLOGY

  • CONFIG_ERROR_CHASSIS_ID_MAPPING_OUT_OF_RANGE - Detected chassis Id value outside of the allowed range. Allowed range is 1 to n, where n is the number of chassis in the NVLink Domain.

  • CONFIG_ERROR_DUPLICATE_CHASSIS_SERIAL_NUMBER - Detected duplicate chassis serial number in the chassis Id mapping

  • CONFIG_ERROR_MISSING_CHASSIS - Detected fewer than expected chassis serial numbers during the initial resource discovery.

  • CONFIG_ERROR_ADDITIONAL_CHASSIS_DETECTED - Detected more than expected chassis serial number(s) during the initial resource discovery

• NMX_CONTROL_PLANE_STATE_DEGRADED - FM detected a misconfiguration. The NVLink Domain continues to work with limited capability. Fixing the misconfiguration restores full functionality.

  • CONFIG_ERROR_MISWIRED_TRUNK_PORTS - Incorrect/mis-wired trunk connection(s) detected. Use GetConnInfoList() GRPC API to get details.

• NMX_CONTROL_PLANE_STATE_CONFIGURED - FM completed configuration validation and initialization

  • CONFIG_DONE - FM completed configuration validation and initialization

The following are the values of nmxControllerHealth:

            

            
NMX_CONTROLLER_HEALTH_UNKNOWN = 0;
NMX_CONTROLLER_HEALTH_HEALTHY = 1;       
NMX_CONTROLLER_HEALTH_DEGRADED = 2;       
NMX_CONTROLLER_HEALTH_UNHEALTHY = 3; 
        

GetTopologyInfo returns (FmTopologyInfo)

The client calls this RPC to receive information on the currently discovered topology:

• Devices (GPUs and Switches) and their properties and state (DeviceTopoInfo contains one of SwitchTopoInfo or GpuTopoInfo)

• Devices ports and their properties and state(PortTopoInfo)

• Devices connectivity information

Response:

FmTopologyInfo.serverHeader.returnCode:

• NMX_ST_SUCCESS - successfully populated the NVLink Domain Properties

• NMX_ST_BADPARAM - Invalid input parameter

• NMX_ST_GENERIC_ERROR - Call to GFM API has failed

DeviceTopoInfo message:

gpuTopoInfo message:

SwitchTopoInfo message:

PortTopoInfo message:

GetComputeNodeCount(GetComputeNodeCountRequest) returns (GetComputeNodeCountResponse)

The client calls this RPC to get the number of compute nodes. It allows the client to filter the compute nodes based on Attribute, Health and Location.

Attribute reflects the allocation status of a compute node to a partition:

• NMX_COMPUTE_NODE_ATTR_ALL = 1 - all compute nodes in the NvLink Domain irrespective of the allocation status

• NMX_COMPUTE_NODE_ATTR_FREE = 2 - compute nodes in the NvLink Domain that are not allocated to any partitions

• NMX_COMPUTE_NODE_ATTR_FULLY_ALLOCATED = 3 - compute nodes in the NvLink Domain where all of its gpus are allocated to partitions

• NMX_COMPUTE_NODE_ATTR_PARTIALLY_ALLOCATED = 4 - compute nodes in the NvLink Domain where one or more(but not all) of its gpus are allocated to partitions

Compute node health:

• NMX_COMPUTE_NODE_HEALTH_HEALTHY = 1 - Fully healthy

• NMX_COMPUTE_NODE_HEALTH_DEGRADED = 2 - Some GPUs are degraded to NO_NVLINK

• NMX_COMPUTE_NODE_HEALTH_UNHEALTHY = 3 - Unable to participate in NVLink

Location and health are optional filters.

Request:

Response message:

GetComputeNodeCountResponse.serverHeader.returnCode:

• NMX_ST_SUCCESS - successfully populated the NVLink Domain Properties

• NMX_ST_BADPARAM - Invalid input parameter

• NMX_ST_GENERIC_ERROR - Call to GFM API has failed

GetComputeNodeLocationList(GetComputeNodeLocationListRequest) returns (GetComputeNodeLocationListResponse)

Client calls this RPC to get the locations of the compute nodes. It allows the client to filter the compute nodes based on Attribute, Health and Location.

Attribute reflects the allocation status of a compute node to a partition:

• NMX_COMPUTE_NODE_ATTR_ALL = 1 - all compute nodes in the NvLink Domain irrespective of the allocation status

• NMX_COMPUTE_NODE_ATTR_FREE = 2 - compute nodes in the NvLink Domain that are not allocated to any partitions

• NMX_COMPUTE_NODE_ATTR_FULLY_ALLOCATED = 3 - compute nodes in the NvLink Domain where all of its gpus are allocated to partitions

• NMX_COMPUTE_NODE_ATTR_PARTIALLY_ALLOCATED = 4 - compute nodes in the NvLink Domain where one or more(but not all) of its gpus are allocated to partitions

Compute node health:

• NMX_COMPUTE_NODE_HEALTH_HEALTHY = 1 - Fully healthy

• NMX_COMPUTE_NODE_HEALTH_DEGRADED = 2 - Some GPUs are degraded to NO_NVLINK

• NMX_COMPUTE_NODE_HEALTH_UNHEALTHY = 3 - Unable to participate in NVLink

Location and health are optional filters.

Request message:

Response message:

GetComputeNodeLocationListResponse.serverHeader.returnCode:

• NMX_ST_SUCCESS - successfully populated the NVLink Domain Properties

• NMX_ST_BADPARAM - Invalid input parameter

• NMX_ST_GENERIC_ERROR - Call to GFM API has failed

GetComputeNodeInfoList(GetComputeNodeInfoListRequest) returns (GetComputeNodeInfoListResponse)

The client calls this RPC to get information on the compute nodes. The client uses it get both static(location and number of gpus in the node) and dynamic information(partition Ids and health) about compute nodes.

Request message:

Response message:

GetComputeNodeInfoListResponse.serverHeader.returnCode:

• NMX_ST_SUCCESS - successfully populated the NVLink Domain Properties

• NMX_ST_BADPARAM - Invalid input parameter

• NMX_ST_GENERIC_ERROR - Call to GFM API has failed

ComputeNodeInfo message:

GetGpuInfoList(GetGpuInfoListRequest) returns (GetGpuInfoListResponse)

The client calls this RPC to get information on the GPUs.

The client can filter GPUs based on location or partition by passing the attribute NMX_GPU_ATTR_LOCATION or NMX_GPU_ATTR_PARTITION_ID respectively.

Notes:

• Specifying NMX_GPU_ATTR_PARTITION_ID and partitionId=0 provide info on GPUs that are not associated with any partition

• Specifying NMX_GPU_ATTR_ALL ignores the location or partition values set in the request

Request message:

GPU health values:

• NMX_ GPU_HEALTH_HEALTHY = 1 //!< Fully healthy

• NMX_ GPU_HEALTH_DEGRADED = 2 //!< One or more links are down

• NMX_ GPU_HEALTH_NO_NVLINK = 3 //!< Unable to participate in NVLink partition

• NMX_GPU_HEALTH_DEGRADED_BW = 4 //!< GPU operates in degraded bandwidth

Note: to get all healthy GPU not associated with any partition set:

• attr to GPU_PARTITION_ID

• partitionId to 0

Response message:

GetGpuInfoListResponse.serverHeader.returnCode:

• NMX_ST_SUCCESS - successfully populated the NVLink Domain Properties

• NMX_ST_BADPARAM - Invalid input parameter

• NMX_ST_GENERIC_ERROR - Call to GFM API has failed

GpuInfo message:

GetSwitchNodeCount(GetSwitchNodeCountRequest) returns (GetSwitchNodeCountResponse)

The client calls this RPC to get the number of switch nodes.

The client can filter the switch nodes using attribute and health. Note that in the NVLink5 topologies do not have switch nodes that are of type NMX_SWITCH_NODE_ATTR_L2.

Request message:

Response message:

GetSwitchNodeCountResponse.serverHeader.returnCode:

• NMX_ST_SUCCESS - successfully populated the NVLink Domain Properties

• NMX_ST_BADPARAM - Invalid input parameter

• NMX_ST_GENERIC_ERROR - Call to GFM API has failed

GetSwitchNodeLocationList(GetSwitchNodeLocationListRequest) returns (GetSwitchNodeLocationListResponse)

The client calls this RPC to get the location of switch nodes.

The client can filter the switch nodes using attribute and health. Note that in the NVLink5 topologies do not have switch nodes that are of type NMX_SWITCH_NODE_ATTR_L2

Request message:

Response message:

GetSwitchNodeLocationListResponse.serverHeader.returnCode:

• NMX_ST_SUCCESS - successfully populated the NVLink Domain Properties

• NMX_ST_BADPARAM - Invalid input parameter

• NMX_ST_GENERIC_ERROR - Call to GFM API has failed

GetSwitchNodeInfoList(GetSwitchNodeInfoListRequest) returns (GetSwitchNodeInfoListResponse)

The client calls this RPC to get information on the switch nodes. The client uses it get both static(location and number of switches in the node) and dynamic information(partition Ids and health) about switch nodes

Request message:

Response message:

GetSwitchNodeInfoListResponse.serverHeader.returnCode:

• NMX_ST_SUCCESS - successfully populated the NVLink Domain Properties

• NMX_ST_BADPARAM - Invalid input parameter

• NMX_ST_GENERIC_ERROR - Call to GFM API has failed

SwitchNodeInfo message:

GetSwitchInfoList(GetSwitchInfoListRequest) returns (GetSwitchInfoListResponse)

The client calls this RPC to get information on the switches.

Request message:

Response message:

GetSwitchInfoListResponse.serverHeader.returnCode:

• NMX_ST_SUCCESS - successfully populated the NVLink Domain Properties

• NMX_ST_BADPARAM - Invalid input parameter

• NMX_ST_GENERIC_ERROR - Call to GFM API has failed

SwitchInfo message:

GetPartitionCount(GetPartitionCountRequest) returns (GetPartitionCountResponse)

The client calls this RPC to get the number of partitions. The number of partitions can be filtered using the numGpus, numNodes or health filters. The infoAttr value decides which filter to choose between numGpus and numNodes. If set to ATTR_ALL, numGpus and numNodes are ignored.

            

            
enum PartitionInfoAttr {  
    NMX_PARTITION_INFO_ATTR_UNDEFINED = 0;                 
    NMX_PARTITION_INFO_ATTR_ALL = 1;                 //!< All Partitions
    NMX_PARTITION_INFO_ATTR_NUM_GPUS = 2;            //!< Number of Partitions with a specific GPU size
    NMX_PARTITION_INFO_ATTR_NUM_COMPUTE_NODES = 3;   //!< Number of Partitions with a specific number of Compute nodes
} 
 
enum PartitionHealth {
    NMX_PARTITION_HEALTH_UNKNOWN = 0;            
    NMX_PARTITION_HEALTH_HEALTHY = 1;            //!< Partition is healthy
    NMX_PARTITION_HEALTH_DEGRADED_BANDWIDTH = 2; //!< Partition is in degraded bandwidth
    NMX_PARTITION_HEALTH_DEGRADED = 3;           //!< One or more GPUs has routing disabled
    NMX_PARTITION_HEALTH_UNHEALTHY = 4;          //!< Partition is unhealthy
}
        

Request message:

Response message:

GetPartitionCountResponse.serverHeader.returnCode:

• NMX_ST_SUCCESS - successfully populated the NVLink Domain Properties

• NMX_ST_BADPARAM - Invalid input parameter

• NMX_ST_GENERIC_ERROR - Call to GFM API has failed

GetPartitionIdList(GetPartitionIdListRequest) returns (GetPartitionIdListResponse)

The client calls this RPC to get the partitions IDs. The partitionIds can be filtered using the numGpus, numNodes or health filters. The infoAttr value decides which filter to choose between numGpus and numNodes. If set to ATTR_ALL, numGpus and numNodes are ignored.

            

            
enum PartitionInfoAttr {  
    NMX_PARTITION_INFO_ATTR_UNDEFINED = 0;                 
    NMX_PARTITION_INFO_ATTR_ALL = 1;                 //!< All Partitions
    NMX_PARTITION_INFO_ATTR_NUM_GPUS = 2;            //!< Number of Partitions with a specific GPU size
    NMX_PARTITION_INFO_ATTR_NUM_COMPUTE_NODES = 3;   //!< Number of Partitions with a specific number of Compute nodes
} 
 
enum PartitionHealth {
    NMX_PARTITION_HEALTH_UNKNOWN = 0;            
    NMX_PARTITION_HEALTH_HEALTHY = 1;            //!< Partition is healthy
    NMX_PARTITION_HEALTH_DEGRADED_BANDWIDTH = 2; //!< Partition is in degraded bandwidth
    NMX_PARTITION_HEALTH_DEGRADED = 3;           //!< One or more GPUs has routing disabled
    NMX_PARTITION_HEALTH_UNHEALTHY = 4;          //!< Partition is unhealthy
}
        

Request message:

Response message:

GetPartitionIdListResponse.serverHeader.returnCode:

• NMX_ST_SUCCESS - successfully populated the NVLink Domain Properties

• NMX_ST_BADPARAM - Invalid input parameter

• NMX_ST_GENERIC_ERROR - Call to GFM API has failed

Partition message:

GetPartitionInfoList(GetPartitionInfoListRequest) returns (GetPartitionInfoListResponse)

The client calls this RPC to get the partitions information. User can pass in a list of partitionIds or partitionNames or both. Only valid partitionIds or partitionNames are considered. If both lists are empty, information for all partitions are returned.

Request message:

Response message:

GetPartitionInfoListResponse.serverHeader.returnCode:

• NMX_ST_SUCCESS - successfully populated the NVLink Domain Properties

• NMX_ST_BADPARAM - Invalid input parameter

• NMX_ST_GENERIC_ERROR - Call to GFM API has failed

PartitionInfo message:

The meaning of the ResiliencyMode values are:

• Full Bandwidth: On a trunk link failure, partition will attempt an automatic recovery. If spare trunk links are not available, GPUs will be excluded from the fabric to maintain full bandwidth for the rest of the GPUs.

• Adaptive Bandwidth: On a trunk link failure, partition will attempt an automatic recovery. If spare trunk links are not available, the partition's GPUs will operate with a lower bandwidth than optimal.

• User Action Required: On a trunk link failure, partition will attempt an automatic recovery. If spare trunk links are not available, the partition will go into an unhealthy state which requires user action for recovery. Example actions would be providing additional trunk links or removing GPUs from the partition.

CreatePartition(CreatePartitionRequest) returns (CreatePartitionResponse)

The client calls this RPC to create a partition. The user provides a list of GPU UIDs or a list of GPU Locations. If both lists are empty, the RPC returns NMX_ST_BADPARAM. The user can specify the attributes for the partition to be created.

• If the partition creation is requested with the resiliency mode NMX_RESILIENCY_MODE_UNDEFINED, a default resiliency mode as specified in the configuration file MNNVL_DEFAULT_RESILIENCY_MODE is used

• If the partition creation is requested with a multicastGroupsLimit that cannot be satisfied, the RPC returns NMX_ST_RESOURCE_EXHAUSTED. If the value is not a multiple of 4, the RPC returns NMX_ST_BADPARAM

partitionId's are allocated from 1 to 0x7FFD. Partition Id 0x7FFE is reserved for Default Partition. The user can specify a partitionId as part of the creation request. If the specified ID is greater than 0x7FFD, the RPC returns NMX_ST_BADPARAM.

• If a partition creation request succeeds, and a later request to create another partition with the same set of parameters is received, the RPC returns NMX_ST_PARTITION_EXISTS

• If a partition cannot be created owing to exhaustion of partitionIds, the RPC returns NMX_ST_RESOURCE_EXHAUSTED

• If a requested GPU is already part of another partition, the RPC returns NMX_ST_RESOURCE_IN_USE

• If a requested GPU does not have a valid UID or a valid location, the RPC returns NMX_ST_RESOURCE_BAD

• If the requested partitionId is already in use by another partition, the RPC returns NMX_ST_PARTITION_ID_IN_USE

• If the requested partitionName is already in use by another partition, the RPC returns NMX_ST_PARTITION_NAME_IN_USE

• If the creation fails due to any other error, the RPC returns NMX_ST_GENERIC_ERROR

Request message:

Response message:

CreatePartitionResponse.serverHeader.returnCode:

• NMX_ST_SUCCESS - successfully populated the NVLink Domain Properties

• NMX_ST_BADPARAM - Invalid input parameter, requested multicastGroupsLimit is not a multiple of 4, requested partitionId is greater than 0x7FFD(allowed range is 0x1-0x7FFD)

• NMX_ST_GENERIC_ERROR - Call to GFM API has failed due to an internal error

• NMX_ST_RESOURCE_EXHAUSTED - Requested multicastGroupsLimit cannot be satisfied, partitionIds are exhausted

• NMX_ST_PARTITION_EXISTS - Partition with requested parameters already exists

• NMX_ST_RESOURCE_IN_USE - Requested resource is already part of another partition

• NMX_ST_RESOURCE_BAD - Requested resource does not have a valid UID or a location

• NMX_ST_PARTITION_ID_IN_USE - Requested partitionId is already in use

• NMX_ST_PARTITION_NAME_IN_USE - Requested partitionName is already in use

DeletePartition(DeletePartitionRequest) returns (DeletePartitionResponse)

The client calls this RPC to delete a partition. User provides either a valid name or a partition Id or both as part of the request

Request message:

Response message:

DeletePartitionResponse.serverHeader.returnCode:

• NMX_ST_SUCCESS - successfully populated the NVLink Domain Properties

• NMX_ST_BADPARAM - Invalid input parameter

• NMX_ST_GENERIC_ERROR - Call to GFM API has failed

• NMX_ST_PARTITION_ID_NOT_IN_USE - Requested partitionId not in use

AddGpusToPartition(UpdatePartitionRequest) returns (UpdatePartitionResponse)

Client calls this RPC to add GPUs to a partition. In a partition that is contained within a chassis boundary, the reroute flag is ignored. In a partition that uses trunk links and hence crosses the chassis boundary, the reroute flag determines if the trunk link routing is adjusted:

• If reroute = true(default), the trunk link routing is adjusted when additional GPUs are added to the partition. This can disrupt applications running in the partition.

• If reroute = false, the trunk link routing is not adjusted when additional GPUs are added to the partition.

The RPC also allows a special operation called "reroute" where both the locationList and gpuUidList are empty, and reroute is set to True. This allows the client to adjust the trunk link routing (i.e. "reroute") for the partition to use an optimal number of trunk links. This operation can cause traffic disruption and must be used with caution.

• When one of the location list or the gpuUid list is populated and this does not match the type (location or GPU UID) with which the partition currently operates, the RPC returns NMX_ST_NOT_SUPPORTED. The type of the partition can be determined from the PartitionInfo message which is returned from the GetPartitionInfoList() RPC call

• If a requested GPU does not have a valid GUID or a valid location, the RPC returns NMX_ST_RESOURCE_BAD

• If the requested partition ID is not in use, the RPC returns NMX_ST_PARTITION_ID_NOT_IN_USE

• If the GPU to be added is already part of a partition, the RPC returns NMX_ST_RESOURCE_IN_USE

Request message:

User can request partition reroute by setting:

• locationList to empty array

• gpuUid to empty array

• reroute to true

Response message:

UpdatePartitionResponse.serverHeader.returnCode:

• NMX_ST_SUCCESS - successfully populated the NVLink Domain Properties

• NMX_ST_BADPARAM - Invalid input parameter, requested partitionId does not exist

• NMX_ST_GENERIC_ERROR - Call to GFM API has failed

• NMX_ST_NOT_SUPPORTED - Requested type(location or gpu UID) does not match the partition type

• NMX_ST_RESOURCE_BAD - Requested resource does not have a valid UID or a location

• NMX_ST_PARTITION_ID_NOT_IN_USE - Requested partitionId not in use

• NMX_ST_RESOURCE_IN_USE - Requested resource is already part of another partition

RemoveGpusFromPartition(UpdatePartitionRequest) returns (UpdatePartitionResponse)

The client calls this RPC to remove GPUs from a partition. UpdatePartitionRequest and UpdatePartitionResponse messages are the same as in AddGpusToPartition.

• If the number of GPUs to be removed is equal to the number of GPUs in the partition, the RPC returns NMX_ST_NOT_SUPPORTED

• If the number of GPUs to be removed is greater than the number of GPUs in the partition, the RPC returns NMX_ST_BADPARAM

• If the GPU to be removed is not part of the partition, the RPC returns NMX_ST_RESOURCE_NOT_IN_USE

UpdatePartitionResponse.serverHeader.returnCode:

• NMX_ST_SUCCESS - successfully populated the NVLink Domain Properties

• NMX_ST_BADPARAM - Invalid input parameter

• NMX_ST_GENERIC_ERROR - Call to GFM API has failed

• NMX_ST_NOT_SUPPORTED - Requested number of gpus to be removed is equal to the number of gpus in the partition

• NMX_ST_BADPARAM - Requested number of gpus to be removed is greater than the number of GPUs in the partition

• NMX_ST_RESOURCE_NOT_IN_USE - Requested resource to be removed is not part of the partition

GetConnCount(GetConnCountRequest) returns (GetConnCountResponse)

The client calls this RPC to get the number of fabric connections. It allows the client to filter the connections based on Attribute, Type and Location.

            

            
enum ConnAttr {
    NMX_NVLINK_CONN_ATTR_UNKNOWN = 0;                  
    NMX_NVLINK_CONN_ATTR_EXPECTED = 1;          //!< All expected connections as per FM Topology
    NMX_NVLINK_CONN_ATTR_DISCOVERED = 2;        //!< All discovered connections
    NMX_NVLINK_CONN_ATTR_EXPECTED_ACTIVE = 3;   //!< All expected active connections as per FM Topology
    NMX_NVLINK_CONN_ATTR_EXPECTED_INACTIVE = 4; //!< All expected inactive connections
    NMX_NVLINK_CONN_ATTR_UNEXPECTED = 5;        //!< All unexpected connections which are discovered
}
        

            

            
enum ConnType {
    NMX_NVLINK_CONN_TYPE_UNKNOWN = 0; 
    NMX_NVLINK_CONN_TYPE_ALL = 1;    //!< Dump the GPU and switch connections
    NMX_NVLINK_CONN_TYPE_GPU = 2;    //!< Dump the GPU connections
    NMX_NVLINK_CONN_TYPE_SWITCH = 3; //!< Dump the Switch connections
}
        

Various combinations of attributes and types are provided below to provide an idea on how connection information can be mined:

Request message:

Response message:

GetConnCountResponse.serverHeader.returnCode:

• NMX_ST_SUCCESS - successfully populated the NVLink Domain Properties

• NMX_ST_BADPARAM - Invalid input parameter

• NMX_ST_GENERIC_ERROR - Call to GFM API has failed

GetConnInfoList(GetConnInfoListRequest) returns (GetConnInfoListResponse)

The client calls this RPC to get information on the fabric connections. It allows the client to filter the connections based on Attribute, Type and Location.

            

            
enum ConnAttr {
    NMX_NVLINK_CONN_ATTR_UNKNOWN = 0;                  
    NMX_NVLINK_CONN_ATTR_EXPECTED = 1;          //!< All expected connections as per FM Topology
    NMX_NVLINK_CONN_ATTR_DISCOVERED = 2;        //!< All discovered connections
    NMX_NVLINK_CONN_ATTR_EXPECTED_ACTIVE = 3;   //!< All expected active connections as per FM Topology
    NMX_NVLINK_CONN_ATTR_EXPECTED_INACTIVE = 4; //!< All expected inactive connections
    NMX_NVLINK_CONN_ATTR_UNEXPECTED = 5;        //!< All unexpected connections which are discovered
}
        

            

            
enum ConnType {
    NMX_NVLINK_CONN_TYPE_UNKNOWN = 0; 
    NMX_NVLINK_CONN_TYPE_ALL = 1;    //!< Dump the GPU and switch connections
    NMX_NVLINK_CONN_TYPE_GPU = 2;    //!< Dump the GPU connections
    NMX_NVLINK_CONN_TYPE_SWITCH = 3; //!< Dump the Switch connections
        

The API returns a list of connections and the state of each connection:

            

            
enum ConnState {
    NMX_NVLINK_CONN_STATE_UNKNOWN = 0;     
    NMX_NVLINK_CONN_STATE_ACTIVE = 1;   //!< Active link or connection state
    NMX_NVLINK_CONN_STATE_INACTIVE = 2; //!< Inactive link or connection state
}
        

GetConnInfoListRequest message:

Response message:

GetConnInfoListResponse.serverHeader.returnCode:

• NMX_ST_SUCCESS - successfully populated the NVLink Domain Properties

• NMX_ST_BADPARAM - Invalid input parameter

• NMX_ST_GENERIC_ERROR - Call to GFM API has failed

ConnInfo message:

LinkEndPoint message:

GetConnInfoCombined(GetConnInfoCombinedRequest) returns (ConnInfoCombined)

The client calls this RPC to get information on the fabric trunk connections that are mis-wired.

Response message:

ConnInfoCombined.serverHeader.returnCode:

• NMX_ST_SUCCESS - successfully completed the call

• NMX_ST_BADPARAM - Invalid input parameter

• NMX_ST_GENERIC_ERROR - Call to GFM API has failed

FactoryReset(FactoryResetRequest) returns (ReturnCode)

The client calls this RPC to perform factory reset to the NMX-Controller. After this call is completed the NMX-Controller configuration and state is as initially delivered from factory.

ReturnCode:

• NMX_ST_SUCCESS - successfully completed the call

• NMX_ST_BADPARAM - Invalid input parameter

• NMX_ST_GENERIC_ERROR - Call has failed