Security in InfiniBand

InfiniBand (IB) is a computer networking communication standard known for its high throughput, low latency, scalability, and performance. These advantages have led to its widespread success in high performance computing, artificial intelligence (AI) data centers as well as hyperscale cloud environments. The IB architecture defines multiple devices and components for system communication:

• Node—A host server running any task that requires communication, connected to the network via a Host Channel Adapter (HCA).

• HCA—Host Channel Adapter. Hardware (HW) is connected to the host via a high-speed connection like PCIe, enabling it to participate in the IB network.

• Switch—A component that transfers IB packets from one port to another, based on routing tables configured by the Subnet Manager (SM). It has both In-band management to the SM, and an out-of-band management connection to the administrative management network.

• Router— Functions as a switch but operates between different subnet domains.

• Subnet Manager (SM) — Configures the local subnet and ensures its continued operation.

• NVIDIA® UFM® Appliance (Unified Fabric Manager)—Hardware running IB management software, including the NVIDIA Subnet Manager.

InfiniBand architecture is designed to address key security aspects. One of its main properties is its definition as a software-defined network. The software element that configures and manages the IB network is called Subnet Manager (SM). This, by design, enables security by providing centralized access control to network resources and preventing any host from arbitrarily adjusting network capabilities.

The SM services include standard mechanisms for configuring security, quality of service (QoS), routing, and collecting status and statistics on various fabric elements. It supports persistent network configuration backups, as well as handles the configuration and initialization of all host channel adapters (HCAs) and switches in the fabric. Each HCA and switch run a Subnet Manager Agent (SMA), which is responsible for communicating with the SM. The SM configures the network by sending Subnet Manager Packets (SMPs) via management datagrams (MADs).

InfiniBand architecture provides isolation and protection services using keys, which are unique values used in messages to authenticate the initiator of a request, based on the protocol being used. Violations of IB protocols, whether due to errors or malicious behavior, are reported and monitored by management agents through MAD trap messages which help detect and isolate possible abnormal behavior as soon as it appears.

Each node and port are identified through a 64-bit Global Unique Identifier (GUID) maintained by hardware, which helps restrict impersonation or spoofing. Communication attributes such as addressing, partitioning and multicast are centrally configured in a way that prevents high-level applications from gaining control over them or maliciously changing those attributes. Any such attempt is blocked by the switch or HCA, recorded, and sent to the administrator via various channels (switch and fabric management GUIs).

The Subnet Manager (SM) is the entity that initializes and configures the IB subnet.

The SM is responsible for:

• Discovering the physical topology of the subnet.

• Assigning Local Identifiers (LIDs) to the end nodes, switches, and routers.

• Establishing possible paths among the end

• Sweeping the subnet, discovering and managing topology changes (e.g., node reboots, link flaps, etc.).

Communication between the master SM and the SMAs (Subnet Manager Agents), and among primary and standby SMs, is performed with subnet management packets (SMPs) Management Datagrams (MADs). SMPs can be either lid-routed (LIDR) or direct-routed (DR). SMPs are sent via Queue Pair (QP) 0 on Virtual Lane 15 (lossy, management VL).

The SM has 3 ways of communicating with the subnet's nodes and switches:

• SM to Nodes: SM initiates a request to a node and gets back response via SMP MADs.

• Nodes to SM: IB nodes notify the SM that something has happened (TRAP) and get an ACK from the SM (TRAP_REPRESS).

• SM to SM: As part of the network mastership election process, candidate SMs communicate via SMINFO(GET/SET) MADs.

The SM has two important security objectives:

1. Maintain SM availability.

2. Enforce access control on subnet resources.

SM security features:

• Enable security:

  • Set IB keys

  • Set subnet partitions

Mastership election:

• Set and monitor the SM_Key provided by administrator.

• Follow the allowed_sm_guids list provided by administrator.

The InfiniBand (IB) architecture provides isolation and protection services using keys. IB keys are not involved in cryptographic operations but rather serve as access tokens. They are unique values used in messages to authenticate the initiator of a request, based on the protocol they are used for.

In InfiniBand, four types of keys are used:

• Management keys (M_Keys)

• Partition keys (P_Keys)

• Memory keys (L Keys and R Keys)

• Communication Queue keys (Q Keys)

We will explore the usage and purpose of these different keys in detail.

Management Key (M_Key)

The Management Key (M_Key) administers the control of a master subnet manager (SM) and restricts the port from any configuration or setting changes that are only allowed by the SM. The SM sets and enables each adapter port with an M_Key. There is one M_Key for a switch.

In the case of an M_Key mismatch, the packet is dropped, and an optional “Bad M_Key” trap is sent to the SM. Only an SM with the right M_Key can alter a node’s fabric configuration.

The SM can prevent the port’s M_Key from being read as long as the SM is active. The port maintains a timeout feature that reverts it to an unmanaged state if the SM becomes inactive.

The SM creates a unique M_Key value for each node in the system, preventing anyone in the network from sending arbitrary MAD messages to it.

Key Lease Period

To enable system recovery in case of a faulty SM, the IB specification introduces the concept of Key Lease Period. When a device receives a MAD with an incorrect key, it reports to the Subnet Manager and waits for the key_lease_period seconds for a response: If no response is received before the period passes, the device is no longer protected by the key, and the SM can re-establish the port. If set to 0, the lease period is infinite.

Key Protect Bit

MAD keys can have different levels of protection, defined according to the KeyProtectBit.

M_KEY supports the following configurations:

0: Allows GET requests without performing a key check. In the case of GET(PortInfo), the device replies with its M_KEY.

1: Allow GETs without performing a key check. In the case of GET(PortInfo), the device replies with an M_KEY of zero.

2, 3: Full protection provided; no commands are allowed without performing a key check.

Other MAD keys support the following configurations:

0: Full protection is provided. Class manager is allowed to read the key.

1: Full protection is provided. Class manager is not allowed to read the key.

Partitioning Key (P_Key)

InfiniBand (IB) provides network isolation through network partitions, like Ethernet VLANs (802.1Q). Partitioning defines which nodes or ports can access network resources, using hardware mechanisms to prevent ports in one partition from accessing another.

Partition members are categorized as full members and limited members. Full members can send and receive packets from both full and limited members within the partition. Limited members can only send and receive packets from full members.

The SM centrally controls partition definitions as part of the Fabric Management entity. Nodes cannot determine their own partitions.

The InfiniBand partitioning mechanism works as follows:

1. The administrator assigns group membership using the UFM management interface.

2. For any new node in the fabric, the SM:

   1. Initializes administrator-configured port attributes on the switch.

   2. Initializes administrator-configured port attributes on the server (Host).

   3. Assigns a local address (LID) to the HCA port.

   4. Assigns partition attributes to the HCA LIDs.

   5. Assigns partition attributes to the HCA’s neighboring switch port (if switch partition enforcement is enabled).

3. Host port attributes are stored in the switch and HCA hardware and are accessible only via the Management Key (M_Key), which is known by the SM and the hardware (IB silicon).

4. When an application connects to a remote node, it first resolves the destination location and partition using application-specific name resolution functions, and then opens a connection.

5. The application cannot specify the partition but uses the existing index in the local partition table, ensuring it cannot connect to a partition of which the node is not a member.

Partitioning ensures isolation among nodes sharing an InfiniBand fabric by requiring traffic packets to include a 16-bit Partition Key (P_Key). The P_Key in the packet must match a P_Key stored in the receiver’s HCA or neighbor switch’s P_Key table; otherwise, the packet is discarded. The P_Key is embedded in the packet’s BTH (L4 header) and is enforced by both the HCA and the switch. The SM controls the contents of the P_Key table via dedicated SMPs.

The P_Key is a 16-bit value comprising two fields:

• 15 LSB (P_Key[14:0]): This field defines the P_KEY value. A match occurs only if the sending (BTH.P_Key) and receiving (one of P_Key table entries) have the same 15 LSB value.

• 1 MSB (P_Key[15]): This field defines the membership type. “1” means full member, “0” means limited member. A P_Key match occurs only if at least one of the nodes (sending / receiving) is a full member. Full members can send packets to both full and limited members; limited members can only send packets to full members).

The IB subnet provides a default P_Key to enable communication between managers and nodes. The default P_Key value is 0xFFFF/0x7FFF, depending on the membership type).

The P_Key mechanism is enforced on all IB packets, except for SMP MADs.

In the diagram:

The SM is a full member of the default partition, allowing it to send messages to all nodes. The rest of the nodes are limited members of the default partition and can only receive messages.

Partition 1:

• HCA1: A full member can send and receive messages to/from HCA2 and HCA3.

• HCA2 and HCA 3: Limited members, can only receive messages.

Partition 2:

• HCA3 and HCA4: Both are full members, allowing them to send and receive messages to/from each other.

Each switch port connected to a host enforces that only the packets that are allowed to be sent or received by the specific host are being transmitted and passed.

Subnet Manager Key (SM_Key)

In the subnet, there can be only one active SM. To support failovers and redundancies, there may be standby SMs that can take over if needed.

At system bring up, all SMs are in discovery mode. Once an SM discovers another SM in the network, the one with the highest priority setting takes precedence. If two SMs have the same priority, the SM with the lower GUID takes over.

The SM_KEY mechanism protects against unregistered SMs. This unique value is negotiated during the transition of responsibility between two SMs.

Setting the SM_KEY to a unique random value configured in all SM instances, protects the system against malicious SMs who do not have this key.

To protect against denial-of-service attacks on the SM, an allowed list of valid SM GUIDs can be set in the configuration file. This ensures that the active SM ignores takeover requests from unknown Subnet Managers.

In InfiniBand fabrics, the Subnet Manager (SM) is responsible for discovering nodes, querying their information, setting network configurations, and controlling their operational state. When a cloud service provider (CSP) defines network configurations, such as Partition Key (PKEY) configurations, the node is identified by its Global Unique Identifier (GUID). These node and port GUIDs are stored on the device’s flash memory and should function as unique identifiers for each device or port. The SM retrieves the node’s GUIDs by querying the nodes.

If a Host Channel Adapter (HCA) is not considered trusted, there is a risk that its GUID response to the SM could be spoofed. Therefore, the SM should not trust the response from the device. The SM has built-in logic that knows to detect duplicate GUIDs, and to isolate (not to configure) devices with duplicate GUIDs. This means that an HCA spoofing another HCP's GUID can harm the availability of the other HCA (if their GUIDs match).

Another value that the HCA or device provides to the SM, is its type (Switch or HCA). A compromised HCA can spoof its type to the SM, leading to a false representation of the fabric.

To entirely mitigate such attack vectors, it is proposed to use a configuration file for the SM (“topo_spec”), that defines the subnet’s topology, including device GUID and device type (Switch/HCA). When using this feature, the SM will discover the subnet just like a regular InfiniBand subnet and verify device responses according to the given file.

The SM will periodically enforce the static configuration file’s definition as part of the sweep process. The static topology file consists of a list of entries, with each entry containing the following:

1. Switch GUID—The GUID of the switch.

2. Port—The Port number of the switch.

3. Neighbor GUID—The GUIDs of Switches/HCAs connected to the above switch.

4. Neighbor Port—The port number of the neighbor connected to the above switch.

5. Neighbor Type—Switch/HCA/Any (connected to the above switch).

6. Link State—The desired state of the link (Active/No-discover, or Disabled).

The static topology file must be maintained by the CSP, and reflect every subnet change to allow the subnet to work properly.

• If a node is in the static topology file, , the SM notifies the Unified Fabric Manager (UFM) and continues configuring the rest of the subnet.

• If the node is not in the static topology file but is discovered by the SM, the SM should notify the UFM, and not configure this node, ignoring or isolating it from the network (according to an SM configurable action).

For multiple virtual instances on a single HCA, the UFM supports additional Physical to Virtual GUID mapping. This feature helps protect the system against virtual GUID spoofing.

Subnet Management Packets (SMPs) are the MADs sent by the SM to its SMAs, and vice-versa. One of the main threats to an InfiniBand subnet is an attacker trying to impersonate the SM in the subnet, by sending SMPs to take control of the subnet.

These types of attacks are easy to detect (as defined in the specification) and should be mitigated by the security solutions defined by the spec (M_Key). The SMP firewall feature provides the ability to block unauthorized hosts from sending SMPs, preventing attackers from pretending to be the SM.

This feature is mainly designed for bare metal use cases, and for multi-tenant deployments. When enabled, the HCA’s firmware will block the hosts ability to send and receive SMPs (QP0 packets).

For bare-metal use cases where the CSP does not want tenants with root access to override the configuration, the Secure Host feature should be enabled. This feature prevents the host from modifying TLVs, including the SMP firewall TLV.

The IB specification defines key operations for every management class. These 64-bit keys are set and managed by KeyInfo operations. Depending on the vendor-specific implementation, keys can be set per subnet, per port (port 0 in the case of a switch) and per tenant.

The following keys are supported in the NVIDIA implementation:

Subnet Administrator (SA) and Service_Keys

The Subnet Administrator (SA) logic runs on the same node running the Subnet Manager (SM) and compliments it by responding to information or registration requests intended for the SM. It provides information such as events, service attributes, topology, switch forwarding tables, and other useful non-algorithmic data.

SA_Key

Sensitive SA operations such as setting or deleting records, require the requestor to be verified as a “trusted request” using the SA_Key.

Service_Key

ServiceRecords is the InfiniBand (IB) mechanism for publishing service providers in the subnet, making all nodes within the same partition aware of their existence. It functions like a bulletin board where all nodes can learn about available services. For example, the aggregation manager sets a ServiceRecord to inform all nodes of its data. To enforce the authenticity of such ServiceRecord request to the SA, a ServiceKey is used. When set (per service), the SA only accepts ServiceRecord SET requests if the ServiceKey matches the expected value.

The SA, like other IB managers, supports many-to-one traffic. Unlike other managers that initiate communication and expect responses (thus pacing the traffic), the SA should handle requests initiated by the nodes.

To increase availability and robustness, NVIDIA’s SA implementation supports enhanced security features in the SA configuration files. These include:

• Limiting information requests from untrusted entities.

• Restricting sensitive requests and records to read-only access.

• Limiting proxy requests when a command applies to an entity different from the request source.

• Limiting the rate of requests from nodes. The limitation can be defined by configuration files and limits the number of operations per second per port. This includes multicast operations, ServiceRecord, and event subscriptions. If an IB port exceeds this limitation, the SA will discard the request.

PM_Key—Performance Management

The PM Key is used for retrieving telemetry and counters from the fabric or, optionally, to limit the ability to reset counters or control telemetry features of the devices.

The Performance Management (PM) class provides mechanisms to retrieve performance and error statistics from IB components, collecting diagnostic data (status, configuration, counters etc.) from all IB nodes.

The PM sends performance MADs (Management Data Aggregates, class 0X04) to all nodes, which are received and answered by the Performance Manager Agent (PMA) in each node. Depending on the MAD, diagnostic data can be collected from the components’ hardware, firmware, and even software. The PMA in each IB node sends the requested diagnostic data back to the PM, which can then be further analyzed by IB diagnostic tools.

These MADs are protected by the PM key.

VS_Key—Vendor-Specific

The VS Key is used by Nvidia IB diagnostic tools, such as ibdiagnet, to monitor and analyze various network attributes.

C_Key (SMA to SMA), N2N Key

These keys secure node management communication for the IB infrastructure features that propagate status and messages between nodes , without directly involving the SM. This is used in adaptive routing algorithms and Proactive Fault Routing Notification (pFRN).

SHARP Keys (AM_Key)

NVIDIA Scalable Hierarchical Aggregation and Reduction (SHARP)™ is a mechanism that reduces the load on network compute nodes by applying compute logic on the switch itself.

SHARP is managed by a dedicated vendor-defined application class, the Aggregation Manager (AM) class. The SHARP management model includes two kinds of management messages:

1. Manager to/from Node messages: Sent from the AM to aggregation nodes (HCA/Switch) to set SHARP configurations. These are protected using the AM_Key mechanism.

2. Node to/from Node messages: Sent between two aggregation nodes to set specific SHARP job configurations. These are protected using the SHARP Job_Key mechanism. This key is set to the nodes by the AM for each job (associated with JobID) and is set for every Node-to-Node MAD.

Periodic Update of MAD keys

The Class manager maintains a database of all end-nodes and their keys, controlling them and setting these keys. Users can set a periodic update policy, which is executed by the class manager. When the periodic update feature is enabled, the class manager updates MAD keys (per class) for each end-node at a defined period of time. This feature is supported only for per-port keys (see mapping above).

In InfiniBand, as a switched fabric, packets are routed through the network to destinations according to their specific Local Identifiers (LIDs) assigned by the SM. This contrasts with Ethernet networks, where traffic may be sent to multiple ports if the specific MAC address is unknown. By design, this InfiniBand behaviour provides enhanced protection for customer traffic.

The InfiniBand protocol defines several transport types, which can be either reliably, or unreliably, connected:

• Reliable Connection (RC)

• Unreliable Datagram (UD)

• Dynamically Connected (DC)

Security in Unreliable Connected Transport (Q_Key)

When using UD traffic, the receiver opens a Queue Pair (QP) and listens to incoming traffic from multiple sources. In addition, the sender opens its own QPs and transmits unicast or multicast traffic over the fabric. The two nodes participating in a certain service communication use a key (Q_Key) that travels with the packets. If the receiver determines that the key doesn’t correspond to its own Q_Key, the packet is dropped. The SM records the event, and a trap is sent to a monitoring system. This provides an additional layer of security between members of the same partition.

Security in Reliable Connected Transport

InfiniBand reliable connected transports (RC or DC) are implemented in hardware, with no software access to the protocol. The two peers first use an entity called Communication Management (CM) to initiate the communication channel, during which each node sends its credentials to the other node via management datagrams (MADs). This is followed by a request to the hardware to move the Queue Pair (QP) to an operational state, including all connection attributes. After a QP is up and running, the hardware generates addresses, a sequence number, and two CRC numbers for every message. If any value is unexpected or wrong, the message is dropped, and the event is registered.

CM Messages Contain Two IDs

1. Local communication ID (32b): An identifier that uniquely identifies this connection from the sender’s point of view. The sender must use the same identifier for all phases of communication establishment and release. It must not reuse a Local Communication ID during the connection’s lifetime or while any messages related to the connection could still be in the fabric. The Communication ID allows the recipient to determine whether the message is a duplicate of an old message or represents a new connection request.

2. Remote communication ID (32b): An identifier that uniquely identifies this connection from the recipient's point of view. The values in the Local and Remote Communication ID fields in the Communication Management MADs are exchanged between requests and replies. The pair of (Local Communication ID, Remote Communication ID) is used to reference connections during establishment, failover management, and release. CM messages with invalid Communication IDs are not processed and are rejected.

Multicast

In IB Multicast, a node must request to join a Multicast group; the switches’ tables are updated only after the SM grants access to the node. This differs from Ethernet, where nodes can simply listen to broadcast or multicast traffic. With IB mechanisms, the host listens only to traffic explicitly destined for it.

Memory Access and Memory Protection [L_Key and R_Key]

InfiniBand transport implements Remote Direct Memory Access (RDMA), allowing data to be written directly to host memory in user space software. This eliminates redundant CPU cycles and significantly reduces end-to-end data transfer latency.

To prevent unauthorized remote memory access, InfiniBand defines memory protection mechanisms. Remote memory access requires a special key (R_Key), which is negotiated between peers and validated at the target’s Host Channel Adapter (HCA). The R_Key (Remote key) is required for external access, while the L_Key (Local Key) is used to authorize local application memory access on the same host.

A Protection Domain (PD) is the mechanism that binds between a set of QPs to a set of R_Keys. A message containing a key that doesn’t correlate with the right QP causes the packet to be dropped. The R_Key mechanism is built into the HCA silicon and cannot be disabled, even if an attack compromises root access to the host server.

IB is a managed network centrally monitored for errors and anomalies, achieved through the MADTrap mechanism:

• A trap (method type) is a message sent by a management agent to its class manager when certain asynchronous management events occur, such as protocol violations.

• The class manager programs the node with information on where to send traps, and the node stops sending the trap upon receiving a TrapRepress from the class manager.

IB devices use SMPs (SM MADs, QP = 0) to send traps to the subnet manager when certain events occur. For example, a switch may send a trap to the subnet manager when it detects a state change on one of its ports (i.e., a topology change and/or device joining or leaving).

There are two types of SM class traps:

1. Mandatory, standardized IB traps that comply with the InfiniBand specification.

2. Vendor-specific traps, which are defined and implemented by Nvidia

For a detailed list of supported traps presented in the UFM, refer to https://docs.nvidia.com/networking/display/ufmenterpriseumv6172/events+and+alarms