What can I help you with?
NVIDIA InfiniBand - Security Overview and Guidelines
NVIDIA Docs Hub  NVIDIA Networking  Networking Software  Adapter Software  NVIDIA InfiniBand - Security Overview and Guidelines  Practical Guidelines

Practical Guidelines

Here are the guidelines for implementing security capabilities over an InfiniBand network. It is preferred that these conf_file configurations will be set before running the Subnet Manager.

  1. Apply a subnet partitioning solution (PKEY) per tenant:

    1. UFM nodes are, by default, defined as full members of the default PKEY.

    2. Define all end nodes as limited members on the default PKEY. In UFM, set the following flags in the cfg (MngNetwork section).

      Copy
      Copied!
                  
      
            
      default_membership = limited

    3. Configure the required subnet partitioning to use UFM as defined in the UFM user manual. Alternatively, you can use the REST API for configuration. For additional information please see Appendix Partitioning: General guidelines

  2. Enable InfiniBand keys for every class in use:

    • SM MADs (SMPs) – enable mkey_per_port as defined in the UFM user manual. This key can be enabled via UFM cfg, or via SM opensm.conf:

      1. Enable:

        Copy
        Copied!
                    
        
            
        cfg->m_key_per_port = true / opensm.conf->m_key_per_port = true
cfg->global_m_key_seed = <set random value> / opensm.conf-> m_key = <set random value>

      2. Other configurations:

        Copy
        Copied!
                    
        
            
        conf->m_key_lease_period 60
m_key_protection_level 2

        Note

        m_key_protection_level 2: This is the recommended default configuration, in case m_key is enabled, SM will set this value to 2.

      3. SM to generate random value for key:

        Copy
        Copied!
                    
        
            
        cfg->global_m_key_seed = 0xFFFFFFFFFFFFFFFF / opensm.conf->m_key = 0xFFFFFFFFFFFFFFFF

    • Tools (MFT) MADs – enable vs_key_per_port. To do so, you should set the following flags in conf:

      1. Enable:

        Copy
        Copied!
                    
        
            
        vs_key_enable 2
key_mgr_seed = <set random value> (this step is not must, if not done – SM will consider m_key seed for this key as well.

      2. Other configurations:

        Copy
        Copied!
                    
        
            
        vs_key_lease_period 60 (recommended default configuration)
vs_key_ci_protect_bits 1 (recommended default configuration)

      3. SM to generate random value for key:

        Copy
        Copied!
                    
        
            
        key_mgr_seed = 0xFFFFFFFFFFFFFFFF

    • Node2Node (pFRN) MADs – enable n2n_key. To do so, you should set the following flags in conf:

      1. Enable:

        Copy
        Copied!
                    
        
            
        n2n_key_enable 2
key_mgr_seed = <set random value> (this step is not a must, if not done – SM will consider m_key seed for this key as well.

        Note

        key_mgr_seed: This step is not a must, if not done – SM will consider m_key seed for this key as well.

      2. Other configurations:

        Copy
        Copied!
                    
        
            
        n2n_key_lease_period 60 (recommended default configuration)
n2n_key_protect_bit 1 (recommended default configuration)

      3. SM to generate random value for key:

        Copy
        Copied!
                    
        
            
        key_mgr_seed = 0xFFFFFFFFFFFFFFFF

    • Congestion control (CC) MADs – enable cc_key. To do so, you should set the following flags in conf:

      1. Enable:

        Copy
        Copied!
                    
        
            
        cc_key_enable 2
key_mgr_seed = <set random value> (this step is not must, if not done – SM will consider m_key seed for this key as well.

        Note

        key_mgr_seed: This step is not a must, if not done – SM will consider m_key seed for this key as well.

      2. Other configurations:

        Copy
        Copied!
                    
        
            
        cc_key_lease_period 60 (recommended default configuration)
cc_key_protect_bit 1 (recommended default configuration)

      3. SM to generate random value for key:

        Copy
        Copied!
                    
        
            
        key_mgr_seed = 0xFFFFFFFFFFFFFFFF

    • Aggregation Manager (AM) MADs – enable AM_Key and JobKey. To do so, you should set the following flags in cfg:

      1. Enable:

        Copy
        Copied!
                    
        
            
        ib_am_key <set random value>

        Note

        When am_Key is enabled, JobKey is also enabled. JobKey value is randomly generated regardless of AM_Key value.

      2. Other configurations:

        Copy
        Copied!
                    
        
            
        ib_am_key_protect_lease_period 300 (recommended default configuration)
ib_am_key_protect_bit 1 (recommended default configuration)

      3. AM to generate random value for key:

        Copy
        Copied!
                    
        
            
        ib_am_key = 0xFFFFFFFFFFFFFFFF

    • SA MADs – enable the option to randomize SA_KEY. can be enabled via UFM cfg, or via SM opensm.conf:

      Copy
      Copied!
                  
      
            
      cfg->randomize_sa_key = true / opensm.conf->sa_key = <set random value>

  3. Enable periodic MAD key update – set the following flag in conf: periodic_key_update <non-zero value> - this value is indicating the period of time (in minutes) for key update flow.

    • Recommended value is 24 hours (1440/0x5A0 minutes).

    • Minimum value is 10 minutes (any value below it will be addressed as 10 minutes).

    • This configuration will affect any per-class key that is enabled.

    • In case this feature is enabled, key seed (global_m_key_seed/mkey, key_mgr_seed) will be addresses as 0xFFFFFFFFFFFFFFFF, meaning that SM will randomly generate the keys.

  4. Enable the following SM Security features:

      1. SM_KEY random value – enable the option to randomize SM_KEY. To do so, you should set the following flags in conf:

        Copy
        Copied!
                    
        
            
        sm_key = <set random value>

      2. Allowed_sm_guids – enable the option to set list of GUIDs of devices that are allowed to run SM in the subnet (active and standby). To do so, you should set the following flags in conf:

        Copy
        Copied!
                    
        
            
        allowed_sm_guids 0 – do not allow any other SM
allowed_sm_guids <GUID A>,<GUID B>, … – specify allowed SM GUIDs

  5. Enable the following SA security features: These features are in the conf file.

    1. Enhanced SA security model – operational limitations, as defined in the UFM User Manual. Recommended values are based on common usage of SA.

    2. Disable the ability to send proxy SA requests (only when not required), as defined in the UFM User Manual.

    3. Set a limitation for the amount of registration requests to the SA, as defined in the UFM User Manual. Default configurations are meaningful and should fit all use cases.

  6. UFM supports the option to create a physical-virtual GUID mapping by sending the physical and virtual GUID list, as defined in the UFM Enterprise REST API Guide.

  7. UFM supports the option to manage static topology configurations, as defined in the UFM Enterprise REST API Guide.

  8. Enable SMP blocker feature (only when relevant):

    1. SMP blocker configuration should be set for every non-SM host.

    2. SMP blocker feature is enabled by sending the following command – “sudo mlxconfig -d <device> s SM_DISABLE =1”.

    3. SMP blocker feature is disabled by sending the following command – “sudo mlxconfig -d <device> s SM_DISABLE =0”.

    4. SMP blocker feature state can be queried by sending the following command - “sudo mlxconfig -d <device> q SM_DISABLE”
© Copyright 2025, NVIDIA. Last updated on May 18, 2025.
content here