ibdiagnet InfiniBand Fabric Diagnostic Tool User Manual v2.11.0
IBUtils2 Utility Release Notes v2.15

Routers

This section specifies options for retrieving InfiniBand Routers configuration and verification of Floating LIDs (FLIDs) configuration on a local and adjacent networks. If FLIDs are enabled, a newly generated ibdiagnet.flid file will contain details about networks FLIDs ranges, FLIDs enabled on the routers, and distribution of FLIDs per switches in the local network.

Parameter

Description

--routers

Discovers and dumps router tables to ibdiagnet2.db_csv file, and runs FLIDs verification.

Examples:

Copy
Copied!
            

ibdiagnet –routers

  • ibdiagnet's Output: FLIDs are disabled/subnet does not have any router

    Copy
    Copied!
                

    Routers -I- Build Routers Info DB finished successfully -I- Build Routers Tables finished successfully -I- Adjacent subnets FLID Table retrieving finished successfully -I- Routers FLID Table retrieving finished successfully -I- Skipping FLID verification

  • ibdiagnet's Output: FLIDS are enabled on some routers

    Copy
    Copied!
                

    Routers -I- Build Routers Info DB finished successfully -I- Build Routers Tables finished successfully -I- Adjacent subnets FLID Table retrieving finished successfully -I- Routers FLID Table retrieving finished successfully   -I- All routers in the subnet have the same global FLID range: start=10 end=1100 -I- All routers in the subnet have the same local FLID range: start=20 end=500 -I- Local FLID range is in the global one. -I- Local subnet LID and global FLID ranges are OK. -I- Local subnet FLID verification finished successfully   -I- Ranges in the subnet: 0x233c are OK -E- Adjacent subnets FLID verification finished with errors -E- Different FLID ranges were detected for the subnet: 0x4789 [0x0002c90000000053, 0x0002c90000000053, 0x0002c9000000004a](total 3), start=583 end=667   -I- HCA and Switches FLID verification finished successfully

  • flid file: global, local and adjacent subnets ranges

    Copy
    Copied!
                

    Global FLID range: start=10 end=1100 Local subnet FLID range: start=20 end=500 Local subnet LID and global FLID ranges are OK   ------------------------------------------------------ Adjacent subnets Subnet: 0x0001 FLID range: start=20 end=500 Subnet: 0x233c FLID range: start=501 end=582 Subnet: 0x4789 different FLID ranges found on routers: [0x0002c90000000053, 0x0002c90000000053, 0x0002c9000000004a](total 3), start=583 end=667 [0x0002c90000000041](total 1), start=583 end=700

  • flid file: local and non-local FLIDs enabled on a router

    Copy
    Copied!
                

    Routers 0x0002c90000000041 - "Router1/RT", enabled FLIDs   local: 20 22 28 30 31 42 …..   non-local: 10 13 16 17 19 501 503 505 …..

  • flid file: FLIDs per switches

    Copy
    Copied!
                

    FLID per switches 0x0002c90000000064 - "SW-1-0/U1": 20 0x0002c90000000066 - "SW-1-1/U1": 21 0x0002c90000000068 - "SW-1-2/U1": 22 0x0002c9000000006a - "SW-1-3/U1": 23

When MKEY protection is enabled by the Subnet Manager on the fabric devices, following command options should be used in ibdiagnet:

Parameter

Description

--m_key <m_key>

Specifies constant MKey for the fabric. The MKey value should be specified when a single MKEY is shared by all InfiniBand devices.

The mkey value can be obtained from the opensm.conf file (m_key parameter).

--m_key_files <path to key files directory>

Specifies the path to the directory with the key_files (guid2lid, guid2mkey, neighbors, guid2cckey, guid2vskey).

Key files usually provided by opensm in /var/cache/opensm/.

Example of guid2mkey/guid2cckey/guid2vskey file:

Copy
Copied!
            

0x0002c9000000001e 0x0000000000000111 0x0002c9000000002a 0x0000000000000222 0x0002c90000000026 0x0000000000000333 0x0002c90000000006 0x0000000000000444

Example of guid2lid file:

Copy
Copied!
            

0x0002c9000000004b 0x0027 0x0027 0x0002c9000000002a 0x001a 0x001b 0x0002c90000000006 0x0004 0x0005 0x0002c90000000047 0x000e 0x000e

Example of neighbours file:

Copy
Copied!
            

0x0002c9000000004d:4 0x0002c9000000000e:1 0x0002c9000000004b:1 0x0002c9000000002e:1 0x0002c90000000049:2 0x0002c90000000022:1 0x0002c90000000006:1 0x0002c9000000004d:2

Example:

Copy
Copied!
            

ibdiagnet --mkey 0x00ff ibdiagnet --m_key_files /var/cache/opensm/ ibdiagnet --m_key_files /tmp/opensm_mkey_files/

Aggregation Management Key (AM key) is sent in SHARP Management MADs to the Aggregation nodes. (default 0). Aggregation management key is configured in SHARP Aggregation Manager configuration file.

Parameter

Description

--am_key <am_key>

Specifies constant am_key.
AM Key parameter value format: 0x<hex_value>

--am_key_file <path_to_am_key_file>

Specifies the path to am_key_file: guid2am_key.
The guid2am_key file includes pair of values GUID and key in the following format:
0x<guid_hex_value> 0x<am_key_hex_value>

Example of guid2amkey file:

Copy
Copied!
            

0x0000000000000111 0x0000000000000001 0x0000000000000222 0x2 0x0000000000000333 0x0000000003 0x0000000000000444 0x0000000000000004

Example:

Copy
Copied!
            

ibdiagnet --am_key 0x0000000000000123 ibdiagnet --am_key 0x123 ibdiagnet --am_key_file /tmp/guid2am_key ibdiagnet --am_key_file /tmp/am_keys

When VendorSpecific keys are generated and provisioned by the Subnet Manager, VendorSpecific keys should be sent in VendorSpecific MADs per endport.

Parameter

Description

--vs_key <vs_key>

Specifies constant vs_key.
VendorSpecific Key parameter value format: 0x<hex_value>

--vs_key_file <path_to_vs_key_file>

Specifies the path to vs_key_file: guid2vs_key.
The guid2vs_key file includes pair of values GUID and key in the following format:
0x<guid_hex_value> 0x<vs_key_hex_value>

Example of guid2vs_key FILE:

Copy
Copied!
            

0x0000000000000111 0x0000000000000001 0x0000000000000222 0x2 0x0000000000000333 0x0000000003 0x0000000000000444 0x0000000000000004

Example:

Copy
Copied!
            

ibdiagnet --vs_key 0x0000000000000123 ibdiagnet --vs_key 0x123 ibdiagnet --vs_key_file /tmp/guid2vs_key ibdiagnet --vs_key_file /tmp/vs_keys

When CongestionControl keys are generated and provisioned by the Subnet Manager, CongestionControl keys should be sent in CongestionControl (CC) MADs perend port.

Parameter

Description

--cc_key <cc_key>

Specifies constant cc_key.
CC Key parameter value format: 0x<hex_value>

--cc_key_file <path_to_cc_key_file>

Specifies the path to cc_key_file: guid2cc_key.
The guid2cc_key file includes pair of values GUID and key in the following format:
0x<guid_hex_value> 0x<cc_key_hex_value>

Example of guid2cc_key FILE:

Copy
Copied!
            

0x0000000000000111 0x0000000000000001 0x0000000000000222 0x2 0x0000000000000333 0x0000000003 0x0000000000000444 0x0000000000000004

Example:

Copy
Copied!
            

ibdiagnet --cc_key 0x0000000000000123 ibdiagnet --cc_key 0x123 ibdiagnet --cc_key_file /tmp/guid2cc_key ibdiagnet --cc_key_file /tmp/cc_keys

When Manager2Node keys are generated and provisioned by the Subnet Manager, Manager2Node keys should be sent in M2N (pFRN) MADs per node.

Parameter

Description

--m2n_key <m2n_key>

Specifies constant m2n_key for the fabric.
M2N Key parameter value format: 0x<hex_value>

--m2n_key_file <path_to_m2n_key_file>

Specifies the path to m2n_key_file: guid2_m2n_key.

The guid2_m2n_key file includes pair of values GUID and key in the following format:

0x<guid_hex_value> 0x<m2n_key_hex_value>

Example of guid2_m2n_key FILE:

Copy
Copied!
            

0x0000000000000111 0x0000000000000001 0x0000000000000222 0x2 0x0000000000000333 0x0000000003 0x0000000000000444 0x0000000000000004

Examples:

Copy
Copied!
            

ibdiagnet --m2n_key 0x0000000000000123 ibdiagnet --m2n_key 0x123 ibdiagnet --m2n_key_file /tmp/guid2_m2n_key ibdiagnet --m2n_key_file /tmp/m2n_keys

© Copyright 2023, NVIDIA. Last updated on May 23, 2023.