Appendix: UFM Subnet Manager Default Properties
The following table provides a comprehensive list of UFM SM default properties.
Category | Property | Config File Attribute | Default | Mode/ Field | Description |
Keys | M_Key | m_key | 0x0000000000000000 | RW | M_Key value sent to all ports -used to qualify the set(PortInfo) |
M_Key Lease Period | m_key_lease_period | 0 | RW | The lease period used for the M_Key on the subnet in [sec] | |
SM_Key | sm_key | 0x0000000000000001 | RO | SM_Key value of the SM used for SM authentication | |
SA_Key | sa_key | 0x0000000000000001 | RO | SM_Key value to qualify rcv SA queries as 'trusted' | |
Subnet Prefix | subnet_prefix | 0xfe80000000000000 | RW | Subnet prefix used on the subnet 0xfe80000000000000 | |
LMC | lmc | 0 | RW | The LMC value used on the subnet: 0-7 Changes to the LMC parameter require a UFM restart. | |
Partition enforcement | part_enforce |
| RO | Partition enforcement type (for switches) | |
MKEY lookup | m_key_lookup | FALSE | RW | If FALSE, SM will not try to determine the m_key of unknown ports. | |
M_Key Per Port | m_key_per_port | FALSE | RW | When m_key_per_port is enabled, OpenSM will generate an M_Key for each port | |
Limits | Packet Life Time | packet_life_time | 0x12 | RW | The maximum lifetime of a packet in a switch. The actual time is 4.096usec * 2^<packet_life_time> The value 0x14 disables the mechanism |
VL Stall Count | vl_stall_count | 0x07 | RO | The number of sequential packets dropped that cause the port to enter the VL Stalled state. The result of setting the count to zero is undefined. | |
Leaf VL Stall Count | leaf_vl_stall_count | 0x07 | RO | The number of sequential packets dropped that causes the port to enter theleaf VL Stalled state. The count is for switch ports driving a CA or gateway port. The result of setting the count to zero is undefined. | |
Head Of Queue Life time | head_of_queue_lifetime | 0x12 | RW | The maximum time a packet can wait at the head of the transmission queue. The actual time is 4.096usec * 2^<head_of_queue_lifetime> The value 0x14 disables the mechanism | |
Leaf Head Of Queue Life time | leaf_head_of_queue_lifetime | 0x10 | RW | The maximum time a packet can wait at the head of queue on a switch port connected to a CA or gateway port. | |
Maximal Operational VL | max_op_vls | 3 | RW | Limit of the maximum operational VLs | |
Force Link Speed | force_link_speed | 15 (Do NOT change) | RO | Force PortInfo: LinkSpeedEnabled on switch ports. If 0, do not modify. Values are: 1: 2.5 Gbps 3: 2.5 or 5.0 Gbps 5: 2.5 or 10.0 Gbps 7: 2.5 or 5.0 or 10.0 Gbps 2,4,6,8-14 Reserved 15: set to PortInfo: LinkSpeedSupported For a fabric with DDR 10 Gb Ethernet Gateways (sRB-20210G) and 4036E version lower than 3.7.0.896, this parameter should be set to None (0) – auto negotiate link speed according to INI settings. | |
Limits | Subnet Timeout | subnet_timeout | 18 (1second) | RW | The subnet_timeout code that will be set for all the ports. The actual timeout is 4.096usec * 2^<subnet_timeout> |
Local PHY Error Threshold | local_phy_errors_threshold | 0x08 | RW | Threshold of local phy errors for sending Trap 129 | |
Overrun Errors Threshold | overrun_errors_threshold | 0x08 | RW | Threshold of credit overrun errors for sending Trap 130 | |
Sweep | Sweep Interval | sweep_interval | 10 | RW | The time in seconds between subnet sweeps (Disabled if 0) |
Reassign Lids | reassign_lids | FALSE (disabled) | RW | If TRUE (enabled), all LIDs are reassigned | |
Force Heavy Sweep | force_heavy_sweep_window | -1 | RW | Forces heavy sweep after number of light sweeps (-1 disables this option and 0 will cause every sweep to be heavy) | |
Sweep On trap | sweep_on_trap | TRUE (enabled) | RW | If TRUE every trap will cause a heavy sweep | |
Alternative Route Calculation | max_alt_dr_path_retries | 4 | RW | Maximum number of attempts to find an alternative direct route towards unresponsive ports | |
Fabric Rediscovery | max_seq_redisc | 2 | RW | Max Failed Sequential Discovery Loops | |
Offsweep Rebalancing Enable | offsweep_balancing_enabled | FALSE | RW | Enable/Disable idle time routing rebalancing | |
Offsweep Rebalancing Window | offsweep_balancing_window | 180 | RW | Set the time window in seconds after sweep to start rebalancing | |
Handover | SM Priority | sm_priority | 15 | RO | SM (enabled). The priority used for deciding which is the master. Range is 0 (lowest priority) to 15 (highest) |
Ignore Other SMs | ignore_other_sm | FALSE (disabled) | RO | If TRUE other SMs on the subnet should be ignored | |
Polling Timeout | sminfo_polling_timeout | 10 | RO | Timeout in seconds between two active master SM polls | |
Polling Retries | polling_retry_number | 4 | RO | Number of failing remote SM polls that declares it non-operational | |
Honor GUID-to-LID File | honor_guid2lid_file | FALSE (disabled) | RO | If TRUE, honor the guid2lid file when coming out of standby state, if the guid2lid file exists and is valid | |
Threading | Max Wire SMPs | max_wire_smps | 8 | RW | Maximum number of SMPs sent in parallel |
Transaction Timeout | transaction_timeout | 200 | RO | The maximum time in [msec] allowed for a transaction to complete | |
Max Message FIFO Timeout | max_msg_fifo_timeout | 10000 | RO | Maximum time in [msec] a message can stay in the incoming message queue | |
Single Thread | single_thread | FALSE (disabled) | RO | Use a single thread for handling SA queries | |
Routing Threads | routing_threads_num | 0 | RW | Number of threads to be used for parallel minhop/updn calculations. If 0, number of threads will be equal to number of processors. | |
Routing Threads Per Core | max_threads_per_core | 0 | RW | Max number of threads that are allowed to run on the same processor during parallel computing. If 0, threads assignment per processor is up to operating system initial assignment. | |
Logging | Log File | log_file | /opt/ufm/files/log/opensm.log | RO | Path of Log file to be used |
Log Flags | log_flags | Error and Info 0x03 | RW | The log flags, or debug level being used. | |
Force Log Flush | force_log_flush | FALSE (disabled) | RO | Force flush of the log file after each log message | |
Log Max Size | log_max_size | 4096 | RW | Limit the size of the log file in MB. If overrun, log is restarted | |
Accumulate Log File | accum_log_file | TRUE (enabled) | RO | If TRUE, will accumulate the log over multiple OpenSM sessions | |
Dump Files Directory | dump_files_dir | /opt/ufm/files/log | RO | The directory to hold the file SM dumps (for multicast forwarding tables for example). The file is used collects information. | |
Syslog log | syslog_log | 0x0 | RW | Sets a verbosity of messages to be printed in syslog | |
Misc | Node Names Map File | node_name_map_name | Null | RW | Node name map for mapping node's to more descriptive node descriptions |
SA database File | sa_db_file | Null | RO | SA database file name | |
No Clients Reregistration | no_clients_rereg | FALSE (disabled) | RO | If TRUE, disables client reregistration | |
Exit On Fatal Event | exit_on_fatal | TRUE (enabled) | RO | If TRUE (enabled), the SM exits for fatal initialization issues | |
Switch Isolation From Routing | held_back_sw_file | Null | RW | File that contains GUIDs of switches isolated from routing | |
Enable Mellanox SHARP support | sharp_enabled | 0 (IGNORED - Don't set any value) | RW | Defines whether to enable/disable Mellanox SHARP on supporting ports. | |
Multicast | Disable Multicast | disable_multicast | FALSE (disabled) | RO | If TRUE, OpenSM should disable multicast support and no multicast routing is performed |
Multicast Group Parameters | default_mcg_mtu | 0 | RW | Default MC group MTU for dynamic group creation. 0 disables this feature, otherwise, the value is a valid IB encoded MTU | |
Multicast | Multicast Group Parameters | default_mcg_rate | 0 | RW | Default MC group rate for dynamic group creation. 0 disables this feature, otherwise, the value is a valid IB encoded rate |
Multicast | Enable incremental multicast routing | enable_inc_mc_routing | FALSE | RW | Enable incremental multcast routing |
Multicast | MC root file | mc_roots_file | null | RW | Specify predefined MC groups root guids |
QoS | Settings | qos | FALSE (disabled) *From UFM v3.7 and on | RW | If FALSE (disabled), UFM will not apply QoS settings |
Unhealthy Ports | Enabling Unhealthy Ports | hm_unhealthy_ports_checks | TRUE | RW | Enables Unhealthy Ports configuration |
Configuration file | hm_ports_health_policy_file | null | RW | Specifies configuration file for health policy | |
Unhealthy actions | hm_sw_manual_action | no_discover | RW | Specifies what to do with switch ports which were manually added to health policy file | |
MADs validation | validate_smp | TRUE | RW | If set to TRUE, opensm will ignore nodes sending non-spec compliant MADs. When set to FALSE, opensm will log the warning in the opensm log file about non-compliant node | |
Routing | Randomization | scatter_ports | 8 | RW | Assigns ports in a random order instead of round-robin. If 0, the feature is disabled, otherwise use the value as a random seed. Applicable to the MINHOP/UPDN routing algorithms |
Randomization | guid_routing_order_no_scatter | TRUE | RO | Do not use scatter for ports defined in guid_routing_order file | |
Unicast Routing Caching | use_ucast_cache | TRUE | RW | Use unicast routing cache for routing computation time improvement | |
GUID Ordering During Routing | guid_routing_order_file | NULL | RW | The file holding guid routing order of particular guids (for MinHop, Up/Down) | |
Torus Routing | torus_config | /opt/ufm/files/conf/opensm/torus-2QoS.con | RW | Torus-2QoS configuration file name | |
Routing Chains | pgrp_policy_file | NULL | RW | The file holding the port groups policy | |
topo_policy_file | NULL | RW | The file holding the topology policy | ||
rch_policy_file | NULL | RW | The file holding the routing chains policy | ||
max_topologies_per_sw | 1 | RO | Defines maximal number of topologies to which a single switch may be assigned during routing engine chain configuration. | ||
Incremental Multicast Routing (IMR) | enable_inc_mc_routing | TRUE | RW | If TRUE, MC nodes will be added to the MC tree incrementally. When set to FALSE, the tree will be recalculated per eachg change. | |
MC Global root | mc_primary_root_guid/mc_secondary_root_guid | 0x0000000000000000 (for both) | RW | Primary and Secondary global mc root guid | |
Scatter ports | use_scatter_for_switch_lid | FALSE | RW | Use scatter when routing to the switch’s LIDs | |
updn lid tracking mode | updn_lid_tracking_mode | FALSE | RW | Controls whether OpenSM will use LID tracking or not when routing the fabric with UpDown | |
Events | Event Subscription Handling | drop_subscr_on_report_fail | FALSE | RW | Drop subscription on report failure (o13-17.2.1) |
Event Subscription Handling | drop_event_subscriptions | TRUE | RW | Drop event subscriptions (InformInfo and ServiceRecords) on port removal and SM coming out of STANDBY | |
Virtualization | Virtualization enabled | virt_enabled | 0 (Ignored) | RW | Enables/disables virtualization support |
Maximum ports in virtualization process | virt_max_ports_in_process | 4 | RW | Sets a number of ports to be handled on each virtualization process cycle. | |
Router | Router aguid enable | rtr_aguid_enable | 0 (Disabled) | RW | Defines whether the SM should create alias GUIDs required for router support for each port. |
Router path record flow label | rtr_pr_flow_label | 0 | RW | Defines flow label value to use in response for path records related to router. | |
Router path record tclass | rtr_pr_tclass | 0 | RW | Defines tclass value to use in response for path records related to router. | |
Router path record sl | rtr_pr_sl | 0 | RW | Defines sl value to use in response for path records related to router. | |
Router path record MTU | rtr_pr_mtu | 4 (IB_MTU_LEN_2048) | RW | Define MTU value to use in response for path records related to router. | |
Router path record rate | rtr_pr_rate | 16 (IB_PATH_RECORD_RATE_100_GBS) | RW | Defines rate value to use in response for path records related to router. | |
SA Security | SA Tnhanced Trust Model (SAETM) | sa_enhanced_trust_model | FALSE | RW | Controls whether SAETM is enabled. |
Untrusted GuidInfo records | sa_etm_allow_untrusted_guidinfo_rec | FALSE | RW | Controls whether to allow Untrusted Guidinfo record requests in SAETM. | |
Guidinfo record requests by VF | sa_etm_allow_guidinfo_rec_by_vf | FALSE | RW | Controls whether to allow Guidinfo record requests by vf in SAETM. | |
Untrusted proxy requests | sa_etm_allow_untrusted_proxy_requests | FALSE | RW | Controls whether to allow Untrusted proxy requests in SAETM. | |
Max number of multicast groups | sa_etm_max_num_mcgs | 128 | RW | Max number of multicast groups per port/vport that can be registered. | |
Max number of service records | sa_etm_max_num_srvcs | 32 | RW | Max number of service records per port/vport that can be registered. | |
Max number of event subscriptions | sa_etm_max_num_event_subs | 32 | RW | Max number of event subscriptions (InformInfo) per port/vport that can be registered. | |
SGID spoofing | sa_check_sgid_spoofing | TRUE | RW | If enabled, the SA checks for SGID spoofing in every request with GRH included, unless the SLID is from a router port at that request. |
This section is relevant for UFM advanced license only.
Single-root I/O virtualization (SR-IOV) enables a PCI Express (PCIe) device to appear to be multiple separate physical PCIe devices.
UFM is ready to work with SR-IOV devices by default. You can fine-tune the configuration using the SM configuration.
To modify UFM configuration for SR-IOV devices, modify the options in the conf/opensm.conf file as described in the following table:
Argument | Value | Description |
sm_assign_guid_func |
| Alias GUID assignment algorithm for a virtualized environment |
sm_assigned_guid | Any value between 0 (default) and ff | Set the SM assigned GUID byte when sm_assign_guid_func configuration option is set to uniq_count. The GUID is formed from OpenFabrics OUI is followed by 40 bits xy 00 ab cd ef where:
The configured SM assigned GUID byte must be subnet unique. |
allow_both_pkeys |
| Allow both full and limited membership on the same partition |
The following arguments are available for Connect-IB, ConnectX-4, and later devices:
Argument | Value | Description |
virt_enabled |
| Virtualization support |
virt_max_ports_in_process | Possible values: 0-65535; where 0 processes all pending ports Default: 2 | Maximum number of ports to be processed simultaneously by the virtualization manager |
virt_default_hop_limit | Possible values: 0-255 Default: 2 | Default value for hop limit to be returned in path records where either the source or destination are virtual ports |
This section is relevant only for ConnectX ® -3 family devices.
UFM has the ability to isolate particular switches from routing in order to perform maintenance of the switches with minimal interruption to the existing traffic in the fabric.
Isolating a switch from routing is done via UFM Subnet Manager as follows:
Create a file that includes either the node GUIDs or system GUID of the switches under maintenance. For example:
0x1234566 0x1234567
Set the filename of the parameter held_back_sw_file in the /conf/opensm.conf file (the same as the file created in Step 1).
Run:
kill -s HUP 'pidof opensm'
Once OpenSM completes rerouting, the traffic does not go through the ports of isolated switches.
To attach the switch to the routing:
Remove the GUID of the switch from the list of isolated switches defined in Step 1 of the isolation process.
Run:
kill -s HUP 'pidof opensm'
Once OpenSM completes rerouting, traffic will go through the switch.