Environment Variables¶

NVSHMEM provides a set of environment variables that allows users to configure the NVSHMEM implementation and receive information about the implementation.

NVSHMEM_VERSION¶

Value: Any

Print the library version at start-up

NVSHMEM_INFO¶

Value: Any

Print helpful text about all these environment variables

NVSHMEM_SYMMETRIC_SIZE¶

Value: Non-negative integer or floating point value with an optional character suffix

Specifies the size (in bytes) of the symmetric heap memory per PE. The resulting size is implementation-defined and must be least as large as the integer ceiling of the product of the numeric prefix and the scaling factor. The allowed character suffixes for the scaling factor are as follows:

k or K multiplies by $2^{10}$ (kibibytes)
m or M multiplies by $2^{20}$ (mebibytes)
g or G multiplies by $2^{30}$ (gibibytes)
t or T multiplies by $2^{40}$ (tebibytes)

For example, string “20m” is equivalent to the integer value 20971520, or 20 mebibytes. Similarly the string “3.1M” is equivalent to the integer value 3250586. Only one multiplier is recognized and any characters following the multiplier are ignored, so “20kk” will not produce the same result as “20m”. Usage of string “.5m” will yield the same result as the string “0.5m”.

An invalid value for NVSHMEM_SYMMETRIC_SIZE is an error, which the NVSHMEM library shall report by either returning a nonzero value from nvshmem_init_thread or causing program termination.

NVSHMEM_DEBUG¶

Value: WARN, INFO, TRACE, or any other value to enable basic debug output

Controls the debug information that is displayed from NVSHMEM.

NVSHMEM_DEBUG_FILE¶

Value: filename

Sets the filename where the debug output is written. The filename may contain %h for the hostname and %p for the PID.

NVSHMEM_ENABLE_NIC_PE_MAPPING¶

Value: Boolean

When this variable is not set or is set to 0, a PE is assigned the NIC on the node that is closest to it. When set to 1, NVSHMEM assigns NICs to PEs on a round-robin basis or uses NVSHMEM_HCA_PE_MAPPING or NVSHMEM_HCA_LIST when they are specified.

NVSHMEM_HCA_PE_MAPPING¶

Value: HCA Mapping

Specifies the mapping of HCAs to PEs as a comma-separated list so that the GPU that corresponds to the PE uses the given HCA for all transfers. Each entry in the comma separated list is in the hca_name:port:count format. For example, mlx5_0:1:2,mlx5_0:2:2 means that PEs 0 and 1 can be mapped to port 1 of mlx5_0, and PEs 2 and 3 can be mapped to port 2 of mlx5_0.

For this variable to be effective, NVSHMEM_ENABLE_NIC_PE_MAPPING must be set to 1.

NVSHMEM_HCA_LIST¶

Value: List of HCAs

Specifies a comma-separated list of HCAs to use in the NVSHMEM application. This variable is useful to skip disabled HCAs or HCAs that operate in a different mode from InfiniBand. Each entry in the comma separated list is in the hca_name:port format. For example, mlx5_1:1,mlx5_2:2 specifies using port 1 of mlx5_1 and port 2 of mlx5_2 in the application. A ^ before the list indicates an exclusion list. For example, ^mlx5_1:1,mlx5_1:2 means that NVSHMEM should not use mlx5_1 port 1 and mlx5_1 port 2.

For this variable to be effective, NVSHMEM_ENABLE_NIC_PE_MAPPING must be set to 1.

NVSHMEM_MPI_LIB_NAME¶

Value: String

Name of the MPI library that is used for bootstrapping NVSHMEM. By default, NVSHMEM looks for libraries with the name libmpi.so.

NVSHMEM_SHMEM_LIB_NAME¶

Value: String

Name of the OpenSHMEM library that is used for bootstrapping NVSHMEM. By default, NVSHMEM looks for libraries with the name liboshmem.so.

NVSHMEM_BARRIER_DISSEM_KVAL¶

Value: Integer radix

Radix of the dissemination algorithm that is used for nvshmem_barrier and nvshmem_barrier_all collectives. By default, NVSHMEM uses radix 2.

NVSHMEM_BARRIER_TG_DISSEM_KVAL¶

Value: Integer radix

Radix of the dissemination algorithm used for nvshmem_barrier_warp, nvshmem_barrier_group, nvshmem_barrier_all_warp, and nvshmem_barrier_all_group collectives. By default, NVSHMEM uses radix 2.

NVSHMEM_BOOTSTRAP_PMI¶

Value: One of ‘PMIX’, ‘PMI-2’, ‘PMI2’ (Alias of PMI-2), or ‘PMI’

Specifies the desired PMI version to use when bootstrapping NVSHMEM applications. NVSHMEM_BOOTSTRAP_PMI overrides the default PMI version set at NVSHMEM compilation time. This variable is ignored by applications being bootstrapped by an external MPI or SHMEM library. Otherwise, setting this environment variable to anything other than the values listed above results in an NVSHMEM initialization error.