MLNX_OFED Documentation Rev 4.9-2.2.6.0-LTS

Physical Address Memory Allocation

Physical Address Memory Region (PA-MR) allows for managing physical memory used for posting send and receive requests. This can benefit the performance of applications that register large memory regions with random access.
When working with a physical address memory region (PA-MR), the memory region that is used must be:

  • Non-swappable (pinned)

  • A continuous address space in physical memory

One way of allocating physical memory is by using Linux huge pages, which reserve a non-swappable, continuous physical address space. Access to this memory can be done using the mmap system call.
There are multiple methods of translating the returned virtual address from mmap to the physical address. Using a physical memory region and supplying the physical address when posting requests improve the random access performance since no conversion from a virtual address to a physical address is needed.

Warning

When using PA-MR, users bypass the memory protection kernel mechanism, which might crash the system when used incorrectly. This feature is recommended for experienced users with an understanding of the possible risks.

Procedure_Heading_Icon.PNG

In order to enable Physical Memory Regions:

PA-MR is not enabled by default in MLNX_OFED. MLNX_OFED sources should be recompiled using the following configuration flag:

Copy
Copied!
            

--with-pa-mr

Physical Address memory registration is done using ibv_exp_reg_mr verb. For more information, please refer to the verb's manual page.
For further information, please refer to Physical Address Memory Region Community post.

© Copyright 2023, NVIDIA. Last updated on Oct 23, 2023.