mlx_fpga – Burning and Debugging Tool for NVIDIA Devices with FPGA

mlx_fpga tool allows the user to burn and update a new FPGA image on NVIDIA Innova™ adapter cards. For instructions on how to burn, please refer to Burning the FPGA’s Flash Device using the mlx_fpga Burning Tool.

The tool also enables the user to read/write individual registers in the FPGA configuration space.

Tool Requirements

• An Innova IPsec 4 Lx EN / Innova Flex 4 Lx EN adapter card with an FPGA device
• For Innova IPsec 4 Lx EN: Load the mlx5_fpga_tools module
• For Innova Flex 4 Lx EN: Load the mlx_accel_tools module

Note: The module is included in the Innova driver which is supplied for this product line only, and available through the Support.

• Start mst service with the fpga lookup flag (mst start --with_fpga)

mlx_fpga Synopsis

where:

Examples of mlx_fpga Usage

Adding FPGA mst Device Interface

For Innova IPsec 4 Lx EN: Load the mlx5_fpga_tools module.

# modprobe mlx5_fpga_tools
# mst start --with_fpga
# mst status
MST modules:
------------
MST PCI module is not loaded
MST PCI configuration module is not loaded
MST devices:
------------
No MST devices were found nor MST modules were loaded.
You may need to run 'mst start' to load MST modules.
FPGA devices:
-------------------
/dev/mst/mt4117_pciconf0_fpga_i2c
/dev/mst/mt4117_pciconf1_fpga_rdma1

Note: In the last line, it is recommended to use the RDMA device as it is faster. I2C is used for recovery purposes when RDMA is not functional.

For Innova Flex 4 Lx EN: Load the mlx_accel_tools module.

# modprobe mlx_accel_tools
# mst start --with_fpga
# mst status
MST modules:
------------
MST PCI module is not loaded
MST PCI configuration module is not loaded
MST devices:
------------
No MST devices were found nor MST modules were loaded.
You may need to run 'mst start' to load MST modules.
FPGA devices:
-------------------
/dev/mst/mt4117_pciconf0_fpga_i2c
/dev/mst/mt4117_pciconf1_fpga_rdma

Note: In the last line, it is recommended to use the RDMA device as it is faster. I2C is used for recovery purposes when RDMA is not functional.

For recovery only, where modules are broken or missing.

# mst start --with_fpga_fw_access
Starting MST (Mellanox Software Tools) driver set
Loading MST PCI module - Success
Loading MST PCI configuration module - Success
Create devices
Unloading MST PCI module (unused) - Success
# mst status
MST modules:
------------
MST PCI module is not loaded
MST PCI configuration module loaded
MST devices:
------------
/dev/mst/mt4117_pciconf0         - PCI configuration cycles access.
                                   domain:bus:dev.fn=0000:01:00.0 addr.reg=88 data.reg=92
                                   Chip revision is: 00
FPGA devices:
-------------------
/dev/mst/mt4117_pciconf0_fpga

Burning the FPGA’s Flash Device using the mlx_fpga Burning Tool

mlx_fpga tool burns a .bin file onto the FPGA flash device. 

The Innova Flex .bin file must be generated according to the instructions listed in the Innova Flex Adapter Card User Manual.

1. Burn the image.

   # mlx_fpga -d <device> burn image.bin

2. Load the FPGA image from flash according to “Loading the Tool” below or power cycle the machine for change to take effect.

Loading the Tool 

Load an FPGA image from user configurable flash:

# mlx_fpga -d <device> l/load <optional: load options>

where <optional: load options> is:

Debugging the Tool

Reading One Debug Register:

# mlx_fpga -d <device> read 0x0

Writing One Debug Register:

# mlx_fpga -d <device> write 0x0 0x0

Updating the FPGA Image

In order to verify the new image burned to the FPGA, the user can use mlx_fpga tool and read the following registers: